GitHub topics: dataops

raystack/homebrew-tap

This repository is home to the original homebrew taps for products throughout the Raystack ecosystem.

Language: Ruby - Size: 264 KB - Last synced at: 41 minutes ago - Pushed at: about 2 hours ago - Stars: 31 - Forks: 4

montara-io/dbt-command-center

Never sift through endless dbt™ logs again. dbt Command Center is a free, open-source, local web application that provides a user-friendly interface to monitor and manage dbt runs.

Language: TypeScript - Size: 3.55 MB - Last synced at: about 14 hours ago - Pushed at: about 15 hours ago - Stars: 28 - Forks: 0

GitDataAI/jzfs

A Git-like Version Control File System for AI & Data Product Management.

Language: Rust - Size: 3.09 MB - Last synced at: about 17 hours ago - Pushed at: about 18 hours ago - Stars: 105 - Forks: 11

flyteorg/flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

Language: Go - Size: 316 MB - Last synced at: 1 day ago - Pushed at: 2 days ago - Stars: 6,182 - Forks: 715

tenzir/tenzir

Tenzir is the data pipeline engine for security teams.

Language: C++ - Size: 1.95 GB - Last synced at: about 20 hours ago - Pushed at: 1 day ago - Stars: 670 - Forks: 95

raystack/dagger

Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.

Language: Java - Size: 12 MB - Last synced at: 2 days ago - Pushed at: over 1 year ago - Stars: 273 - Forks: 41

DataRecce/recce

The data-validation toolkit for enhanced dbt (data build tool) PR review

Language: Python - Size: 18.7 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 336 - Forks: 12

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.

Language: TypeScript - Size: 62.9 MB - Last synced at: 3 days ago - Pushed at: 4 days ago - Stars: 3,988 - Forks: 368

jostrm/azure-enterprise-scale-ml

Enterprise Scale AIFactory (esml) - on Azure

Language: Jupyter Notebook - Size: 118 MB - Last synced at: 3 days ago - Pushed at: 4 days ago - Stars: 40 - Forks: 12

meltano/meltano

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

Language: Python - Size: 140 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 2,029 - Forks: 175

lancedb/lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Language: Rust - Size: 21.8 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 4,470 - Forks: 280

lensesio/fast-data-dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, , 20+ connectors

Language: Shell - Size: 17.9 MB - Last synced at: 3 days ago - Pushed at: 4 days ago - Stars: 2,039 - Forks: 337

kelvins/awesome-dataops

:sunglasses: A curated list of awesome DataOps tools

Language: Python - Size: 112 KB - Last synced at: 1 day ago - Pushed at: 6 months ago - Stars: 184 - Forks: 29

yashj1301/virat-cricket-analysis

This repository is an end-to-end project to extract cricket statistics for Indian Cricketer, Virat Kohli, from the ESPN CricInfo website. The rights to the data belong to ESPN CricInfo.

Language: Jupyter Notebook - Size: 3.16 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 0 - Forks: 0

elementary-data/elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

Language: HTML - Size: 205 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 2,047 - Forks: 182

raystack/frontier

Frontier is an all-in-one user management platform that provides identity, access and billing management to help organizations secure their systems and data. (Open source alternative to Clerk, WorkOS)

Language: Go - Size: 19 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 284 - Forks: 35

datachecks/dcs-core

Open Source Data Quality Monitoring.

Language: Python - Size: 4.32 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 154 - Forks: 22

TobikoData/sqlmesh

Scalable and efficient data transformation framework - backwards compatible with dbt.

Language: Python - Size: 74.1 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 2,231 - Forks: 200

tulibraries/tulflow

TU Libraries Python Library for functions used in indexing ETL, particularly for Airflow

Language: Python - Size: 2.52 MB - Last synced at: 7 days ago - Pushed at: 7 days ago - Stars: 3 - Forks: 1

tulibraries/cob_datapipeline

Airflow Data Processing Pipeline for TUL Catalog on Blacklight Data

Language: Python - Size: 2.79 MB - Last synced at: 7 days ago - Pushed at: 7 days ago - Stars: 6 - Forks: 0

goto/optimus Fork of raystack/optimus

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Language: Go - Size: 33.1 MB - Last synced at: about 6 hours ago - Pushed at: about 7 hours ago - Stars: 7 - Forks: 4

whylabs/whylogs

An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈

Language: Jupyter Notebook - Size: 181 MB - Last synced at: 4 days ago - Pushed at: 3 months ago - Stars: 2,707 - Forks: 125

raystack/meteor

Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog.

Language: Go - Size: 14.5 MB - Last synced at: 2 days ago - Pushed at: 6 months ago - Stars: 204 - Forks: 42

alibaba/SREWorks

Cloud Native DataOps & AIOps Platform | 云原生数智运维平台

Language: Java - Size: 227 MB - Last synced at: 7 days ago - Pushed at: about 1 year ago - Stars: 1,872 - Forks: 409

datavane/tis

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI

Language: Java - Size: 28.5 MB - Last synced at: 8 days ago - Pushed at: 8 days ago - Stars: 1,124 - Forks: 240

jimyoshida/dx-devops-landscape

Overview of technologies and concepts for DataMLDevSecOps, with history

Language: JavaScript - Size: 177 KB - Last synced at: 1 day ago - Pushed at: 9 days ago - Stars: 4 - Forks: 0

Azure-Samples/modern-data-warehouse-dataops

DataOps for Microsoft Data Platform technologies. https://aka.ms/dataops-repo

Language: Shell - Size: 115 MB - Last synced at: 8 days ago - Pushed at: 11 days ago - Stars: 632 - Forks: 495

opendatadiscovery/awesome-data-catalogs

📙 Awesome Data Catalogs and Observability Platforms.

Size: 109 KB - Last synced at: 10 days ago - Pushed at: 19 days ago - Stars: 826 - Forks: 60

microsoft/devsquad-dataops

DevSquad In a Day - DataOps. A hands-on lab for exercising DataOps practices on Azure.

Language: PowerShell - Size: 52.8 MB - Last synced at: 1 day ago - Pushed at: 11 days ago - Stars: 36 - Forks: 21

cleanlab/cleanlab

The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

Language: Python - Size: 11.5 MB - Last synced at: 12 days ago - Pushed at: about 1 month ago - Stars: 10,446 - Forks: 822

datacoves/docs

Datacoves public documentation

Language: Python - Size: 47.5 MB - Last synced at: about 21 hours ago - Pushed at: 12 days ago - Stars: 3 - Forks: 2

tulibraries/funnel_cake_index

Index PA Digital metadata into Solr for Funnel Cake

Language: Ruby - Size: 217 KB - Last synced at: 13 days ago - Pushed at: 13 days ago - Stars: 0 - Forks: 0

polyaxon/traceml

Engine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.

Language: Python - Size: 118 MB - Last synced at: 10 days ago - Pushed at: 14 days ago - Stars: 514 - Forks: 44

polyaxon/cli

Polyaxon Core Client & CLI to streamline MLOps

Language: Python - Size: 4.52 MB - Last synced at: 6 days ago - Pushed at: 14 days ago - Stars: 19 - Forks: 18

vandresen/DatabaseManager

Web based tool to manage PPDM databases that includes data quality management

Language: C# - Size: 3.48 MB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 4 - Forks: 3

raystack/optimus

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Language: Go - Size: 12.2 MB - Last synced at: 13 days ago - Pushed at: 11 months ago - Stars: 748 - Forks: 155

tulibraries/funcake_dags

Airflow DAGs for PA Digital aggregation processes

Language: Python - Size: 2.31 MB - Last synced at: 17 days ago - Pushed at: 17 days ago - Stars: 4 - Forks: 1

awslabs/aws-ddk

An open source development framework to help you build data workflows and modern data architecture on AWS.

Language: TypeScript - Size: 5.62 MB - Last synced at: 2 days ago - Pushed at: 6 days ago - Stars: 263 - Forks: 21

DataKitchen/dataops-testgen

DataOps Data Quality TestGen is part of DataKitchen's Open Source Data Observability. DataOps TestGen delivers simple, fast data quality test generation and execution by data profiling, new dataset hygiene review, AI generation of data quality validation tests, ongoing testing of data refreshes, & continuous anomaly monitoring

Language: Python - Size: 5.22 MB - Last synced at: 15 days ago - Pushed at: 19 days ago - Stars: 55 - Forks: 3

raptor-ml/raptor

Transform your pythonic research to an artifact that engineers can deploy easily.

Language: Go - Size: 4.55 MB - Last synced at: 14 days ago - Pushed at: about 1 month ago - Stars: 152 - Forks: 12

DataKitchen/data-observability-installer

Installer for DataKitchen's Open Source Data Observability Products. Data breaks. Servers break. Your toolchain breaks. Ensure your team is the first to know and the first to solve with visibility across and down your data estate. Save time with simple, fast data quality test generation and execution. Trust your data, tools, and systems end to end.

Language: Python - Size: 207 KB - Last synced at: 17 days ago - Pushed at: 27 days ago - Stars: 112 - Forks: 9

aabouzaid/modern-data-platform-poc

My M.Sc. dissertation: Modern Data Platform using DataOps, Kubernetes, and Cloud-Native ecosystem to build a resilient Big Data platform based on Data Lakehouse architecture which is the base for Machine Learning (MLOps) and Artificial Intelligence (AIOps).

Language: Jupyter Notebook - Size: 5.52 MB - Last synced at: 1 day ago - Pushed at: 11 months ago - Stars: 8 - Forks: 1

awesome-mlops/awesome-ml-monitoring

A curated list of awesome open source tools and commercial products for monitoring data quality, monitoring model performance, and profiling data 🚀

Size: 4.88 KB - Last synced at: 12 days ago - Pushed at: 12 months ago - Stars: 75 - Forks: 7

flowerfine/scaleph

Open data platform based on Kubernetes. Scaleph supports SeaTunnel、Flink and Doris backended by SeaTunnel on Flink engine、Flink Kubernetes Operator and Doris operator.

Language: Java - Size: 20.9 MB - Last synced at: 17 days ago - Pushed at: 3 months ago - Stars: 379 - Forks: 108

tulibraries/cob_az_index

Language: Ruby - Size: 592 KB - Last synced at: 21 days ago - Pushed at: 21 days ago - Stars: 0 - Forks: 0

tulibraries/cob_web_index

Language: Ruby - Size: 9.64 MB - Last synced at: 21 days ago - Pushed at: 21 days ago - Stars: 0 - Forks: 0

taivop/awesome-data-annotation

A list of tools for annotating data, managing annotations, etc.

Size: 58.6 KB - Last synced at: 9 days ago - Pushed at: 9 months ago - Stars: 596 - Forks: 66

lensesio/lenses-go

Lenses.io CLI (command-line interface)

Language: Go - Size: 13.7 MB - Last synced at: 3 days ago - Pushed at: 5 months ago - Stars: 35 - Forks: 8

GlueOps/glueops-dev

This repository contains the GlueOps documentation website built using Docusaurus 2. It provides comprehensive guides and tutorials for deploying and managing applications using the GlueOps platform. The site includes setup instructions, configuration details, and best practices for GitOps workflows.

Language: JavaScript - Size: 106 MB - Last synced at: about 11 hours ago - Pushed at: 13 days ago - Stars: 2 - Forks: 1

lensesio/lenses-docker

❤for real-time DataOps - where the application and data fabric blends - Lenses

Language: Shell - Size: 495 KB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 156 - Forks: 23

DataKitchen/dataops-observability

DataOps Observability is part of DataKitchen's Open Source Data Observability. DataOps Observability monitors every data journey from data source to customer value, from any team development environment into production, across every tool, team, environment, and customer so that problems are detected, localized, and understood immediately.

Language: Python - Size: 3.91 MB - Last synced at: 11 days ago - Pushed at: 27 days ago - Stars: 43 - Forks: 3

raystack/firehose

Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

Language: Java - Size: 15.2 MB - Last synced at: 2 days ago - Pushed at: 7 months ago - Stars: 336 - Forks: 62

google/space

Unified storage framework for the entire machine learning lifecycle

Language: Python - Size: 825 KB - Last synced at: 1 day ago - Pushed at: about 1 year ago - Stars: 156 - Forks: 8

ricardolsmendes/aws-glue-ci-cd-blueprint

Companion repository for the "Streamlining AWS Glue CI/CD — A Comprehensive Blueprint" blog post

Language: HCL - Size: 518 KB - Last synced at: 22 days ago - Pushed at: 5 months ago - Stars: 12 - Forks: 3

gni/bonjour-python

🐍 Formation Python Intégrale 🌟: Plongez dans Python depuis les bases jusqu'aux projets avancés. Clarté, pratique, et innovation au rendez-vous. Commencez votre aventure Python maintenant!

Size: 1.07 MB - Last synced at: 7 days ago - Pushed at: 4 months ago - Stars: 19 - Forks: 0

AhmedNasef3/Software-Task

This task explains OOP system types , GUI frameworks, Methods for connecting database with python application. and also explains Devops, MLops, Dataops tools.

Size: 845 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 0

MeltanoLabs/Singer-Working-Group

Working group for ongoing development and iteration of the Singer Spec, the de-facto protocol for open source data connectors. Please use "Issues" to create discussion items - or use "Discussions" for general questions.

Size: 28.3 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 15 - Forks: 4

eyecan-ai/pipelime-python

A swiss army knife for data processing!

Language: Python - Size: 5.08 MB - Last synced at: 6 days ago - Pushed at: about 1 month ago - Stars: 19 - Forks: 0

SiyaMathe/Modern-Data-Architecture-Concepts

This project aims to provide a comprehensive overview of modern data architecture concepts, including data lakes, data meshes, cloud-based solutions, and real-time processing, and their application in addressing contemporary data challenges.

Size: 8.79 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

raystack/raccoon

Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols.

Language: Go - Size: 12.2 MB - Last synced at: 2 days ago - Pushed at: 7 months ago - Stars: 209 - Forks: 30

gibbsbravo/DataDelta

The best Python package for comparing two dataframes

Language: Python - Size: 514 KB - Last synced at: 15 days ago - Pushed at: over 3 years ago - Stars: 11 - Forks: 2

Gabigol123456/versioned-data-lakehouse

🌊 Git-like Version Control for Data with Nessie, Iceberg, and Spark

Size: 1.95 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0

pbi-tools/pbi-tools

Power BI DevOps & Source Control Tool

Language: C# - Size: 15.5 MB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 340 - Forks: 62

google/grizzly

End-to-end DataOps platform deployed by Terraform.

Language: Python - Size: 113 MB - Last synced at: 1 day ago - Pushed at: about 1 month ago - Stars: 66 - Forks: 10

c-3lab/dim

📦 dim: Manage the open data in your project like a package manager.

Language: TypeScript - Size: 294 MB - Last synced at: 1 day ago - Pushed at: about 1 year ago - Stars: 129 - Forks: 22

nellaivijay/awesome-my-notes

A comprehensive repository for generative AI research, platform engineering, updates, data and software engineering, system design and architecture, interview resources, notebooks, and much more!

Size: 437 KB - Last synced at: 8 days ago - Pushed at: about 2 months ago - Stars: 20 - Forks: 3

raystack/compass

Compass is an enterprise data catalog that makes it easy to find, understand, and govern data.

Language: Go - Size: 4.69 MB - Last synced at: 16 days ago - Pushed at: over 1 year ago - Stars: 64 - Forks: 7

awesome-mlops/awesome-data-management

A curated list of awesome open source tools and commercial products to catalog, version, and manage data 🚀

Size: 4.88 KB - Last synced at: 12 days ago - Pushed at: about 3 years ago - Stars: 32 - Forks: 3

TheDrillDown/TheDrillDown

The Drill Down with Ahmad & James

Size: 7.66 MB - Last synced at: 5 days ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0

dinhanhthi/mooc-de

The resources when I learned the course Data Engineering provided by DeepLearning.AI on Coursera.

Language: Jupyter Notebook - Size: 93.9 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 2 - Forks: 2

Gokcan61/console

A console in computing refers to a text-based interface where users can interact with a computer system through commands. It typically provides a direct way to execute tasks, access system resources, and troubleshoot issues without the need for a graphical user interface.

Size: 1000 Bytes - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

BMSihlas/dataops-greenflow-sage

A sustainability insights project that processes sensor data, stores it in PostgreSQL, and exposes insights via an API and an interactive dashboard. [edit. dataops-bootcamp]

Language: Python - Size: 1000 Bytes - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

raystack/siren

Siren provides an easy-to-use universal alert, notification, channels management framework for the entire observability infrastructure.

Language: Go - Size: 7.03 MB - Last synced at: 1 day ago - Pushed at: 9 months ago - Stars: 78 - Forks: 15

pirocheto/data-engineering-projects

Website about my data engineering projects

Language: HTML - Size: 1.85 MB - Last synced at: 15 days ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

abeltavares/versioned-data-lakehouse

🌊 Git-like Version Control for Data with Nessie, Iceberg, and Spark

Language: Jupyter Notebook - Size: 3.95 MB - Last synced at: about 1 month ago - Pushed at: 3 months ago - Stars: 3 - Forks: 2

merantix-momentum/squirrel-core

A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way :chestnut:

Language: Python - Size: 3.16 MB - Last synced at: 9 days ago - Pushed at: about 1 month ago - Stars: 281 - Forks: 8

noahgift/data-engineering-and-dataops

Duke MIDS: Data Engineering and DataOps Course

Language: HTML - Size: 164 KB - Last synced at: about 2 months ago - Pushed at: 3 months ago - Stars: 64 - Forks: 19

zncdatadev/kubedoop

The modular open source data platform using kubernetes and cloud-native ecosystem which is the base for DataOps/MLOps(LLMOps)

Size: 1.88 MB - Last synced at: 18 days ago - Pushed at: 4 months ago - Stars: 3 - Forks: 3

brunocampos01/data-engineering

Language: Python - Size: 165 MB - Last synced at: 6 days ago - Pushed at: 3 months ago - Stars: 11 - Forks: 2

eZWALT/ADSDB-DS-EtE-Project

MDS-FIB Algorithms, Data Structures and Databases (ADSDB) Subject 2024-25 Q1, Data-Science End-to-End project path

Language: Jupyter Notebook - Size: 14.8 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

raystack/charts

This repository is home to the original helm charts for products throughout the open data platform ecosystem.

Language: Smarty - Size: 10.8 MB - Last synced at: 16 days ago - Pushed at: 3 months ago - Stars: 41 - Forks: 11

abroniewski/TPC-DI-MS-SQL-Benchmark

Using TPC-DI to benchmark MS SQL server using SQL script for extract, transform and load (ETL).

Language: TSQL - Size: 3.83 MB - Last synced at: about 1 month ago - Pushed at: almost 3 years ago - Stars: 2 - Forks: 1

beneath-hq/beneath 📦

Beneath is a serverless real-time data platform ⚡️

Language: Go - Size: 11 MB - Last synced at: 2 days ago - Pushed at: about 3 years ago - Stars: 84 - Forks: 10

goto/optimus Fork of raystack/optimus

beneath-hq/beneath 📦

merantix-momentum/squirrel-datasets-core 📦