Topic: "data-orchestration"
kestra-io/kestra
:zap: Workflow Automation Platform. Orchestrate & Schedule code in any language, run anywhere, 600+ plugins. Alternative to Airflow, n8n, Rundeck, VMware vRA, Zapier ...
Language: Java - Size: 55.9 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 17,577 - Forks: 1,476
Alluxio/alluxio
Alluxio, data orchestration for analytics and machine learning in the cloud
Language: Java - Size: 196 MB - Last synced at: 3 days ago - Pushed at: 9 days ago - Stars: 6,981 - Forks: 2,941
cubefs/cubefs
cloud-native distributed storage
Language: Go - Size: 243 MB - Last synced at: about 13 hours ago - Pushed at: 1 day ago - Stars: 5,051 - Forks: 659
apache/incubator-graphar
An open source, standard data file format for graph data storage and retrieval.
Language: C++ - Size: 6.84 MB - Last synced at: 4 days ago - Pushed at: 11 days ago - Stars: 274 - Forks: 66
iam-mhaseeb/Skytrax-Data-Warehouse 📦
A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
Language: Python - Size: 1.34 MB - Last synced at: 3 months ago - Pushed at: about 5 years ago - Stars: 133 - Forks: 28
jonathanneo/data-aware-orchestration
Data-aware orchestration with dagster, dbt, and airbyte
Language: Python - Size: 1.36 MB - Last synced at: 3 days ago - Pushed at: over 2 years ago - Stars: 31 - Forks: 0
kestra-io/examples
Best practices for data workflows, integrations with the Modern Data Stack (MDS), Infrastructure as Code (IaC), Cloud Provider Services
Language: HCL - Size: 3.28 MB - Last synced at: 17 days ago - Pushed at: about 1 month ago - Stars: 25 - Forks: 9
ozkary/data-engineering-mta-turnstile
Data Engineering - Metropolitan Transportation Authority (MTA) Subway Data Analysis
Language: Jupyter Notebook - Size: 12.9 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 25 - Forks: 4
SAP-samples/btp-data-to-value-workshop
This repo contains a dataset, exercises, and sample code for an end-to-end SAP BTP data-to-value bootcamp covering SAP HANA Cloud, SAP Data Warehouse Cloud, SAP Data Intelligence Cloud, and SAP Analytics Cloud.
Language: Jupyter Notebook - Size: 167 MB - Last synced at: 25 days ago - Pushed at: about 2 months ago - Stars: 24 - Forks: 24
astronomer/airflow-provider-fivetran-async
A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran
Language: Python - Size: 196 KB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 22 - Forks: 9
anna-geller/kestra-ci-cd
CI/CD repository template to automate deployments of your production flows
Language: HCL - Size: 104 KB - Last synced at: about 1 month ago - Pushed at: 10 months ago - Stars: 12 - Forks: 5
Alluxio/k8s-operator
An operator for managing Alluxio system on Kubernetes cluster
Language: Go - Size: 270 KB - Last synced at: about 1 month ago - Pushed at: over 1 year ago - Stars: 12 - Forks: 9
dagster-io/dagster-quickstart 📦
Get started with Dagster ASAP
Language: Python - Size: 20.5 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 11 - Forks: 122
anna-geller/kestra-terraform-examples
Bring Infrastructure as Code best practices to your data workflows with Kestra and Terraform
Language: HCL - Size: 746 KB - Last synced at: about 1 month ago - Pushed at: almost 2 years ago - Stars: 5 - Forks: 0
kestra-io/data-engineering-zoomcamp
Code for the Data Engineering Zoomcamp course
Size: 470 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 3 - Forks: 1
longNguyen010203/Finance-Data-Ingestion-Pipeline-with-Kafka
Develop a real-time data ingestion pipeline using Kafka and Spark. Collect minute-level stock data from Yahoo Finance, ingest it into Kafka, and process it with Spark Streaming, storing the results in Cassandra. Orchestrated the workflow using Airflow deployed on Docker.
Language: Python - Size: 250 KB - Last synced at: about 1 month ago - Pushed at: 5 months ago - Stars: 3 - Forks: 1
taquynhnga2001/proptech-dagster
Build an ELT pipeline with dagster and dbt to schedule loading HDB resale transactions in Singapore into Google BigQuery data warehouse, then create Power BI dashboard to enhance insight exploration.
Language: Python - Size: 3.42 MB - Last synced at: 24 days ago - Pushed at: about 1 month ago - Stars: 2 - Forks: 0
zpencerguy/superdoppler
Data orchestration repo with Docker deployment
Language: Python - Size: 38.1 KB - Last synced at: almost 2 years ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0
jasontanx/prefect-learning
Prefect - Data orchestration tool practice & learning
Language: Python - Size: 314 KB - Last synced at: about 1 month ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 0
stemitom/postgres-pipeline
A simple pipeline infrastructure with ETL pipeline contained in a Docker environment on Apache Airflow for orchestration and Postgres for data warehousing
Language: Python - Size: 217 KB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 3
Annielytix/azure-data-factory-data-vault
Working with SCD Type (Change Data Capture) and need a Data Vault model to test Azure Data Factory v2? - This Code with Help!
Size: 2.75 MB - Last synced at: about 1 year ago - Pushed at: almost 6 years ago - Stars: 1 - Forks: 0
GADES-DATAENG/webinar
Code, scripts, and resources for the Data Engineering Fundamentals Course Webinar, covering Python, data pipelines, Apache Airflow, and more.
Language: Python - Size: 26.9 MB - Last synced at: about 1 month ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0
Wireforce-LLC/m3
☕ Data Orchestrator. Without abstractions
Language: TypeScript - Size: 139 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0
kingabzpro/5-Airflow-Alternatives-for-Data-Orchestration-Tutorial
Code examples of Luigi, Prefect, Kedro, Dagster, and MageAI
Language: Python - Size: 40 KB - Last synced at: about 2 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0
philiporlando/dagster_university
I created this repo to follow along with the examples in the Dagster University Essentials course.
Language: Python - Size: 90.8 KB - Last synced at: 12 months ago - Pushed at: 12 months ago - Stars: 0 - Forks: 0
ddeutils/data-orchestra
❌ Full-Stack Data Orchestration config by Yaml template with Flask & HTMX
Language: Python - Size: 3.81 MB - Last synced at: about 20 hours ago - Pushed at: 12 months ago - Stars: 0 - Forks: 0
MostafaNabilll/end2end_pipeline
End to End data engineering project
Language: Python - Size: 3.87 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
jacquessham/airflow_notes
Repository to store scripts and notes on Airflow
Language: Python - Size: 186 KB - Last synced at: 12 months ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
