GitHub topics: elt-pipeline
covalenthq/bsp-agent
Decodes, packs, encodes, proves, stores and uploads block-replicas (primarily "block-specimens") produced by EVM or non-EVM byte code based blockchains.
langage: Go - taille: 181 Mo - dernière synchronisation: il y a 4 jours - enregistré: il y a 4 jours - étoiles: 16 - forks: 12
titanium0202/Coffee_Shop_AI_Agents
About This project is an innovative coffee shop application designed to bring an engaging and personalized experience to coffee lovers. The app leverages AI-powered agents for chat-based interactions and integrates modern web and mobile development techniques to provide seamless ordering and delivery services.
taille: 31,9 Mo - dernière synchronisation: il y a 4 jours - enregistré: il y a 4 jours - étoiles: 0 - forks: 0
paty-oliveira/carris-data-pipeline
Repository for Extraction, Loading and Transformation of Carris data.
langage: Python - taille: 34,2 ko - dernière synchronisation: il y a 8 jours - enregistré: il y a 8 jours - étoiles: 0 - forks: 0
dataforgelabs/dataforge-core
DataForge helps data teams write functional transformation pipelines by leveraging software engineering principles
langage: PLpgSQL - taille: 750 ko - dernière synchronisation: il y a 4 jours - enregistré: il y a 4 jours - étoiles: 48 - forks: 1
andres-chirinos/github-actions-etl-template
Template for ETL processes using GitHub Actions
langage: Jupyter Notebook - taille: 40 ko - dernière synchronisation: il y a 13 jours - enregistré: il y a 15 jours - étoiles: 0 - forks: 0
PragMath-Analytics/E2E-DataPipeline-SelfHosted-OpenSource
An end-to-end ELT project for self hosted environments using open source tools - PostgresSQL (database), Sling (ingestion), dbt (transformations), and metabase (visualizations)
langage: Python - taille: 352 ko - dernière synchronisation: il y a 13 jours - enregistré: il y a 20 jours - étoiles: 0 - forks: 0
underradicals/bring_me_a_bucket_and_i_will_show_you_a_bucket
An ELT, API and UI
langage: Python - taille: 449 ko - dernière synchronisation: il y a 26 jours - enregistré: il y a 26 jours - étoiles: 0 - forks: 0
vigneshSs-07/Google-Cloud-Professional-Data-Engineer-ACompleteGuide
This Repo contains all study, lab and supportive materials for Udemy course on "Google Cloud Professional Data Engineer - A Complete Guide".
langage: Python - taille: 46,5 Mo - dernière synchronisation: il y a environ un mois - enregistré: il y a environ un mois - étoiles: 4 - forks: 3
ClaytonAllenThompsonII/WebApp
Django Web Application for inventory and invoice management.
langage: Python - taille: 7,15 Mo - dernière synchronisation: il y a environ un mois - enregistré: il y a environ un mois - étoiles: 0 - forks: 0
hordiales/dataform-example-citibike-trips
Ejemplo Clases. Dataform example. SQL. Assertions. Data Quality
taille: 4,88 ko - dernière synchronisation: il y a environ 2 mois - enregistré: il y a environ 2 mois - étoiles: 0 - forks: 0
stellar/stellar-dbt-public
Public DBT instance to aid in data transformation for analytics purposes
langage: Shell - taille: 491 ko - dernière synchronisation: il y a environ un mois - enregistré: il y a environ 2 mois - étoiles: 4 - forks: 4
kkumyk/server-logs-daily-data-pipeline
A data engineering project with dbt, Docker, Kestra, Terraform, GCP and Looker.
langage: HCL - taille: 755 ko - dernière synchronisation: il y a 3 mois - enregistré: il y a 3 mois - étoiles: 1 - forks: 0
ShaheerKhan200/gads-modern-data-stack
ELT Batch Pipeline using modern data stack (dbt, postgres, docker, ec2 etc)
langage: Shell - taille: 267 Mo - dernière synchronisation: il y a 3 mois - enregistré: il y a 3 mois - étoiles: 0 - forks: 0
eriksszva/ELT-using-DBT-and-GCP
An ELT pipeline in GCP using dbt, Apache Airflow, and BigQuery, transforming raw retail data into a structured, optimized format.
langage: Python - taille: 2,03 Mo - dernière synchronisation: il y a 3 mois - enregistré: il y a 3 mois - étoiles: 0 - forks: 0
ndomah/ELT-Pipeline
Simple ELT pipeline using dbt, Snowflake, and Apache Airflow.
langage: Python - taille: 244 ko - dernière synchronisation: il y a 2 mois - enregistré: il y a 3 mois - étoiles: 0 - forks: 0
ayoubmesquiny/Employee-Performance-Analysis-Pipeline
langage: Python - taille: 70,3 ko - dernière synchronisation: il y a 4 mois - enregistré: il y a 4 mois - étoiles: 0 - forks: 0
zythedeveloper/first-elt-project
This project is based on Justin B. Chau's tutorial on FreeCodeCamp's YouTube channel. It explores the creation and deployment of a custom Extract, Load, and Transform (ELT) pipeline, demonstrating practical data engineering concepts and techniques.
langage: Python - taille: 150 ko - dernière synchronisation: il y a 2 mois - enregistré: il y a 4 mois - étoiles: 0 - forks: 0
akshay-gera/dbt_bigquery_project
End to End Data Pipeline Project on Google Cloud Warehouse, DBT Data Modelling and Power BI Data Visualization
taille: 69,3 ko - dernière synchronisation: il y a 4 mois - enregistré: il y a 4 mois - étoiles: 0 - forks: 0
tanhoang1808/northwind_data_warehouse
This project perform an ELT solution to design a Data Warehouse in Snowflake
taille: 2,27 Mo - dernière synchronisation: il y a 4 mois - enregistré: il y a 4 mois - étoiles: 1 - forks: 0
ibnufajar1994/elt-data-warehouse
Build and Orchestrate an ELT Data Pipeline Using Luigi
langage: Python - taille: 40,4 Mo - dernière synchronisation: il y a 5 mois - enregistré: il y a 5 mois - étoiles: 0 - forks: 0
rizkipragustono/ecommerce_elt_project
Modern ELT with Snowflake, dbt, and Star Schema for E-Commerce
langage: Python - taille: 23,4 ko - dernière synchronisation: il y a environ un mois - enregistré: il y a 5 mois - étoiles: 1 - forks: 0
mfajarandikha/DataEngineering_ELT_Airflow
This repository demonstrates an end-to-end ELT (Extract, Load, Transform) pipeline that extracts data from a source PostgreSQL database, loads it into a destination PostgreSQL database, and performs data transformations using dbt (Data Build Tool).
langage: Python - taille: 101 ko - dernière synchronisation: il y a 5 mois - enregistré: il y a 5 mois - étoiles: 0 - forks: 0
MettaSurendhar/DataEngineeringProject
Data Engineering project which involves ETL using PostgreSQL and Python
langage: Python - taille: 10,4 Mo - dernière synchronisation: il y a 6 mois - enregistré: il y a environ un an - étoiles: 2 - forks: 0
OtmaneDaoudi/azure-migration-pipeline
ELT pipeline for sales analysis using Azure cloud.
langage: HCL - taille: 569 ko - dernière synchronisation: il y a environ 2 mois - enregistré: il y a 8 mois - étoiles: 0 - forks: 0
dondogecl/cool_data_pipeline
Data pipeline from RDBMS to AWS
langage: Python - taille: 41 ko - dernière synchronisation: il y a 8 mois - enregistré: il y a 8 mois - étoiles: 0 - forks: 0
longNguyen010203/FDE-Course-2024-W4-DBT
💻💛Fundamental Data Engineering Course 2024 Week4 Learn DBT Transform Data with Models, Macro, ELT-Pipeline with Dagster 🌎
langage: Python - taille: 14,6 ko - dernière synchronisation: il y a 24 jours - enregistré: il y a environ un an - étoiles: 5 - forks: 0
longNguyen010203/ECommerce-ELT-Pipeline
🌄📈📉 A Data Engineering Project 🌈 that implements an ELT data pipeline using Dagster, Docker, Dbt, Polars, Snowflake, PostgreSQL. Data from kaggle website 🔥
langage: Python - taille: 6,84 Mo - dernière synchronisation: il y a 3 mois - enregistré: il y a 11 mois - étoiles: 3 - forks: 0
bennyaustin/synapse-dataplatform
A modern data platform implemented on Azure Synapse Analytics using ELT Framework - https://github.com/bennyaustin/elt-framework. Data platform infrastructure provisioned using https://github.com/bennyaustin/iac-synapse-dataplatform
langage: TSQL - taille: 1,79 Mo - dernière synchronisation: il y a 5 mois - enregistré: il y a 9 mois - étoiles: 7 - forks: 6
lksprado/DW-ETL-end_to_end
ELT com python, AWS RDS, dbt-core
langage: HTML - taille: 734 ko - dernière synchronisation: il y a 9 mois - enregistré: il y a 9 mois - étoiles: 0 - forks: 0
elmezianech/Snowflake_dbt_Airflow_ELT
This project is an ELT Pipeline using Dbt (dbt-core) for transformation, Snowflake for data warehousing and Airflow for orchestration.
langage: Python - taille: 13,7 ko - dernière synchronisation: il y a 2 mois - enregistré: il y a 9 mois - étoiles: 0 - forks: 1
nabilraihann/ELT-Pipeline-Bike-Store
This repository contains the implementation of an ELT (Extract, Load, Transform) pipeline for a Bike Store dataset using modern data tools. The pipeline integrates Airbyte for data extraction, dbt for data transformation, Airflow for orchestration, and Snowflake as the data warehouse.
langage: Python - taille: 960 ko - dernière synchronisation: il y a environ un mois - enregistré: il y a environ un mois - étoiles: 0 - forks: 0
p0lyMth/data_sec_ops
ELT pipeline portfolio
langage: Python - taille: 18,6 ko - dernière synchronisation: il y a 10 mois - enregistré: il y a 10 mois - étoiles: 0 - forks: 0
nadyavoynich/Instacart
A deep dive into North American grocery e-commerce behaviour based on Instacart's open dataset. [ELT, EDA, ML clustering]
langage: Jupyter Notebook - taille: 75,3 Mo - dernière synchronisation: il y a 10 mois - enregistré: il y a 10 mois - étoiles: 2 - forks: 0
SalvatoreAmaddio/PipelineWebsite
This a console line application is an Ad-hoc Solution for a client who needed a way of extracting data from their own website and print them onto a spreadsheet.
langage: C# - taille: 263 ko - dernière synchronisation: il y a 2 mois - enregistré: il y a 10 mois - étoiles: 0 - forks: 0
KSwaviman/ETL_with_Airbyte
This project showcases an ELT pipeline that extracts JSON data, loads it into a PostgreSQL database, applies transformations using Python scripts, saves the transformed data in a CSV file, and shares it through a FastAPI endpoint.
langage: Python - taille: 10,7 ko - dernière synchronisation: il y a 10 mois - enregistré: il y a 11 mois - étoiles: 0 - forks: 0
sravanigodavarthi/Automated-ELT-Pipeline-AWS
An Apache Airflow data pipeline is designed to perform ELT operations, utilizing Amazon S3 and Amazon Redshift Serverless.
langage: Python - taille: 48 Mo - dernière synchronisation: il y a environ 5 heures - enregistré: il y a 11 mois - étoiles: 0 - forks: 0
BadreeshShetty/Data-Engineering-ELT-NBA-New-Stats
This project involves fetching and analyzing recent NBA scores, player statistics, and news. Technologies used include AWS S3, EC2, Airflow, Snowflake, DBT, Streamlit, Python, and SQL.
langage: Python - taille: 1,92 Mo - dernière synchronisation: il y a 11 mois - enregistré: il y a 11 mois - étoiles: 0 - forks: 0
kayazay/zomato-restaurant-analytics
An end to end ELT project that uses data from the Zomato Restaurant, an Indian multinational restaurant aggregator and food delivery company. The project extracts data from Kaggle dataset, loads it into Snowflake tables, then is transformed and modelled in dbt Labs.
taille: 55,7 ko - dernière synchronisation: il y a 11 mois - enregistré: il y a 11 mois - étoiles: 1 - forks: 1
agnivchtj/dbt-pipeline-project
Building a data pipeline using DBT and Snowflake to load sample TPCH data and perform basic data modeling techniques, such as building data marts, fact tables, macros and tests.
langage: Python - taille: 33,9 Mo - dernière synchronisation: il y a environ 2 mois - enregistré: il y a 11 mois - étoiles: 0 - forks: 0
oli2v/flight-radar-gcp
FlightRadar ELT pipeline on GCP
langage: HCL - taille: 982 ko - dernière synchronisation: il y a 11 mois - enregistré: il y a 11 mois - étoiles: 0 - forks: 0
Fozan-Talat/divvy-bikeshare-de-project
An end-to-end data pipeline which extracts divvy bikeshare data from web loads it into data lake and datawarehouse transforms it using dbt and finally , a dashboard to visualize the data using looker studio, the pipeline is orchestrated using prefect
langage: Python - taille: 620 ko - dernière synchronisation: il y a 10 mois - enregistré: il y a environ 2 ans - étoiles: 34 - forks: 5
seilylook/custom-elt-project
langage: Python - taille: 14,6 ko - dernière synchronisation: il y a 3 mois - enregistré: il y a environ un an - étoiles: 0 - forks: 0
andressagomes26/adventure_works_analytics
Neste projeto, são realizadas as transformações dos dados brutos da empresa Adventure Works (AW). 🚴♀️
langage: Jupyter Notebook - taille: 22 Mo - dernière synchronisation: il y a environ un an - enregistré: il y a environ un an - étoiles: 0 - forks: 0
jolares/demo-dbt
Example ELT data pipeline project using dbt
langage: Shell - taille: 21,5 ko - dernière synchronisation: il y a environ 2 mois - enregistré: il y a environ un an - étoiles: 0 - forks: 3
arunp77/SQL
SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline
taille: 18,7 Mo - dernière synchronisation: il y a environ un an - enregistré: il y a plus d'un an - étoiles: 2 - forks: 1
johnkdunyo/Postgres-ELT-Data-Pipeline
An ELT data pipeline project utilizing docker and postgress (both source and destination dbs)
langage: Python - taille: 3,91 ko - dernière synchronisation: il y a environ un an - enregistré: il y a plus d'un an - étoiles: 0 - forks: 0
Baitur5/reddit_api_elt
langage: Python - taille: 272 ko - dernière synchronisation: il y a environ un an - enregistré: il y a environ un an - étoiles: 2 - forks: 1
shahinyusifli/dw-credit-risk
A system has been set up for analyzing credit risk, involving a data warehouse and pipeline. The tools used for this solution include Prefect for workflow management, Redshift for the data warehouse, and an S3 bucket for storage.
langage: Python - taille: 1,33 Mo - dernière synchronisation: il y a plus d'un an - enregistré: il y a plus d'un an - étoiles: 0 - forks: 0
judeleonard/e-commerce_activity_tracking
This is an ELT data pipeline setup to track the activities of an e-commerce website based on orders, reviews, deliveries and shipment date. This project utilized technologies like Airflow, AWS RDS-Postgres, Python etc.
langage: Python - taille: 596 ko - dernière synchronisation: il y a plus d'un an - enregistré: il y a presque 2 ans - étoiles: 0 - forks: 0
vishu-tyagi/BigQuery-ELT
BigQuery data pipeline with dbt, Spark, Docker, Airflow, Terraform, GCP
langage: Python - taille: 1,19 Mo - dernière synchronisation: il y a plus d'un an - enregistré: il y a plus de 2 ans - étoiles: 0 - forks: 0
jgrove90/rick-and-morty-deltalake
🔫 🍺A data engineering project showcasing an ELT pipeline using modern technologies such as Delta-rs, and Apache Airflow.
langage: Python - taille: 2,45 Mo - dernière synchronisation: il y a presque 2 ans - enregistré: il y a presque 2 ans - étoiles: 0 - forks: 0
nauqh/Echodb-app
🏬Tiny elt system
langage: Python - taille: 262 ko - dernière synchronisation: il y a presque 2 ans - enregistré: il y a presque 2 ans - étoiles: 1 - forks: 0
jgrove90/ufo-deltalake
🛸 This project showcases an Extract, Load, Transform (ELT) pipeline built with Python, Apache Spark, Delta Lake, and Docker. The objective of the project is to scrape UFO sighting data from NUFORC and process it through the Medallion architecture to create a star schema in the Gold layer that is ready for analysis.
langage: Python - taille: 1,54 Mo - dernière synchronisation: il y a presque 2 ans - enregistré: il y a presque 2 ans - étoiles: 0 - forks: 0
hdt94/dtc-de-project
Data engineering project for TLC taxi Parquet data following an ELT model (extraction, load, transform)
langage: Python - taille: 268 ko - dernière synchronisation: il y a presque 2 ans - enregistré: il y a presque 2 ans - étoiles: 0 - forks: 1
Mg30/pydwt
Modeling tool like DBT to use SQL Alchemy core with a DataFrame interface like
langage: Python - taille: 196 ko - dernière synchronisation: il y a environ 2 ans - enregistré: il y a environ 2 ans - étoiles: 3 - forks: 0
jackmulligan-ire/ppr-pipeline
Irish Property Price Register transformed into a data warehouse via an EtLT pipeline.
langage: TypeScript - taille: 22,7 Mo - dernière synchronisation: il y a environ 2 ans - enregistré: il y a environ 2 ans - étoiles: 2 - forks: 0
Suprame4/Data_Engineering_Projects
Data engineering projects
langage: Jupyter Notebook - taille: 136 Mo - dernière synchronisation: il y a environ 2 ans - enregistré: il y a plus de 2 ans - étoiles: 1 - forks: 0
teddyk251/traffic-data-ELT-pipeline
An ELT pipeline built for the pNEUMA open dataset of naturalistic trajectories of half a million vehicles collected by a swarm of drones in a congested downtown area of Athens, Greece.
langage: Python - taille: 20 Mo - dernière synchronisation: il y a environ 2 ans - enregistré: il y a presque 3 ans - étoiles: 0 - forks: 0
