airflow-docker | Topic | Ecosyste.ms: Repos

coder2j/airflow-docker

Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)

Language: Python - Size: 342 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 205 - Forks: 191

Wittline/uber-expenses-tracking

The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

Language: Jupyter Notebook - Size: 28.2 MB - Last synced at: 2 months ago - Pushed at: almost 3 years ago - Stars: 121 - Forks: 36

data-burst/airflow-git-sync

Sync DAG changes from Git to Airflow

Size: 126 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 57 - Forks: 11

sergio11/lyric_wave_architecture

🎵 LyricWave – AI Music Composer (Proof of Concept) 🎶 A personal project exploring automatic generation of unique MP4 songs. LyricWave blends lyrics with AI-generated melodies and synthetic vocals to experiment with new forms of musical expression. A creative testbed to push your ideas into sound. 🚀🎧

Language: Python - Size: 29 MB - Last synced at: 7 days ago - Pushed at: 3 months ago - Stars: 34 - Forks: 6

abhioncbr/docker-airflow

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

Language: Python - Size: 1.04 MB - Last synced at: about 1 month ago - Pushed at: about 6 years ago - Stars: 32 - Forks: 5

mikestack15/orangutan-stem

An open-source project dedicated to constructing robust data pipelines and scalable software infrastructure. We leverage industry-standard tools favored by developers to enhance efficiency and reliability. Uniquely, these pipelines are field-tested on farms across Sumatra, Indonesia, ensuring real-world applicability and resilience.

Language: Python - Size: 151 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 29 - Forks: 2

DatacollectorVN/Airflow-Tutorial

My self-learning about Apache Airflow

Language: Python - Size: 18.6 KB - Last synced at: over 2 years ago - Pushed at: almost 3 years ago - Stars: 29 - Forks: 12

mikeroyal/Apache-Airflow-Guide

Apache Airflow Guide

Language: Python - Size: 279 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 28 - Forks: 12

anilkulkarni87/airflow-docker

This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

Language: Python - Size: 106 KB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 22 - Forks: 10

jksinghpro/docker-airflow

Docker for airflow with mysql as backend

Language: Shell - Size: 9.77 KB - Last synced at: over 2 years ago - Pushed at: over 6 years ago - Stars: 13 - Forks: 8

rahul765/Machine-Learning-Pipelines

From data gathering to model deployment. Complete ML pipeline using Docker, Airflow and Python.

Language: Python - Size: 335 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 12 - Forks: 3

vjgpt/twitter-pipeline

In this project, you will be building a Twitter Scheduler using Apache Airflow on Docker.

Language: Python - Size: 26.4 KB - Last synced at: 2 months ago - Pushed at: almost 5 years ago - Stars: 10 - Forks: 7

knowsuchagency/docker-compose-airflow

An airflow deployment configuration with sane defaults

Language: Python - Size: 47.9 KB - Last synced at: 6 days ago - Pushed at: about 6 years ago - Stars: 10 - Forks: 2

ericdaat/data-stack

A starting point for a data stack using Python, Apache Airflow and Metabase.

Language: Python - Size: 53.7 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 9 - Forks: 5

mpavanetti/airflow

This set of code and instructions has the porpouse to instanciate a compiled environment with set of docker images like airflow webserver, airflow scheduler, postgresql, pyspark, Data Pipeline consuming data from weather api , processing with pyspark and storing in postgresql

Language: PHP - Size: 1.62 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 8 - Forks: 3

okzapradhana/etl-flatfile-airflow

Building Data Warehouse on BigQuery which takes flat file as the data sources with Airflow as the Orchestrator

Language: Python - Size: 34.9 MB - Last synced at: over 2 years ago - Pushed at: about 4 years ago - Stars: 8 - Forks: 6

pacuna/airflow-docker

Run Apache Airflow using Docker containers

Language: Python - Size: 6.84 KB - Last synced at: 2 months ago - Pushed at: almost 7 years ago - Stars: 8 - Forks: 6

judeleonard/Prescriber-ETL-data-pipeline

An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS application using Apache Airflow as an orchestration tool and various data warehouse technologies and finally using Apache Superset to connect to DWH for generating BI dashboards for weekly reports

Language: Python - Size: 7.42 MB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 7 - Forks: 1

resdevd/docker-compose-airflow

Quick Docker Compose deployment for Apache Airflow version 2.x. Ideal for local development

Language: Shell - Size: 3.91 KB - Last synced at: over 2 years ago - Pushed at: almost 3 years ago - Stars: 7 - Forks: 6

AhmetFurkanDEMIR/airflow-spark-kafka-example

Airflow, Spark and Kafka example

Language: Dockerfile - Size: 532 KB - Last synced at: 2 months ago - Pushed at: over 1 year ago - Stars: 6 - Forks: 0

Data-Bishop/Team5-BuildItAll-Data-Platform

This repository contains the codebase for the BuildItAll Big Data Processing Platform, a case study project designed to manage large daily data for a hypothetical Belgian client.

Language: HCL - Size: 180 KB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 5 - Forks: 0

Aiscalate/aiscalator

Tools to streamline Jupyter Notebook Prototypes into robust Data Products

Language: Python - Size: 1.53 MB - Last synced at: 26 days ago - Pushed at: over 2 years ago - Stars: 5 - Forks: 2

DunnBC22/Data_Engineering_Projects

This repository includes data engineering projects using Apache Airflow. I hope to add more projects using different technologies soon!

Language: TSQL - Size: 62.7 MB - Last synced at: 17 days ago - Pushed at: 17 days ago - Stars: 4 - Forks: 0

rushabh2390/SharesETL

Shares ETL to scrape historical data of one shares and import in metabase

Language: Python - Size: 70.3 KB - Last synced at: almost 2 years ago - Pushed at: over 2 years ago - Stars: 4 - Forks: 0

sreekesh93/airflow_beginners

to start airflow in local with basic setup

Language: Python - Size: 2.93 KB - Last synced at: over 1 year ago - Pushed at: over 3 years ago - Stars: 4 - Forks: 0

gustschaefer/Twitter-Batch-ETL

ETL que extraí diariamente os trending topics do Twitter em diversos países, realiza transformações com PySpark e envia os dados para o Amazon S3 utilizando Apache Airflow como orquestrador.

Language: Python - Size: 1.03 MB - Last synced at: 6 months ago - Pushed at: about 4 years ago - Stars: 4 - Forks: 0

saianupkumarp/airflow-docker

Airflow docker boiler plate with LocalExecutor, CeleryExecutor. #TODO Kubernetes

Language: Python - Size: 33.2 KB - Last synced at: over 2 years ago - Pushed at: over 5 years ago - Stars: 4 - Forks: 1

Pirate-Emperor/BigData-Pipeline

BigData Pipeline is a local testing environment for experimenting with various storage solutions (RDB, HDFS), query engines (Trino), schedulers (Airflow), and ETL/ELT tools (DBT). It supports MySQL, Hadoop, Hive, Kudu, and more.

Language: Dockerfile - Size: 7.95 MB - Last synced at: 2 months ago - Pushed at: 8 months ago - Stars: 3 - Forks: 0

abhishekbhakat/Turbine

Simple python script for easy local airflow deployment with docker. Packed with additional components. Will be adding more going forward.

Language: Python - Size: 113 KB - Last synced at: 7 days ago - Pushed at: 8 months ago - Stars: 3 - Forks: 0

Chinaskidev/ETL-Clima-ElSalvador

MLOps, haciendo un ETL sencillo usando Docker y Airflow y Google Cloud

Language: Python - Size: 50.8 KB - Last synced at: 2 months ago - Pushed at: about 1 year ago - Stars: 3 - Forks: 1

aimlnerd/data-pipelines-with-airflow

Airflow DAG tutorial with docker compose local setup

Language: Python - Size: 16.6 KB - Last synced at: over 1 year ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 0

vcwild/airflow-init

Apache Airflow quick setup orchestrated with Docker Compose

Language: Python - Size: 317 KB - Last synced at: 7 days ago - Pushed at: almost 4 years ago - Stars: 3 - Forks: 0

slatawa/Airflow-Monitoring-ElasticSearch-LogStash-Kibana

In this project we look to set up Airflow monitoring using ElasticSearch-LogStash-Kibana (ELK stack). We will set up the stack using docker images.

Language: Shell - Size: 3.31 MB - Last synced at: 2 months ago - Pushed at: almost 4 years ago - Stars: 3 - Forks: 3

chattarajoy/airflow-kube-helm Fork of BrechtDeVlieger/airflow-kube-helm

Airflow helm chart (KubernetesExecutor)

Language: Shell - Size: 47.9 KB - Last synced at: over 2 years ago - Pushed at: over 5 years ago - Stars: 3 - Forks: 5

Jiansiyu/airflow_docker_compose

This is a simplified version of Apache Airflow manage with docker compose v2, It support running docker image inside DAG file

Language: Python - Size: 240 KB - Last synced at: 11 days ago - Pushed at: 11 days ago - Stars: 2 - Forks: 0

judeleonard/Data-Engineering

This is a repo to keep track of all my projects related to data engineering

Language: Python - Size: 24 MB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 2 - Forks: 2

ALEXUSCR-27/Amazon-Data-ETL

ETL pipeline for Amazon product sales data, using Apache Airflow for data orchestration and Supabase for storage, by containerizing the environment with Docker, the setup is scalable and easily deployable, supporting data-driven decision-making.

Language: Python - Size: 4.49 MB - Last synced at: 2 months ago - Pushed at: 8 months ago - Stars: 2 - Forks: 1

kaushil24/llm-finetune-pipeline

Airflow pipeline to finetune LLM on Kubernetes

Language: Python - Size: 5.8 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 2 - Forks: 0

frankcash/airflow_compose

A simple docker-compose ready repository for using Apache Airflow.

Language: Python - Size: 57.6 KB - Last synced at: 3 months ago - Pushed at: about 2 years ago - Stars: 2 - Forks: 0

vikrantdeshpande09876/Credit_Card_Fraud_Detection

Deploying Classification Model on GCP for detecting anomalous behavior in credit-card transactions. A crucial aspect for credit-card companies to recognize fraudulent activity and not charge customers for items they didn't purchase. A subtle line needs to be drawn and my main focus here is to reduce blocked-transactions of legitimate customers.

Language: Jupyter Notebook - Size: 41.7 MB - Last synced at: 2 days ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 1

dataopstix/modelt

Modelt(mow·delt) is a modern data integration solution that connects data to data for advanced analytics.

Language: Shell - Size: 463 KB - Last synced at: 3 months ago - Pushed at: about 3 years ago - Stars: 2 - Forks: 1

thiagoheron1/study_airflow

Repositório criado para estudar Airflow baseado nos cursos do Marc Lamberti

Language: Python - Size: 1.03 MB - Last synced at: over 2 years ago - Pushed at: almost 4 years ago - Stars: 2 - Forks: 0

netflame/docker-airflow

Dockerized Airflow based on Alpine

Language: Python - Size: 6.84 KB - Last synced at: over 2 years ago - Pushed at: about 6 years ago - Stars: 2 - Forks: 0

micheldpd24/cust_review_analysis

End-to-End Customer Review Analysis Pipeline : Scrape, process, and analyze customer reviews from Trustpilot using Apache Airflow and Python | Interactive Dash Dashboard : Visualize sentiment, trends, and textual insights with a dynamic, Dockerized Dash app.

Language: Jupyter Notebook - Size: 2.78 MB - Last synced at: 6 days ago - Pushed at: 11 days ago - Stars: 1 - Forks: 0

yuhexiong/airflow-dag-kafka-flink-doris-python

Language: Python - Size: 43 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0

choval/devairflow

Simple local development airflow image

Size: 111 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 1 - Forks: 0

ByteMeDirk/apache-airflow-data-engineer

Apache Airflow For Data Engineers Tutorial

Language: Python - Size: 6.69 MB - Last synced at: 4 months ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 1

stevehoober254/ETL_Data_Pipeline_For_Retail_Store

ETL (Extract, Transform, Load) pipeline to integrate sales data from various sources into a central data warehouse

Language: Python - Size: 5.86 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

Serge-Ntumba/Daily_automated_ETL

This project focuses on the seamless integration of data from a CSV file into a PostgreSQL database using Python. Additionally, Airflow is employed to perform a daily extraction of this data, transferring it to an S3 Bucket for further analysis and storage.

Language: Python - Size: 228 KB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

Serge-Ntumba/stream_data_with_kafka_docker_airflow_spark

A complete data pipeline, from data extraction to storage, using a combination of tools for specific purposes: Python for data retrieval from API, Airflow for scheduling task, Kafka for data streaming, Spark for data processing, and Cassandra for data storage.

Language: Python - Size: 298 KB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

CanaanGM/databases-infrastructure

on demand databases deployment, varuios kinds, adding more as i use them!

Language: Python - Size: 4.02 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

Gakhramanzode/Airflow

Bash-скрипт по установке Airflow

Language: Shell - Size: 14.6 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 1 - Forks: 1

anastasiia-p/airflow-ml

Airflow Pipeline for Machine Learning

Language: Python - Size: 9.77 KB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 0

tier940/docker-repo

Language: Python - Size: 115 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0

seanleong339/Twitdockair

An ETL(Extract, Transform, Load) Data pipeline project, using Apache Airflow on Docker

Language: Python - Size: 72.3 KB - Last synced at: over 2 years ago - Pushed at: almost 3 years ago - Stars: 1 - Forks: 0

AakashDeorukhkar/Airflow_NYC_Taxi_Data

Ingesting NYC Yellow Taxi Data to GCP using Airflow running on Docker

Language: Python - Size: 4.08 MB - Last synced at: 11 months ago - Pushed at: almost 3 years ago - Stars: 1 - Forks: 0

julientoucoula17/apache_airflow-with-Docker

Apache Airflow installation with Docker 🌬️

Language: Python - Size: 10.7 KB - Last synced at: 24 days ago - Pushed at: about 3 years ago - Stars: 1 - Forks: 0

finloop/grafana-airflow-etl

End to end data pipeline with Apache Airflow and Grafana

Language: Python - Size: 41.9 MB - Last synced at: 6 days ago - Pushed at: about 3 years ago - Stars: 1 - Forks: 2

ruanbekker/docker-apache-airflow

Apache Airflow using Docker Compose

Language: Python - Size: 12.7 KB - Last synced at: 2 days ago - Pushed at: almost 4 years ago - Stars: 1 - Forks: 0

BruceDone/airflow-docker

Docker Apache Airflow

Language: Shell - Size: 9.77 KB - Last synced at: 3 months ago - Pushed at: almost 8 years ago - Stars: 1 - Forks: 1

ethanwebber123/Airflow-ETL-ELT

Airflow-ETL-ELT is a robust data pipeline tool that enables efficient extraction, transformation, loading, and orchestration of data workflows. It offers a scalable and customizable solution for managing complex ETL and ELT processes with ease.

Size: 1000 Bytes - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 0 - Forks: 0

navs-svan/nba-dashboard

PowerBI Dashboard using scraped box score data of the NBA Season 2024-25

Language: Python - Size: 12 MB - Last synced at: 13 days ago - Pushed at: 14 days ago - Stars: 0 - Forks: 0

opt-nc/airflow-extended-image

Image Docker Airflow personnalisée avec des providers additionnels pour étendre les fonctionnalités de base et simplifier le déploiement.

Language: Dockerfile - Size: 12.7 KB - Last synced at: 27 days ago - Pushed at: 27 days ago - Stars: 0 - Forks: 0

rahulsamant37/FraudDetection

Build Realtime Fraud Detection AI from Scratch - End to End Machine Learning Project

Language: Shell - Size: 10.7 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

aakashsyadav1999/Covid-Data-Engineering-Project

This is repo which will guide you about the data engineering project which will Extract, Load and then Transform into the database.

Language: Python - Size: 444 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

enricogoerlitz/explore-airflow

Apache Airflow project for orchestrating ETL workflows in a data warehouse environment. Implements a medallion architecture (Bronze, Silver, Gold layers) with Postgres integration for scalable and modular data processing pipelines.

Language: Python - Size: 1.33 MB - Last synced at: 2 months ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

micheldpd24/mlops_air_msr

MLOps Pipeline for Music Recommendation - Spotify playlist continuation

Language: Python - Size: 25.5 MB - Last synced at: 6 days ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

Ramshankar07/Stock-HMM

Streamlit application designed to forecast stock prices using a Hidden Markov Model (HMM) based on historical stock price data. The application enables users to view 1-day and multi-day forecasts for several major tech stocks, including: Apple Inc. (AAPL) Tesla Inc. (TSLA) NVIDIA Corporation (NVDA)

Language: Python - Size: 47.9 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

nasirkadri2601/Live_Cricket_Data_Pipeline

This project captures live cricket data in raw JSON format, cleans and transforms it, and stores it in a centralized data warehouse. The data is then used for analysis, including match outcome predictions, player performance, and team strategy insights, enabling data-driven decisions.

Language: Jupyter Notebook - Size: 25.6 MB - Last synced at: 3 months ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0

RayanGAtech/HR-Roster-Change-Detection-Pipeline-with-Apache-Airflow-and-PostgreSQL

Automated data pipeline tailored for HR systems to process and analyze roster data. The pipeline leverages Apache Airflow to ingest and transform CSV files containing roster information, and PostgreSQL to store, deduplicate, and extract unique data points.

Language: Python - Size: 6.84 KB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0

wlopezm-unal/reddit_project_airflow_aws

This project focuses on implementing an ETL pipeline using Apache Airflow to efficiently extract data from Reddit, transform it as needed, and load it into an AWS S3 bucket. The use of Airflow allows for robust orchestration of the data workflow, ensuring that each step of the ETL process is executed in a reliable and repeatable manner.

Language: Python - Size: 38.1 KB - Last synced at: 2 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

TuanAnh30/ETL_With_AirFlow

Simulate loading log data in JSON files into a database using an ETL process with Airflow to automate execution.

Language: Python - Size: 25.4 MB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

jasontrandev/ETL-Data-Pipeline-Airflow-Project1

Build an ETL Data Pipelines that uses Airflow DAGs to extract employees’ data from PostgreSQL Schemas, load it into AWS Data Lake, then transform it with Python script, and finally load it into Snowflake Data warehouse using SCD type 2.

Language: Python - Size: 3.74 MB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

joshua-dada-mayowa/Data-Engineer-ETL-Project

ETL Project using Airflow, DBT, Bigquery,Postgres,Docker and Docker Compose

Language: Python - Size: 41.3 MB - Last synced at: 10 months ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

mashalk1/airflow-using-docker-compose

A configuration file to run Apache Airflow 2.0 via Docker. The attached file is a Docker Compose file which is much easier to run. All you have to do is install the file, go to that directory and run "docker-compose up" and the rest will done by docker itself.

Size: 3.91 KB - Last synced at: 18 days ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

igorlangoni/online_retail_data_pipeline

An end-to-end pipeline that ingests raw data from CSV files through Airflow DAGS into BigQuery. From there, it uses dbt to normalize and clean the data and afterwards to make the transformations and come up with relevan reports.

Language: Python - Size: 15.4 MB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

RafaelQSantos-RQS/windturbine-airflow-project

Este projeto foi desenvolvido no curso de Airflow da Udemy, lecionado pelo professor Fernando Amaral.

Language: Python - Size: 186 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

djamier/docker-airflow

Orchestrate data pipeline using airflow

Language: Python - Size: 43.9 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

aohus/etl-pipeline-with-modeling

Docker를 사용하여 Hadoop 생태계의 구성 요소와 기타 필수 서비스를 컨테이너화하여 강력한 데이터 엔지니어링 환경을 설정하는 방법을 보여줍니다. 설정에는 Hadoop (HDFS, YARN), Apache Hive, PostgreSQL 및 Apache Airflow가 포함되며, 이들 모두가 원활하게 작동하도록 구성되어 있습니다.

Language: Shell - Size: 20.4 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

shiv-rna/Airflow-Basics

Welcome to my Apache Airflow learning journey repository! 🚀 This repository serves as a comprehensive documentation of my exploration and understanding of Apache Airflow, an open-source platform for orchestrating complex workflows.

Language: Python - Size: 65.8 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

AlexanderM-T/AirFlow-Training

En este repositorio se encuentran algunos dags realizados siguiendo el entrenamiento de data engineer con el fin de aprender y practicar airflow

Language: Python - Size: 18.6 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

dorokhin/apache_airflow_docker

Apache airflow packed in docker compose

Language: Python - Size: 6.84 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

IanDanielM/BuyRentScrap-LuxAirflow

Automating Data Scrapers With Python and Airflow

Language: Python - Size: 5.86 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

Sheitak/datalake-jljq

Data Lake project for ingest and transform financial data and dashboard BI proposal

Language: Python - Size: 41 KB - Last synced at: 9 days ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

Booss3my/Datascraper

Automated Indeed Job Offer Scraper: Airflow Orchestrated and Scheduled, Data Loaded into PostgreSQL Database

Language: Python - Size: 1.45 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

martinsejas/FlightPricePrediction

An end-to-end MLOps Pipeline, from data ingestion to inference. Powered by Azure, FastAPI, Apache Airflow, Streamlit and Sklearn.

Language: Jupyter Notebook - Size: 3.87 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

SoukainaBoufousse/ETL_Sales_Data

This repository demonstrates a data engineering ETL (Extract, Transform, Load) pipeline for processing sales data. It utilizes Docker and Apache Airflow for orchestration and PostgreSQL for data storage. The pipeline extracts, transforms, and loads sales data.

Language: Python - Size: 385 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

niyotham/Data-engineer-end-to-end-project-airflow-dbt-soda-bigquery

An end to end data engineering project for loading data into bigquery with airflow and perform transformations using dbt and data quality check with soday

Language: Python - Size: 7.03 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

ndangalasinh/Toronto-Apartment-Evaluation-Analysis

Data Engineering End-to-End Project

Language: Python - Size: 2.87 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

prasadanilmore/ETL-JustEat-TestCase

This documentation provides an overview of the tasks completed in this project. The project comprises three key tasks: ETL (Extract, Transform, Load), API development, and Data Orchestration using Airflow. Each task is detailed below along with explanations of design choices and considerations.

Language: Python - Size: 79.1 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0