GitHub | Ecosyste.ms: Repos

adrianmarino/thesis-paper

Collaborative and hybrid recommendation systems

Language: Jupyter Notebook - Size: 353 MB - Last synced: 2 days ago - Pushed: 2 days ago - Stars: 0 - Forks: 1

tulibraries/funcake_dags

Airflow DAGs for PA Digital aggregation processes

Language: Python - Size: 1.75 MB - Last synced: 2 days ago - Pushed: 3 days ago - Stars: 4 - Forks: 2

tulibraries/manifold_airflow_dags

Airflow DAGs for the Manifold (TUL Website) application

Language: Python - Size: 1.54 MB - Last synced: 2 days ago - Pushed: 3 days ago - Stars: 2 - Forks: 0

xennen/DataEngineerYP

Data Engineer projects

Language: Python - Size: 513 KB - Last synced: 5 days ago - Pushed: 6 days ago - Stars: 0 - Forks: 0

tulibraries/cob_datapipeline

Airflow Data Processing Pipeline for TUL Catalog on Blacklight Data

Language: Python - Size: 2.33 MB - Last synced: 7 days ago - Pushed: 7 days ago - Stars: 6 - Forks: 0

matsudan/airflow-dag-examples

Apache Airflow DAG examples

Language: Python - Size: 96.7 KB - Last synced: 8 days ago - Pushed: 8 days ago - Stars: 1 - Forks: 1

rohithlanka/weatherdatapipeline

A project leveraging Python scripts to fetch weather data from an API, transformed it using DBT in Snowflake, and orchestrated the workflow with Apache Airflow for seamless data integration into reporting tool, ensuring streamlined data-driven insights.(reporting tool- work in progress)

Language: Python - Size: 11.7 KB - Last synced: 13 days ago - Pushed: 13 days ago - Stars: 0 - Forks: 0

Azazel0203/ocr_captcha

This project creates a basic web service for solving image-based CAPTCHAs. Using the Flask framework, it allows users to upload CAPTCHA images and employs an Optical Character Recognition (OCR) pipeline to extract the embedded text.

Language: Python - Size: 41.5 MB - Last synced: 12 days ago - Pushed: about 2 months ago - Stars: 6 - Forks: 0

AnthonyByansi/Airflow-Data-Pipeline-Automation

Automate your data pipelines using Apache Airflow with this ready-to-use DAG for data integration, ETL and workflow automation.

Language: Python - Size: 15.6 KB - Last synced: 13 days ago - Pushed: 13 days ago - Stars: 7 - Forks: 0

data-burst/airflow-git-sync

Sync DAG changes from Git to Airflow

Size: 118 KB - Last synced: 12 days ago - Pushed: 13 days ago - Stars: 33 - Forks: 6

Chinaskidev/ETL-Clima-ElSalvador

MLOps, haciendo un ETL sencillo usando Docker y Airflow y Google Cloud

Language: Python - Size: 50.8 KB - Last synced: 14 days ago - Pushed: 14 days ago - Stars: 0 - Forks: 0

anilkulkarni87/airflow-docker

This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

Language: Python - Size: 106 KB - Last synced: 13 days ago - Pushed: 3 months ago - Stars: 22 - Forks: 10

essraahmed/Data-Pipeline-with-Airflow

Data Pipeline with Apache Airflow

Language: Python - Size: 444 KB - Last synced: 15 days ago - Pushed: over 1 year ago - Stars: 1 - Forks: 1

DunnBC22/Data_Engineering_Projects

This repository includes data engineering projects using Apache Airflow. I hope to add more projects using different technologies soon!

Language: Python - Size: 15.1 MB - Last synced: 16 days ago - Pushed: 17 days ago - Stars: 4 - Forks: 0

arjunan-k/Twitter_Pipeline

Tweets extractor using Twitter API by Tweepy library. Created the workflow in Apache Airflow with the help of AWS EC2 & S3 bucket.

Language: Python - Size: 865 KB - Last synced: 19 days ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

nick-roberson/comp-blocks

Computation as Composable Blocks, can be extended to run on remote pipeline frameworks if necessary.

Language: Python - Size: 2.96 MB - Last synced: 19 days ago - Pushed: 19 days ago - Stars: 1 - Forks: 0

akhilkn8/speech-emotion-detection-GCP

This repo contains the codes to perform Speech Emotion Recognition (SER) from 4 popular datasets. This projects aims to do a complete end-to-end MLOps orchestration using Gogle Cloud Platform's Cloud composer

Language: Python - Size: 3.62 MB - Last synced: 22 days ago - Pushed: 22 days ago - Stars: 0 - Forks: 0

gestaogovbr/Ro-dou

Gerador de DAGs no Airflow para fazer clipping do Diário Oficial da União.

Language: Python - Size: 484 KB - Last synced: 24 days ago - Pushed: 25 days ago - Stars: 62 - Forks: 12

polarbeargo/Data-Pipelines-with-Airflow

Language: Python - Size: 1020 KB - Last synced: 25 days ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

casassg/corrent

Corrent: Experimental Airflow functional DAG API

Language: Python - Size: 187 KB - Last synced: 28 days ago - Pushed: over 4 years ago - Stars: 4 - Forks: 0

tulibraries/tulflow

TU Libraries Python Library for functions used in indexing ETL, particularly for Airflow

Language: Python - Size: 2.26 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 3 - Forks: 1

ShamilNur/data-mining-kfu-2021

Homework assignments from courses in Data Mining, KFU, 4th semester, 2021.

Language: Jupyter Notebook - Size: 1.24 MB - Last synced: about 1 month ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

BaliDataMan/airflow-Tut

Airflow Tutorial to learn practical implementation with basics Concepts, Architecture, Working & Configuration practises.

Language: Shell - Size: 425 KB - Last synced: about 1 month ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

status-im/airflow-dags

Status BI python DAGs for Airflow

Language: Python - Size: 99.6 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 2

nbdevs/RDWA

Real-time Data Warehousing with Airflow: An events based microservices pipeline.

Size: 13.7 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

tangphucnhan/Deploy-Airflow-server-with-docker

Deploy Apache Airflow server in docker. This docker will run three main services: Postgresql, Airflow weberver and Airflow scheduler.

Language: Python - Size: 5.86 KB - Last synced: about 1 month ago - Pushed: about 2 months ago - Stars: 0 - Forks: 0

jesslossa/airflow-docker-localexecutor

Implements Airflow through docker on localhost (LocalExecutor), using docker-desktop.

Language: Python - Size: 171 KB - Last synced: about 1 month ago - Pushed: 10 months ago - Stars: 0 - Forks: 0

coderjolly/spotify-api-data-analysis

The project leverages Apache Airflow for automating Spotify API data analysis, focusing on user activity. Extracting, transforming, and loading data efficiently, it provides insights via PowerBI dashboards.

Language: Python - Size: 4.33 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

Julianadev/DAG_Operators

Automating pipelines using airflow operators

Language: Python - Size: 5.86 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

MatheusSC017/Message-Manager

This project aims to create a messaging system to manage messages sent via email and WhatsApp, working on them in an asynchronous and scalable way. It also has the functionality to save logs and send personalized messages for each type of subject.

Language: Python - Size: 70.3 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 2 - Forks: 0

yosh0555/airflow_with_mysql_and_snowflake

This is a small project where I used Airflow DAG program to create table in MySQL database and insert values into it. I also created a pipeline using spark to fetch data from MySQL database and process it and store the processed data into Snowflake data warehouse, I used Apache Airflow to automate this process

Language: Python - Size: 1000 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 0 - Forks: 0

Shreyjain203/Spotify-recommendation-system

Built a Spotify recommendation system: fetched data from Spotify API, stored it on Google Cloud Storage, orchestrated with Airflow (Composer) to load into MongoDB Atlas, and developed a recommendation engine based on the data.

Language: Python - Size: 3.51 MB - Last synced: 30 days ago - Pushed: 2 months ago - Stars: 0 - Forks: 0

khaz-dev/TollData_ETL_AirflowBash

Build an Toll data simulation ETL Pipeline using Airflow (Python and Bash)

Language: Python - Size: 899 KB - Last synced: about 2 months ago - Pushed: 4 months ago - Stars: 1 - Forks: 0

mikeroyal/Apache-Airflow-Guide

Apache Airflow Guide

Language: Python - Size: 279 KB - Last synced: 1 day ago - Pushed: 13 days ago - Stars: 17 - Forks: 8

stevehoober254/ETL_Data_Pipeline_For_Retail_Store

ETL (Extract, Transform, Load) pipeline to integrate sales data from various sources into a central data warehouse

Language: Python - Size: 5.86 KB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 1 - Forks: 0

WALIDAADI/ETL_using_Airflow

This project presents a robust data pipeline using Apache Airflow for orchestration, Apache Kafka for real-time data streaming, and MongoDB for data storage. It automates the process of web scraping to collect large companies' data, transforms and processes this data, and then stores it efficiently.

Size: 69.3 KB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 0 - Forks: 0

chandulal/airflow-testing

Airflow Unit Tests and Integration Tests

Language: Python - Size: 465 KB - Last synced: about 2 months ago - Pushed: over 1 year ago - Stars: 249 - Forks: 45

JBris/time-series-airflow-kafka-spark

A simple demonstration of an Airflow-Kafka-Spark (AKS) stack for online time series forecasting.

Language: Python - Size: 699 KB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 1 - Forks: 0

AhmetFurkanDEMIR/airflow-spark-kafka-example

Airflow, Spark and Kafka example

Language: Dockerfile - Size: 532 KB - Last synced: 13 days ago - Pushed: 6 months ago - Stars: 4 - Forks: 0

visheshgupta-BA/MLOps---Airflow-Docker

Language: Python - Size: 1.87 MB - Last synced: about 1 month ago - Pushed: 2 months ago - Stars: 0 - Forks: 0

shiv-rna/Airflow-Basics

Welcome to my Apache Airflow learning journey repository! 🚀 This repository serves as a comprehensive documentation of my exploration and understanding of Apache Airflow, an open-source platform for orchestrating complex workflows.

Language: Python - Size: 65.8 MB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

Melcade/zelda-api

Utilize Airflow and JupyterLab to ETL data from the Zelda API, transform it, and load it into SQLite for SQL analysis in JupyterLab.

Language: Jupyter Notebook - Size: 521 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

astronomer/astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows

Language: Python - Size: 11.1 MB - Last synced: 17 days ago - Pushed: about 1 month ago - Stars: 20 - Forks: 10

AlexanderM-T/AirFlow-Training

En este repositorio se encuentran algunos dags realizados siguiendo el entrenamiento de data engineer con el fin de aprender y practicar airflow

Language: Python - Size: 18.6 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

ohadmata/simple-dag-editor

Zero configuration Airflow plugin that let you manage your DAG files.

Language: Python - Size: 394 KB - Last synced: 3 months ago - Pushed: over 2 years ago - Stars: 35 - Forks: 3

skalskibukowa/Airflow-python-practice

Project to practice airflow and python - ETL

Language: Python - Size: 15.7 MB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

arunp77/SQL

SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

Size: 18.7 MB - Last synced: 18 days ago - Pushed: 7 months ago - Stars: 2 - Forks: 1

dronectl/windfarm

Micro-service deployments for aggregation and analysis of propulsion tests.

Language: Jsonnet - Size: 220 KB - Last synced: 4 months ago - Pushed: about 1 year ago - Stars: 0 - Forks: 0

akhich551995/Data-streaming-project-airflow-Kafka-spark-t-cassandra-docker

building a real-time data streaming pipeline, covering each phase from data ingestion to processing and finally storage. We'll utilize a powerful stack of tools and technologies, including Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra—all neatly containerized using Docker.

Language: Python - Size: 1.95 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

igorlangoni/final_project_data_eng_makers Fork of jdench1989/data_eng_final_project

Final project for the Makers Academy Data Engineering Bootcamp! In this amazing, complex group project we had to analyse a massive dataset and extract insightful data that could be used to improve education world-wide!

Language: Python - Size: 125 MB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 1

Susanhuynh/etl-API-data-to-AWS-S3-using-Airflow

This project focuses on utilizing Apache Airflow to orchestrate an ETL (Extract, Transform, Load) process using data from the Stack Overflow API. The primary objective is to determine the most prominent tags on Stack Overflow for the current month.

Language: Python - Size: 1.61 MB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

benkeyben/10alytics_air_realtor

10alytics_air_realtor ,a dynamic repository, hosts an AWS-driven data pipeline. Utilizing Apache Airflow, AWS S3, and EC2, it performs efficient ETL operations, extracting comprehensive real estate data from the Realty Mole Property API via RapidAPI. This tool empowers real estate professionals with timely insights for strategic decision-making.

Language: Jupyter Notebook - Size: 656 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

borgettas/apache-airflow

This project is destinate a study for create a environment Apache Airflow

Language: Makefile - Size: 8.79 KB - Last synced: 4 months ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

igorlangoni/online_retail_data_pipeline

An end-to-end pipeline that ingests raw data from CSV files through Airflow DAGS into BigQuery. From there, it uses dbt to normalize and clean the data and afterwards to make the transformations and come up with relevan reports.

Language: Python - Size: 15.4 MB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

sergio11/lyric_wave_architecture

🎵 LyricWave - Your AI Music Composer 🎶 Compose Unique MP4 Songs Effortlessly! LyricWave uses AI to create personalized music by harmonizing lyrics with captivating melodies and synthetic vocals. Unleash your musical creativity today! 🚀🎶

Language: Python - Size: 29 MB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 8 - Forks: 2

raphaelmansuy/mwaa_cli

A simple AirFlow mwaa cli command utility. It can be used to pause all the DAGS for a MWAA environment

Language: Shell - Size: 123 KB - Last synced: about 1 month ago - Pushed: over 1 year ago - Stars: 4 - Forks: 1

aimlnerd/data-pipelines-with-airflow

Airflow DAG tutorial with docker compose local setup

Language: Python - Size: 16.6 KB - Last synced: 4 months ago - Pushed: over 1 year ago - Stars: 3 - Forks: 0

philipobiorah/airflow_project

Weather Data ETL Project : This project involves a Python script that performs an Extract, Transform, Load (ETL) process for weather data. The script retrieves current weather information for a specified location and stores this data in an AWS S3 bucket.

Language: Python - Size: 8.79 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

sangwanamit621/learning-and-experiments

Welcome to the Learning and Experiments Hub—a dynamic repository capturing my journey of exploration and experimentation in the vast world of technology. This space serves as a digital canvas where I document my learning process, experiments, and discoveries.

Language: Jupyter Notebook - Size: 50.8 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

cristian-rincon/action-composer-sync

This is a simple action that helps you to fetch your Apache Airflow DAGs to Google Cloud Composer

Language: Shell - Size: 3.91 KB - Last synced: 2 months ago - Pushed: about 2 years ago - Stars: 6 - Forks: 0

jashshah-dev/AWS-Big-Data-Pipeline-orchestrated-with-Airflow

A robust data pipeline leveraging Amazon EMR and PySpark, orchestrated seamlessly with Apache Airflow for efficient batch processing

Language: Python - Size: 16.6 KB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

pulkit6559/FlaskSparkAirflow-MFAnalytics

Flask-powered application for orchestrating Spark jobs through a dashboard in a Dockerized Airflow and Spark environment, for Mutual Fund Data retrieval and analysis (https://www.mfapi.in/)

Language: Python - Size: 1.14 MB - Last synced: about 1 month ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

makarov-m/Yandex.Practicum.DE

Here I added 6 projects which have been made by me during my apprenticeship in Yandex.Practicum as data engineer.

Language: Python - Size: 3.01 MB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 3 - Forks: 3

Penosh22/Market-news-and-forex-exchange-ETL-pipeline

ETL pipeline to extract webscraped forex data and google news library data on daily basis and storing it in a postgres database for market analysis and insights

Language: Python - Size: 1.95 KB - Last synced: 5 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

MinaBasem/San-Francisco-Fire-Incidents-Streaming

An Ariflow, Kafka, AWS and Python data streaming project for Fire incidents in San Francisco

Language: Python - Size: 14.6 KB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

yassinessadi/Fraud-Detection-in-Financial-Transactions

Detect and prevent fraudulent transactions in real-time. Our project utilizes advanced data analysis, focusing on transactional, customer, and external data to enhance security, maintain customer trust, and minimize financial losses.

Language: Python - Size: 143 KB - Last synced: 6 months ago - Pushed: 6 months ago - Stars: 0 - Forks: 0

owenhiggins/ML-Network-Intrusion-Detection-Model

Created by: Owen Higgins ([email protected]) & Casey Gary ([email protected])

Language: Python - Size: 273 KB - Last synced: 6 months ago - Pushed: 6 months ago - Stars: 1 - Forks: 1

Ashutosh27ind/airflow-cert-dag-authoring

This repository contains examples and template files for DAG Authoring

Language: Python - Size: 23.4 KB - Last synced: 18 days ago - Pushed: 10 months ago - Stars: 0 - Forks: 0

karlapereira/airflow-example

Language: Python - Size: 29.3 KB - Last synced: 6 months ago - Pushed: 6 months ago - Stars: 0 - Forks: 0

lgthevinh/ev-stock-etl-pipeline

An ETL pipeline that extracts, transforms, and loads data from various sources related to electric vehicle (EV) stocks.

Language: Python - Size: 14.6 KB - Last synced: about 1 month ago - Pushed: 6 months ago - Stars: 0 - Forks: 0

andressagomes26/airflow_challenge

Repositório destinado à resolução do Desafio Módulo 5 LH - Orquestração com Airflow

Language: Python - Size: 454 KB - Last synced: 6 months ago - Pushed: 7 months ago - Stars: 0 - Forks: 0

pgrondein/etl_data_pipeline_airflow

Pipeline that analyzes the web server log file, extracts the required lines and fields, transforms, and load (append to an existing file.)

Language: Python - Size: 7.81 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 1 - Forks: 0

michaelosthege/apache-airflow-flowitems

This package helps to reduce the amount of boilerplate code when creating Airflow DAGs from Python callables.

Language: Python - Size: 33.2 KB - Last synced: 8 days ago - Pushed: about 1 year ago - Stars: 4 - Forks: 0

teenbress/Weather_Data_Pipeline_with_Airflow_and_Slack_Alert

End-to-end data pipeline --OpenWeather Alert, monitored with Slack alert

Language: Python - Size: 1.64 MB - Last synced: 7 months ago - Pushed: 7 months ago - Stars: 1 - Forks: 0

sagardua297/udacity-data-engineering-nd

Data Pipeline Analytics Platform is an end-to-end generic Big Data pipeline. Involves following tech stack: AWS S3, AWS Redshift, AWS EMR Cluster, Apache Spark, Apache Airflow.

Language: Python - Size: 1.81 MB - Last synced: 7 months ago - Pushed: over 3 years ago - Stars: 1 - Forks: 0

Manny-Brar/DataEngineeringNanodegree-P5-DataPipelines-Airflow-Spark-AWS

Utilizing Airflow's built-in functionalities creating a reusable ETL pipeline. Source data resides in a S3 bucket, and the pipeline should include data quality checks and data should be processed within AWS Redshift.

Language: Python - Size: 216 KB - Last synced: 7 months ago - Pushed: over 3 years ago - Stars: 0 - Forks: 0

mpavanetti/airflow

This set of code and instructions has the porpouse to instanciate a compiled environment with set of docker images like airflow webserver, airflow scheduler, postgresql, pyspark, Data Pipeline consuming data from weather api , processing with pyspark and storing in postgresql

Language: PHP - Size: 1.62 MB - Last synced: 7 months ago - Pushed: 7 months ago - Stars: 8 - Forks: 3

sreekesh93/airflow_beginners

to start airflow in local with basic setup

Language: Python - Size: 2.93 KB - Last synced: 7 months ago - Pushed: over 2 years ago - Stars: 4 - Forks: 0

Mouhamed-Jinja/postgres-s3-data-migration-with-airflow

Airflow Data Migration Project: A comprehensive Airflow project demonstrating data migration from PostgreSQL to AWS S3. Leverage the power of Airflow's operators, connections, and hooks to build robust and scalable data pipelines. Ideal for data engineering enthusiasts looking to learn and implement Airflow in real-world scenarios

Language: Python - Size: 7.08 MB - Last synced: 13 days ago - Pushed: 7 months ago - Stars: 0 - Forks: 0

prasadanilmore/ETL-JustEat-TestCase

This documentation provides an overview of the tasks completed in this project. The project comprises three key tasks: ETL (Extract, Transform, Load), API development, and Data Orchestration using Airflow. Each task is detailed below along with explanations of design choices and considerations.

Language: Python - Size: 79.1 KB - Last synced: 8 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

zarexalvindaria/data-engineering

This repo contains the Data Engineering exercises I took in Datacamp.

Language: Python - Size: 76.7 MB - Last synced: 8 months ago - Pushed: 8 months ago - Stars: 3 - Forks: 0

njoppi2/data-pipeline

🔄 A data pipeline that automates daily data extraction and loading between two Postgres databases.

Language: Jupyter Notebook - Size: 81.1 KB - Last synced: 8 months ago - Pushed: 8 months ago - Stars: 1 - Forks: 0

alopezgo/ETL-Python-Airflow-Bigquery

Extract Transform and Load Data scripts with Python, Bigquery/SQL API and Airflow orchestration

Language: Jupyter Notebook - Size: 388 KB - Last synced: 8 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

scriptstar/airflow-weather-station

Get started with Apache Airflow. Check the README for instructions on how to run your first DAGs today. 🚀

Language: Python - Size: 12.5 MB - Last synced: 8 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

at-akshat-2107/Codepro-EdTech-Lead-Scoring-Classification-MLOps-Assignment

In this project, you will create an end-to-end Airflow pipeline, integrated with MLflow, for CodePro, an EdTech startup, to perform lead scoring and maximize profitability while minimizing the Customer Acquisition Cost (CAC). The assignment involves data collection, preprocessing, and periodic model retraining within Airflow, alongside development

Language: Jupyter Notebook - Size: 13.9 MB - Last synced: 8 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

lediau/bigdata-data-engineering-ai-masters

Language: Python - Size: 122 KB - Last synced: 8 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

scalactic/airflow-example

airflow-example in docker

Language: Python - Size: 7.81 KB - Last synced: 9 months ago - Pushed: 10 months ago - Stars: 0 - Forks: 0

seblum/mlops-airflow-DAGs

Airflow DAGs repository synced for the MLOps plattform of the repository mlops-eks-mlplatform

Language: Python - Size: 165 MB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 1 - Forks: 0

immu0001/Udacity-Data-Engineer-nanodegree

Classwork projects and home works done through Udacity data engineering nano degree

Language: Jupyter Notebook - Size: 101 MB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 64 - Forks: 71

k0rsakov/dag_factory

Фабрика DAG

Language: Python - Size: 16.6 KB - Last synced: 9 months ago - Pushed: about 1 year ago - Stars: 0 - Forks: 1

shlin168/airflow-local-playground

docker compose files and some example DAGs for playing and learning airflow

Language: Python - Size: 148 KB - Last synced: 9 months ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

luelhagos/Data-warehouse

Data Engineering: Data warehouse tech stack with MySQL, DBT, Airflow, and Spark

Language: Python - Size: 2.93 KB - Last synced: 9 months ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

avikbesu/AirflowPractice

Language: Python - Size: 43.9 KB - Last synced: 9 months ago - Pushed: over 3 years ago - Stars: 0 - Forks: 0

prasanthsagarkottakota/Twitter-Pipeline

Tools : Python, Twitter API, Apache airflow, Amazon S3, EC2

Language: Python - Size: 7.81 KB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 1 - Forks: 0

belladzhu/airflow-projects

Создание дагов для автоматизации отчетности

Language: Python - Size: 5.86 KB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 0 - Forks: 0

rikiafifuddin/airflow-docker

Airflow Docker with pip instalation using requirement.txt and using DAG from cloned repository

Language: Dockerfile - Size: 16.6 KB - Last synced: 9 months ago - Pushed: 11 months ago - Stars: 0 - Forks: 0

mehroosali/s3-redshift-batch-etl-pipeline

Built functional python ETL script with functions that initialized spark clusters using pyspark library to extract songs stored in S3 bucket. Partitioned songs data by year and artist_id and compressed in parquet output files to increase load performance. Used the overwrite mode in spark to ensure every new run of ELT script is overwritten in the data lake to avoid duplicates. Orchestrated ELT data pipeline that extracts from S3, loads in redshift for transformation and loads output back to S3. Used hooks in airflow to make connection credentials configurable in order to separate access rights from code base for security. Used operators to execute loading and transformation scripts for redshift with airflow DAG.

Language: Python - Size: 944 KB - Last synced: 9 months ago - Pushed: over 2 years ago - Stars: 4 - Forks: 3

GitHub topics: airflow-dags

igorlangoni/final_project_data_eng_makers Fork of jdench1989/data_eng_final_project