GitHub / vaxdata22 / City-Weather-and-S3File-RDS-S3-BigQuery-ETL-by-Airflow-on-EC2

This is my third AWS Cloud ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2. It demonstrates how to build an ETL data pipeline that would perform data extraction to a database in parallel to a loading process into the same database, join the tables, copy joined data to S3 and finally copy the S3 file to BigQuery DW.

JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vaxdata22%2FCity-Weather-and-S3File-RDS-S3-BigQuery-ETL-by-Airflow-on-EC2

Topics: apache-airflow, aws-ec2, aws-rds-postgres, aws-s3, bigquery, business-intelligence, dags, data-warehousing, etl-pipeline, openweathermap-api, orchestration, python3, sql

• Files
• Readme
• Dependencies