GitHub / khushal2405 / ETL-pipeline-using-Airflow-and-AWS-EMR

We Build an ETL pipeline using Airflow that accomplishes the following: Downloads data from an AWS S3 bucket, Runs a Spark/Spark SQL job on the downloaded data producing a cleaned-up dataset of delivery deadline missing orders and then Upload the cleaned-up dataset back to the same S3 bucket in a folder primed for higher level analytics

JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khushal2405%2FETL-pipeline-using-Airflow-and-AWS-EMR

Topics: airflow, apache-spark, aws, aws-emr, data-engineering, etl, pyspark, python, s3, s3-bucket, scala, spark

• Files
• Readme
• Dependencies