nathadriele/S3-folder-cleanup

This script automates cleaning up a specific folder in an S3 bucket, deleting all objects within it. It uses secure AWS credentials and is built on the Mage.ai platform. Additionally, the roadmap includes approach handling and logging for greater robustness and monitoring.

Language: Python - Size: 9.77 KB - Last synced at: 18 days ago - Pushed at: 18 days ago - Stars: 6 - Forks: 0

nathadriele/DMS-CDC-task-status-validator

Automated script developed in Mage.ai to monitor and validate the status of AWS DMS CDC tasks, ensuring data integrity and synchronization. Sends notifications for any detected validation issues.

Language: Python - Size: 31.3 KB - Last synced at: 11 days ago - Pushed at: 20 days ago - Stars: 6 - Forks: 0

nathadriele/DMS-missing-or-duplicate-data-validation-script

Contains a Python script designed to validate data replication tasks in AWS Database Migration Service (DMS). The script checks for potential issues such as missing or duplicate data in the tables being replicated.

Language: Python - Size: 24.4 KB - Last synced at: 11 days ago - Pushed at: 20 days ago - Stars: 9 - Forks: 0

nathadriele/data-engineering-zoomcamp

The Data Engineering Zoomcamp covers essential skills in containerization, workflow orchestration, data warehousing, analytics engineering, batch, and streaming processing. It includes tools like Docker, Terraform, BigQuery, dbt, Spark, Kafka, Kestra, Postgres, Google Data Studio, and Metabase.

Language: Python - Size: 12.5 MB - Last synced at: 21 days ago - Pushed at: 21 days ago - Stars: 8 - Forks: 1

komminarlabs/terraform-provider-mageai

Terraform provider to manage Mage AI

Language: Go - Size: 170 KB - Last synced at: 22 days ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

nathadriele/llm-zoomcamp

The Zoomcamp LLM Course focuses on tools for working with LLMs and RAG, including OpenAI API, HuggingFace, Elasticsearch, and Streamlit. It covers vector search, embedding creation, data ingestion with Mage, and monitoring using Grafana, emphasizing practical applications and best practices.

Language: Jupyter Notebook - Size: 1.82 MB - Last synced at: 13 days ago - Pushed at: 2 months ago - Stars: 7 - Forks: 1

nathadriele/datamart-tables-data-type-validation

This project is a Data Engineering solution implemented to validate the data types of columns in PostgreSQL tables in a DataMart. It aims to validate whether the data stored in tables conforms to the expected data types, improving data integrity and reliability.

Language: Python - Size: 16.6 KB - Last synced at: 28 days ago - Pushed at: 9 months ago - Stars: 5 - Forks: 0

nathadriele/AWS-DMS-task-restart-and-status-checker

The AWS DMS Task Restart and Status Checker is a Python script designed to restart various AWS Database Migration Service (DMS) tasks and check their status. This script leverages AWS SDK (Boto3) and Mage.ai for DMS task integration and management, ensuring efficient and reliable task handling.

Language: Python - Size: 2.48 MB - Last synced at: 28 days ago - Pushed at: 3 months ago - Stars: 8 - Forks: 0

lunaSnowflake/CryptoCurrency

Engineered a real-time Cryptocurrency dashboard on AWS using Python, MageAI, SQL, and NLP for sentiment analysis, with ML forecasting and PowerBI integration for dynamic reporting and decision-making.

Language: Jupyter Notebook - Size: 43.5 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 1 - Forks: 1

kingabzpro/5-Airflow-Alternatives-for-Data-Orchestration-Tutorial

Code examples of Luigi, Prefect, Kedro, Dagster, and MageAI

Language: Python - Size: 40 KB - Last synced at: about 1 month ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

JBris/mage_ai_docker

Docker image for Mage AI deployment using Docker

Language: Dockerfile - Size: 8.79 KB - Last synced at: 12 days ago - Pushed at: 4 months ago - Stars: 1 - Forks: 0

ShubhamMohanty680/Uber_Data_Analysis

Language: Python - Size: 3.99 MB - Last synced at: about 1 month ago - Pushed at: 10 months ago - Stars: 1 - Forks: 0

Hamagistral/de-zoomcamp-ui

🎨 UI for the Free Data Engineering Zoomcamp Course provided by DataTalksClub

Language: Python - Size: 103 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 224 - Forks: 100

manuelandersen/football-pipeline

DE Zoomcamp 2024 Final Project 🧙

Language: Python - Size: 975 KB - Last synced at: about 1 month ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

ChukwuemekaAham/uber-gcp-etl-project

Data Engineering Zoomcamp Final Project

Language: Jupyter Notebook - Size: 33.7 MB - Last synced at: about 2 months ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0

ChukwuemekaAham/data-engineering-zoomcamp

Datatalks Club Free Data Engineering Zoomcamp Project

Language: Jupyter Notebook - Size: 4.63 MB - Last synced at: about 2 months ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0

benitomartin/de-ch-weather

Data Engineering Swiss Air Quality

Language: Python - Size: 8.03 MB - Last synced at: about 2 months ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

Siddha911/Citibike-data-project

Data engineering project for the DEZoomcamp 2024

Language: Python - Size: 2.6 MB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

AymanSulaiman/pokeFastAPI

pokéFastApi: A backend API with Docker Compose integration for retrieving and searching Pokémon data using FastAPI and DuckDB.

Language: Python - Size: 49.8 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

meysamraz/laptop-price-prediction-end-to-end-project-using-ecommerce-website-data

Using machine learning, feature engineering, and web scraping, I created an end-to-end laptop price prediction website by scraping data from a popular Iranian source. Empowering users with accurate pricing estimates and model comparisons

Language: Jupyter Notebook - Size: 2.22 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

AviatorIfeanyi/etl_with_mage_ai

An ETL data pipeline that extracts data from source and loads it to destination, automated using mage.ai

Language: Python - Size: 246 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 2

thedatanerdz/DEP-8

Analyzing Uber data using various tools and technologies, including GCP Storage, Python, Compute Instance, Mage Data Pipeline Tool, BigQuery, and Looker Studio.

Language: Jupyter Notebook - Size: 120 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

Hamagistral/NYCTaxi-Analytics-ETL

🚕 Performing Data Analytics on NYC Taxi data using GCP and MageAI

Language: Jupyter Notebook - Size: 4.2 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0