aws/aws-sdk-pandas

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Language: Python - Size: 15.6 MB - Last synced at: 8 days ago - Pushed at: 8 days ago - Stars: 4,057 - Forks: 715

AmritPrakash3/Reddit-ETL-in-AWS-using-Airflow

Reddit ETL in AWS using Airflow is a full-stack data engineering project that builds a scalable ETL pipeline using cloud-based tools. It extracts data from Reddit via API, processes it with Apache Airflow, and leverages AWS services like S3, Glue, Athena, and Redshift for transformation, querying, and warehousing.

Language: Python - Size: 122 KB - Last synced at: 9 days ago - Pushed at: 9 days ago - Stars: 0 - Forks: 0

dacort/demo-code

Bits of code I use during live demos

Language: Jupyter Notebook - Size: 774 KB - Last synced at: 9 days ago - Pushed at: 9 months ago - Stars: 30 - Forks: 24

zaneli/scalikejdbc-athena

Library for using Amazon Athena JDBC Driver with ScalikeJDBC

Language: Scala - Size: 77.1 KB - Last synced at: 15 days ago - Pushed at: 15 days ago - Stars: 23 - Forks: 5

Kiran8053/Reddit-ETL-in-AWS-using-Airflow

Reddit ETL in AWS using Airflow is a full-stack data engineering project that builds a scalable ETL pipeline using cloud-based tools. It extracts data from Reddit via API, processes it with Apache Airflow, and leverages AWS services like S3, Glue, Athena, and Redshift for transformation, querying, and warehousing.

Language: Python - Size: 122 KB - Last synced at: 26 days ago - Pushed at: 26 days ago - Stars: 0 - Forks: 0

ghdna/athena-express

Athena-Express can simplify executing SQL queries in Amazon Athena AND fetching cleaned-up JSON results in the same synchronous or asynchronous request - well suited for web applications.

Language: JavaScript - Size: 214 KB - Last synced at: 1 day ago - Pushed at: over 1 year ago - Stars: 181 - Forks: 70

aws-samples/amazon-rds-export-to-s3-automation

This repository contains source code for the AWS Database Blog Post Reduce data archiving costs for compliance by automating RDS snapshot exports to Amazon S3

Size: 235 KB - Last synced at: 3 months ago - Pushed at: over 2 years ago - Stars: 17 - Forks: 2

aws-samples/analyzing-reddit-sentiment-with-aws

Learn how to use Kinesis Firehose, AWS Glue, S3, and Amazon Athena by streaming and analyzing reddit comments in realtime. 100-200 level tutorial.

Language: Python - Size: 3.48 MB - Last synced at: 3 months ago - Pushed at: over 4 years ago - Stars: 44 - Forks: 15

Ditectrev/Amazon-Web-Services-Certified-AWS-Certified-Machine-Learning-MLS-C01-Practice-Tests-Exams-Question

⛳️ PASS: Amazon Web Services Certified (AWS Certified) Machine Learning Specialty (MLS-C01) by learning based on our Questions & Answers (Q&A) Practice Tests Exams.

Size: 6.88 MB - Last synced at: 4 months ago - Pushed at: 11 months ago - Stars: 63 - Forks: 38

aws-samples/cloud-experiments 📦

Open innovation with 60 minute cloud experiments on AWS

Language: Jupyter Notebook - Size: 22.8 MB - Last synced at: about 2 months ago - Pushed at: over 1 year ago - Stars: 88 - Forks: 56

husskhosravi/athena-glue-s3-pipeline

End-to-end project using S3, Glue, Athena, and QuickSight to build a secure, automated data processing and visualisation workflow

Size: 463 KB - Last synced at: 4 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

aws-samples/aws-usage-queries

This application bootstraps everything needed to query the AWS Cost and Usage reports through Amazon Athena. It also includes reference data and preconfigured SQL queries.

Language: TypeScript - Size: 1.03 MB - Last synced at: 3 months ago - Pushed at: 5 months ago - Stars: 51 - Forks: 5

aws-samples/aws-fargate-with-rstudio-open-source

This project delivers AWS CDK Python code to provision serverless infrastructure in AWS Cloud to run Open Source RStudio Server and Shiny.

Language: Python - Size: 1.47 MB - Last synced at: about 1 month ago - Pushed at: about 2 years ago - Stars: 78 - Forks: 14

vaxdata22/Customer-Churn-Data-Analytics-ETL-Pipeline-by-Airflow-on-EC2

This is an end-to-end AWS Cloud ETL project. This orchestration uses Apache Airflow on AWS EC2 as well as AWS Glue. It demonstrates how to build ETL pipeline that would perform data transform using Glue job/crawler as well as loading into a Redshift table. It also shows how to connect Amazon Athena to Glue Data Catalog, and Power BI to Redshift.

Language: Python - Size: 7.2 MB - Last synced at: 3 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

awslabs/amazon-athena-cross-account-catalog 📦

🌉 Reference implementation for granting cross-account AWS Glue Data Catalog access from Amazon Athena

Language: Python - Size: 150 KB - Last synced at: 3 days ago - Pushed at: about 3 years ago - Stars: 30 - Forks: 19

archived-blueprints/amazonathena-blueprints 📦

Simplified blueprints for building data pipelines with Amazon Athena.

Language: Python - Size: 15.6 KB - Last synced at: 6 days ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 0

Wolf-nord/Customer-Churn-Data-Analytics-ETL-Pipeline-by-Airflow-on-EC2

This is an end-to-end AWS Cloud ETL project. This orchestration uses Apache Airflow on AWS EC2 as well as AWS Glue. It demonstrates how to build ETL pipeline that would perform data transform using Glue job/crawler as well as loading into a Redshift table. It also shows how to connect Amazon Athena to Glue Data Catalog, and Power BI to Redshift.

Language: Python - Size: 7.17 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0

hrbrmstr/awsathena

rJava Interface to AWS Athena SDK

Language: R - Size: 26.4 KB - Last synced at: 5 months ago - Pushed at: over 5 years ago - Stars: 4 - Forks: 0

dacort/metabase-athena-driver

An Amazon Athena driver for Metabase 0.32 and later

Language: Clojure - Size: 143 KB - Last synced at: 5 months ago - Pushed at: almost 3 years ago - Stars: 223 - Forks: 32

cookpad/minerva 📦

Serverless Log Search Architecture for Security Monitoring based on Amazon Athena

Language: Go - Size: 465 KB - Last synced at: 30 days ago - Pushed at: 7 months ago - Stars: 8 - Forks: 0

CloudFay/Sports-Data-Lake

This repository houses the setup_nba_data_lake.py script, which automates the entire process of building a cloud-based data lake for NBA analytics. With this script, you can seamlessly integrate Amazon S3, AWS Glue, and Amazon Athena to store, process, and query NBA-related data—all in a fully scalable and serverless environment!

Language: Python - Size: 7.81 KB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 0 - Forks: 0

dominique-jacque/NBA-Data-Lake

NBA Data Lake Repository contains the setup_nba_data_lake.py script, which automates the creation of a data lake for NBA analytics using AWS services. The script integrates Amazon S3, AWS Glue, and Amazon Athena, and sets up the infrastructure needed to store and query NBA-related data.

Language: Python - Size: 9.77 KB - Last synced at: 6 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

Ditectrev/Amazon-Web-Services-Certified-AWS-Certified-Data-Analytics-DAS-C01-Practice-Tests-Exams-Question

⛳️ PASS: Amazon Web Services Certified (AWS Certified) Data Analytics Specialty (DAS-C01) by learning based on our Questions & Answers (Q&A) Practice Tests Exams.

Size: 9.44 MB - Last synced at: 5 months ago - Pushed at: 10 months ago - Stars: 7 - Forks: 7

aws-samples/Simplify-natural-language-query-with-Anthropic-Claude-on-Amazon-Bedrock

Utilize the multimodal capabilities of Anthropic Claude on Amazon Bedrock to simplify the natural language query by feeding entity relationship diagram (ERD).

Language: Jupyter Notebook - Size: 663 KB - Last synced at: 3 months ago - Pushed at: about 1 year ago - Stars: 3 - Forks: 0

dacort/athena-sqlite

A SQLite driver for S3 and Amazon Athena 😳

Language: Python - Size: 85 KB - Last synced at: 5 months ago - Pushed at: over 5 years ago - Stars: 95 - Forks: 5

AhmetFurkanDEMIR/dataengineering-youtube-project

Data Engineering Youtube Project

Language: Jupyter Notebook - Size: 902 KB - Last synced at: 3 months ago - Pushed at: about 2 years ago - Stars: 12 - Forks: 0

aws-samples/query-data-in-s3-with-amazon-athena-and-aws-sdk-for-dotnet

This Project provides a sample implementation that will show how to leverage Amazon Athena from .NET Core Application using AWS SDK for .NET to run standard SQL to analyze a large amount of data in Amazon S3.

Language: C# - Size: 5.65 MB - Last synced at: 3 months ago - Pushed at: almost 2 years ago - Stars: 24 - Forks: 8

jxareas/Athena-SpringKlient

POC app to show how to query Athena and integrate the AWS SDK in Spring Boot.

Language: Kotlin - Size: 78.1 KB - Last synced at: 6 months ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

aws-samples/amazon-athena-queries-via-aws-lambda-cdk

Language: TypeScript - Size: 117 KB - Last synced at: 3 months ago - Pushed at: 5 months ago - Stars: 12 - Forks: 3

MayraVillarrubia/YouTube-channels-stats

Extraction of daily stats from YouTube channels

Language: Jupyter Notebook - Size: 7.81 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0

m-radzikowski/aws-creating-athena-tables

Example of different ways to create Amazon Athena tables

Language: JavaScript - Size: 86.9 KB - Last synced at: about 2 months ago - Pushed at: over 4 years ago - Stars: 3 - Forks: 1

aws-samples/aws-athena-cqrs-pattern

Example of CQRS(Command and Query Responsibility Segregation) Pattern using Amazon Athena

Language: Python - Size: 256 KB - Last synced at: 3 months ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0

marwan116/aws-parquet

a toolkit that provides an object-oriented interface for working with parquet datasets on AWS

Language: Python - Size: 43.9 KB - Last synced at: 3 months ago - Pushed at: about 2 years ago - Stars: 3 - Forks: 0

hrbrmstr/awsathenajars

Java Archive Wrapper Supporting the 'awsathena' Package

Language: R - Size: 17 MB - Last synced at: 6 months ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 1

ngonghi/athena_s3_sample

Use Amazon AWS SDK to upload data to S3 bucket and query by Athena Sample Program

Language: Go - Size: 17.6 KB - Last synced at: 4 months ago - Pushed at: about 8 years ago - Stars: 7 - Forks: 2

ungikim/zeppelin-athena-interpreter 📦

Amazon Athena interpreter for Apache Zeppelin

Language: Scala - Size: 39.1 KB - Last synced at: over 2 years ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 0

kyopark2014/iot-analytics-for-thermometer

It shows IoT analytics infrastructure to gather sensor information using M5Stack, AWS IoT Edukit.

Language: C - Size: 274 MB - Last synced at: 5 months ago - Pushed at: about 2 years ago - Stars: 8 - Forks: 0

AWS-Big-Data-Projects/HeartRate-Monitoring-using-AWS-IOT-and-AWS-KINESIS

you run a script to mimic multiple sensors publishing messages on an IoT MQTT topic, with one message published every second. The events get sent to AWS IoT, where an IoT rule is configured. The IoT rule captures all messages and sends them to Firehose. From there, Firehose writes the messages in batches to objects stored in S3. In S3, you set up a table in Athena and use QuickSight to analyze the IoT data.

Language: Python - Size: 15.6 KB - Last synced at: 5 days ago - Pushed at: over 2 years ago - Stars: 10 - Forks: 1

joedementri/athena2pd

Amazon Athena to Pandas Dataframe

Language: Python - Size: 12.7 KB - Last synced at: 15 days ago - Pushed at: over 5 years ago - Stars: 2 - Forks: 1

contactsunny/AmazonAthenaSpringBootPOC

This is a simple Spring Boot POC to demonstrate how we can query an Amazon Athena table from a Spring Boot application, or any other Java code in general.

Language: Java - Size: 2.93 KB - Last synced at: over 1 year ago - Pushed at: almost 6 years ago - Stars: 1 - Forks: 4

swx0/reddit-sentiment-analysis-with-AWS

Analyse user sentiments and identify entities on subreddits using AWS serverless architecture.

Language: Python - Size: 51.8 KB - Last synced at: over 2 years ago - Pushed at: almost 4 years ago - Stars: 0 - Forks: 0

rkshiyaniya/Amazon-Product-Review-Sentiment-Classification

Build end-to-end Data Science (NLP) project for Amazon's Product Review Sentiment Classification using AWS tools.

Language: Jupyter Notebook - Size: 7.08 MB - Last synced at: over 2 years ago - Pushed at: about 4 years ago - Stars: 0 - Forks: 0

hrbrmstr/metis-jars

JAR files for metis package

Language: R - Size: 32.6 MB - Last synced at: 6 months ago - Pushed at: over 5 years ago - Stars: 0 - Forks: 0

adilkhash/aws-meetup-almaty-2019-data-lake

Resources for AWS Almaty Meetup: Building scalable Data Lake with AWS

Language: Python - Size: 2.93 KB - Last synced at: 7 months ago - Pushed at: over 5 years ago - Stars: 0 - Forks: 0

TrainingByPackt/Serverless-Architecture-with-AWS-eLearning

Discover how you can migrate from traditional deployments to serverless architectures with AWS

Language: JavaScript - Size: 98.6 KB - Last synced at: about 2 months ago - Pushed at: over 6 years ago - Stars: 1 - Forks: 3

jspooner/athena-jruby-connection-example

Access Amazon Athena using a Java Database Connectivity (JDBC) driver with JRuby. For information on other connections or to download the Athena driver please refer to the documentation.

Language: Ruby - Size: 7.91 MB - Last synced at: over 2 years ago - Pushed at: about 3 years ago - Stars: 1 - Forks: 0