GitHub topics: aws-athena
Antonio-Borges-Rufino/Build-an-Analytical-Platform-for-eCommerce-using-AWS-Services
Neste projeto, usaremos um conjunto de dados de comércio eletrônico para simular os registros de compras do usuário, visualizações de produtos, histórico de carrinho e jornada do usuário na plataforma online para criar dois pipelines analíticos, Lote e Tempo Real.
Size: 256 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
ksmin23/transactional-datalake-using-amazon-msk-iceberg-on-aws-glue
Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming and MSK Connect (Debezium)
Language: Python - Size: 679 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
ksmin23/transactional-datalake-using-amazon-msk-serverless-iceberg-on-aws-glue
Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming and MSK Connect (Debezium)
Language: Python - Size: 618 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
byungjun0689/aws-glue-etl-sample-hist-2022
Batch datapipeline on aws using glue in 2022
Size: 10.1 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
uhussain/WebCrawlerForOnlineInflation
Price Crawler - Tracking Price Inflation
Language: Python - Size: 387 KB - Last synced at: over 1 year ago - Pushed at: about 5 years ago - Stars: 155 - Forks: 47
nischaybikramthapa/dbt-athena-tpch
This project demonstrates how you can build downstream data pipeline using dbt in athena
Language: Python - Size: 297 KB - Last synced at: over 1 year ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
abduljaleel/workshop-appflow-athena-quicksight
AWS Programming and Tools meetup workshop
Size: 2.96 MB - Last synced at: 6 days ago - Pushed at: over 4 years ago - Stars: 9 - Forks: 0
gerardodavidlopezcastillo/GrafanaStepFunctionsSNS_Public
Data streaming project with Apache Druid & Grafana: Real-time data processing, alerts, integration with AWS. It uses a combination of technologies and services, including Confluent-Kafka, Apache Druid, AWS SNS, EC2, Athena, S3, Glue and EventBridge, StepFunctions. Contribute to this powerful solution!
Language: Jupyter Notebook - Size: 0 Bytes - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
Kelo/Athena-DBVisualizer
How to setup an AWS Athena Database connection in DBVisualizer!
Size: 9.77 KB - Last synced at: over 1 year ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
Jefersonalves/chess-data-lake
Data Lake project using AWS Services for a Data Engineering bootcamp
Language: Python - Size: 655 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
gerardodavidlopezcastillo/Cloud9KinesisAthena_Public
Streaming data analysis using AWS tools such as Cloud9 to generate events in the cloud, using boto3 to send records to Kinesis Data Firehose to connect to the S3 bucket destination, saving files in .parquet format. With the help of Glue, a data catalog will be created to enable real-time querying of all records with Athena.
Language: Python - Size: 1.3 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
byungjun0689/aws-glue-etl-sample-hist-2023
Batch datapipeline on aws using glue
Language: Python - Size: 104 MB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
dhvani-k/YouTrend_Insights_Analyzing_YouTube_Video_Landscape
An end-to-end solution for managing and analyzing YouTube video data from Kaggle, leveraging AWS services and visualized through Quicksight and Tableau
Language: Python - Size: 59.6 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
andersonesanto/igti-pa-repositorio-logs
IGTI MBA Engenharia de dados - Projeto Aplicado - Repositório de logs
Language: Dockerfile - Size: 13.7 KB - Last synced at: almost 2 years ago - Pushed at: almost 3 years ago - Stars: 0 - Forks: 0
QuiNovas/appsync-athena-resolver
AWS AppSync resolver that provides GraphQL access to Athena databases
Language: Python - Size: 37.1 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 14 - Forks: 0
ibrahimmoursy/STEDI-Human-Balance-Analytics
Data Lakehouse solution for data produced by STEDI Step Trainer sensors and the mobile app so that it can train the machine learning module.
Language: Python - Size: 7.19 MB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
cevoaustralia/data-lake-demo
Data lake demo using change data capture (CDC) on AWS
Language: PLpgSQL - Size: 215 KB - Last synced at: about 1 year ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 3
niravshah2705/DynamoDB_to_Athena
AWS has Athena service which can query structured data from S3. The DynamoDB is managed NoSQL database. So we have to convert Unstructured data to Structured data. The code written in python & performs this objective.
Language: Python - Size: 16.6 KB - Last synced at: almost 2 years ago - Pushed at: over 6 years ago - Stars: 1 - Forks: 0
arzanish/Spotify_ETL
Spotify ETL Pipeline project
Language: Jupyter Notebook - Size: 545 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 1 - Forks: 0
JamesJJ/dmarc-report-ses-tsv
Convert DMARC reports to TSV (or CSV) format for easier analysis and visualisation
Language: Go - Size: 430 KB - Last synced at: 7 days ago - Pushed at: over 1 year ago - Stars: 4 - Forks: 0
riship1095/YouTube-ETL
Transformed YouTube’s raw JSON data to parquet & loaded it in an S3 bucket, used Glue Data Catalog for storing metadata & Athena to query the cleaned data. Developed an ETL process using a Lambda job that would be triggered when raw data is loaded into an S3 bucket, processed, and stored for analytical purposes in an S3 bucket.
Language: Python - Size: 9.77 KB - Last synced at: almost 2 years ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
AirtonLira/aws-bigdata-glue-athena
Este projeto tem como objetivo realizar a coleta, catalogo, governança, processamento e visualização de dados.
Size: 3.76 MB - Last synced at: almost 2 years ago - Pushed at: about 2 years ago - Stars: 0 - Forks: 0
gborn/Serverless-ETL-Pipeline-on-AWS
Design of an ETL Pipeline to process and transform incrementally loaded data in datalake using AWS Lambda, Glue Jobs, EMR, and Athena.
Language: Python - Size: 445 KB - Last synced at: 2 months ago - Pushed at: about 3 years ago - Stars: 3 - Forks: 1
ndleah/STEDI
Data Lakehouse solution for machine learning data
Language: Python - Size: 693 KB - Last synced at: 4 months ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
fausa/Flight-Research-and-Analytics-Group
Expanding on Flight-Analytics project using AWS: build a classification model that predicts flight cancellation based on weather conditions for various US airports
Language: Jupyter Notebook - Size: 35.5 MB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 1
thedatanerdz/DEP-7
AWS Covid data engineering project
Size: 8.79 KB - Last synced at: over 1 year ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
IanLKaplan/javaorc
A Java library that makes writing and reading ORC files easy
Language: Java - Size: 285 KB - Last synced at: almost 2 years ago - Pushed at: almost 4 years ago - Stars: 2 - Forks: 1
somanathkshirsagar/Practical_Data_Science_on_the_AWS-Cloud-Specialization
The Practical Data Science Specialization brings together these disciplines using purpose-built ML tools in the AWS cloud. It helps you develop the practical skills to effectively deploy your data science projects and overcome challenges at each step of the ML workflow using Amazon SageMaker. This Specialization is designed for data-focused develop
Language: Jupyter Notebook - Size: 11.5 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 3 - Forks: 0
slouc/aws-rds-export-to-athena
AWS setup for loading daily DB snapshots into a data lake: RDS -> S3 -> Glue -> Athena, automatised via EventBridge. Includes the CloudFormation template.
Size: 130 KB - Last synced at: over 1 year ago - Pushed at: about 4 years ago - Stars: 13 - Forks: 3
stephenconnolly1/jupyter-athena
Demonstration of how to run interactive Athena queries using the ODBC driver in a Jupyter Notebook running in Docker
Language: Python - Size: 5.86 KB - Last synced at: almost 2 years ago - Pushed at: almost 7 years ago - Stars: 4 - Forks: 0
Undisputed-jay/SpotifyAPI-Data-Engineering-Project
This projects uses ETL (Extract, Transform and Load) pipeline to extract data from Spotify using its API and loads the data to a data source(AWS Athena). The entire pipeline will be built using Amazon Web Services (AWS).
Language: Jupyter Notebook - Size: 2.22 MB - Last synced at: 3 months ago - Pushed at: almost 2 years ago - Stars: 1 - Forks: 0
KoteiIto/node-athena
a nodejs simple aws athena client
Language: TypeScript - Size: 183 KB - Last synced at: about 1 month ago - Pushed at: about 4 years ago - Stars: 105 - Forks: 73
rahulrajan15/Stock_Market_Kafka
Real-Time Stock Market Data Science Project using Apache Kafka: Analyzing and predicting stock market trends in real-time for informed decision-making. Scalable and low-latency data processing.
Language: Jupyter Notebook - Size: 2.48 MB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
aimee0317/ETL-Data-Pipelines
Python ETL Data Pipeline with AWS Glue and Athena
Size: 662 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
akrennmair/koios
A SQL database client for text terminals, named after the Greek Titan Koios (Coeus) whose name means "query".
Language: Go - Size: 73.2 KB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0
aws-samples/aws-glue-athena-tcga
This repository has the code for the AWS blog post - Enabling the aggregation and query of The Cancer Genome Atlas using AWS Glue and Amazon Athena
Language: Jupyter Notebook - Size: 285 KB - Last synced at: 21 days ago - Pushed at: over 4 years ago - Stars: 2 - Forks: 2
dhruv007patel/Impact-of-Covid-19-on-Aviation-Industry
This project analyzes the correlation between COVID-19 and the US aviation industry. By studying data on passenger/freight traffic and delays alongside COVID-19 trends, it provides insights into airline and passenger responses. The findings help airlines adapt to the pandemic's impact.
Language: Python - Size: 504 KB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 2 - Forks: 0
ovokpus/AWS-ETL-Pipeline
Data Engineering Batch Pipeline with scheduled API calls as Ingestion, transformation with Glue Workflows, querying with Athena and consumption set up for Quicksight
Language: Python - Size: 491 KB - Last synced at: over 2 years ago - Pushed at: almost 3 years ago - Stars: 1 - Forks: 0
rochitasundar/TwitterSentimentAnalysis-BigDataProject
Scrapped tweets using twitter API (for keyword ‘Netflix’) on an AWS EC2 instance, ingested data into S3 via kinesis firehose. Used Spark ML on databricks to build a pipeline for sentiment classification model and Athena & QuickSight to build a dashboard
Language: Jupyter Notebook - Size: 8.79 MB - Last synced at: over 2 years ago - Pushed at: about 3 years ago - Stars: 4 - Forks: 0
burtcorp/athena-jdbc
A JDBC driver for AWS Athena
Language: Java - Size: 388 KB - Last synced at: about 1 year ago - Pushed at: over 3 years ago - Stars: 23 - Forks: 17
masood2iq/AWS-Athena-Glue-S3-Bucket-Deployment-Through-AWSConsole
AWS Athena, Glue Database, Glue Crawler and S3 buckets deployment through AWS GUI console.
Size: 3.18 MB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0
zsolt-halo/aws-config-advanced-query
A collection of useful queries that can be used to verify compliance/security across your AWS assets
Language: TSQL - Size: 5.86 KB - Last synced at: about 2 years ago - Pushed at: about 6 years ago - Stars: 25 - Forks: 4
omerh/aws-athena-query-counter
AWS Athena query counter
Language: Go - Size: 12.7 KB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
serratus-bio/serratus-summary-api-athena 📦
POC for serving Serratus summary data via AWS Athena
Language: JavaScript - Size: 13.7 KB - Last synced at: over 2 years ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 0
paoliniluis/shift-to-spectrum 📦
An automated SQL script generator to migrate AWS Redshift schemas (or tables) to AWS Redshift Spectrum
Language: JavaScript - Size: 8.79 KB - Last synced at: over 2 years ago - Pushed at: over 7 years ago - Stars: 1 - Forks: 0
ljofreflor/athena2pyspark
Very simple library to consume aws athena from spark or lambda services
Language: Python - Size: 6.9 MB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 1
u2i/egis
Egis - a handy Ruby interface for AWS Athena
Language: Ruby - Size: 317 KB - Last synced at: 7 months ago - Pushed at: over 3 years ago - Stars: 41 - Forks: 2
epomatti/aws-glue-athena
Glue ETL crawler and jobs with Athena queries
Language: HCL - Size: 138 KB - Last synced at: 24 days ago - Pushed at: almost 3 years ago - Stars: 1 - Forks: 0
zxkane/serverless-docker-images-analytics
Serverless Analytics app for analyzing docker image layers
Language: TypeScript - Size: 8.92 MB - Last synced at: 3 months ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
kyopark2014/aws-athena
It decribes how to use aws athena for data analytics
Size: 31.3 KB - Last synced at: 3 months ago - Pushed at: almost 3 years ago - Stars: 2 - Forks: 0
mlnrt/pexip-logs-in-aws
Pexip Infinity log analysis on the AWS cloud
Size: 1.91 MB - Last synced at: about 2 years ago - Pushed at: about 6 years ago - Stars: 4 - Forks: 1
ShubhamJagtap2000/Amazon-AWS
☁️ This repository contains what I learned about Amazon AWS during my job role as Amazon Developer
Size: 52.7 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0
desaikun1996/Youtube-Data-Preparation-and-Analysis
An end to end Data Engineering project demonstrating the complete flow of Data Gathering, Data Ingestion, dumping data into Data Lake using Data Pipeline, Data Processing, Data Lookup, Monitoring the data pipeline activities/logs and eventually Visualizing important insights from the final data. For the entire project, I will be making use of AWS services like S3, AWS Glue, AWS Lambda, Data Catalog, AWS Athena, AWS QuickSight, AWS CloudWatch.
Size: 2.93 KB - Last synced at: over 2 years ago - Pushed at: almost 3 years ago - Stars: 0 - Forks: 0
gakas14/car_insurance_with_aws
Get the dataset intro a S3 bucket, use AWS glue to transform the dataset, write a Lambda script to clean the dataset, query the dataset via AWS Athena then build a dashboard using AWS Quicksight.
Language: Python - Size: 1.11 MB - Last synced at: 25 days ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
dvu4/udacity-data-engineering
Data Engineering Projects including Data Modeling, Data Warehouse, Data Lake Development
Language: Jupyter Notebook - Size: 2.09 MB - Last synced at: over 2 years ago - Pushed at: about 5 years ago - Stars: 2 - Forks: 2
alexisfacques/aws-serverless-dataplatform
A fully serverless and cost efficient data platform framework built on AWS.
Language: Python - Size: 3 MB - Last synced at: over 2 years ago - Pushed at: about 4 years ago - Stars: 2 - Forks: 0
Devalent/aws-realtime-predictions
An end-to-end example of a serverless machine learning pipeline for multiclass classification on AWS with SageMaker Pipelines, Data Wrangler, Athena and XGBoost.
Language: TypeScript - Size: 1.17 MB - Last synced at: over 2 years ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 0
ShivamJoker/AWS-Athena-Query
Query AWS Athena using AWS SDK v3 in JavaScript
Language: TypeScript - Size: 15.6 KB - Last synced at: about 1 year ago - Pushed at: almost 3 years ago - Stars: 0 - Forks: 0
vt-digital-libraries-platform/FixityEventRule
A CloudWatch schedule that invokes a Lambda function to query Athena and execute Step functions.
Language: Python - Size: 3.1 MB - Last synced at: over 2 years ago - Pushed at: almost 5 years ago - Stars: 0 - Forks: 0
mincloud1501/DevOps
DevOps에 대한 개념 이해와 AWS 개발자 도구를 활용한 실습 및 연구
Language: Java - Size: 3.21 MB - Last synced at: over 2 years ago - Pushed at: over 4 years ago - Stars: 12 - Forks: 1
andre-marcos-perez/ifood-arch-readme
The application is the documentation of my solution for the iFood data architect test.
Size: 454 KB - Last synced at: 3 months ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 0
markoshlima/crimes-map
This Big Data project consists of obtaining data on vehicle theft in the city of São Paulo and consolidating it in a counting and heat map, in order to show areas with a higher index of this type of crime. All applicable in AWS Resources.
Language: Scala - Size: 13.7 MB - Last synced at: over 1 year ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 0
ricardo-farias/CovidDataProduct
This repository will be used to understand data science and data engineering concepts
Language: Scala - Size: 641 KB - Last synced at: about 1 year ago - Pushed at: over 4 years ago - Stars: 2 - Forks: 1
cloudemprise/cfn-siem-cli
An automatically deployed event monitoring and analyses solution for the AWS Cloud.
Language: Shell - Size: 29.3 KB - Last synced at: about 1 year ago - Pushed at: about 3 years ago - Stars: 0 - Forks: 0
mariamrf/athena-buildings
A script that queries OSM using AWS Athena for buildings in a given bounding box. Demo:
Language: Python - Size: 5.72 MB - Last synced at: over 2 years ago - Pushed at: over 6 years ago - Stars: 14 - Forks: 3
dacort/athena-query-stats
Query your Athena query history using Athena 🙆♂️
Language: Python - Size: 3.91 KB - Last synced at: 2 months ago - Pushed at: over 5 years ago - Stars: 2 - Forks: 2
mgorski-mg/cloud-front-patterns
Samples of CloudFront patterns
Language: PowerShell - Size: 29.3 KB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 0
productiveAnalytics/aws-cdk-constructs-sandbox
Cloud Development Kit (AWS CDK) using TypeScript, Python and Java
Language: Java - Size: 5.49 MB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 0 - Forks: 0
xianchen2/Financal_Data_Ingestion
Financial data ingestion and analysis with AWS Kinesis and Athena
Language: Jupyter Notebook - Size: 6.34 MB - Last synced at: over 2 years ago - Pushed at: almost 5 years ago - Stars: 1 - Forks: 0
serratus-bio/serratus-summary-uploader
parse and upload Serratus summary files to various AWS offerings via AWS Lambda
Language: Python - Size: 116 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
tamdilip/aws-kinesis-firehose-springboot
Simple Spring Boot project to stream data through Amazon Kinesis Firehose
Language: Java - Size: 598 KB - Last synced at: 3 months ago - Pushed at: about 4 years ago - Stars: 0 - Forks: 1
donjude/data-lakes-with-spark
This project is about building a data lake and creating an ETL pipeline in Spark that loads data from Amazon S3, processes the data into analytics tables, and loads them back into S3
Language: Python - Size: 412 KB - Last synced at: over 2 years ago - Pushed at: about 4 years ago - Stars: 0 - Forks: 0
msantino/aws_plugin
Airflow plugin to interact with AWS to complement Airflow's native AWS functionalities.
Language: Python - Size: 4.88 KB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 4 - Forks: 1
j4c0bs/spectron
AWS Redshift Athena and Spectrum Utilities
Language: Python - Size: 112 KB - Last synced at: 26 days ago - Pushed at: over 4 years ago - Stars: 0 - Forks: 1
grbinho/athena-azure-ad
Athena JDBC Authentication provider for Azure AD
Language: Scala - Size: 6.59 MB - Last synced at: 5 days ago - Pushed at: about 6 years ago - Stars: 4 - Forks: 0
gnokoheat/serverless-aws-athena-ui
Serverless AWS Athena UI - Simple & Easy query view of AWS Athena.
Language: HTML - Size: 226 KB - Last synced at: over 2 years ago - Pushed at: over 5 years ago - Stars: 3 - Forks: 2
scriptbuzz/aws-datalake-poc-video
AWS hosted enterprise Data Lake with both batch and realtime data pipelines.
Size: 349 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 1 - Forks: 0
youngwjung/aws-cloud-trail
AWS CloudTrail Hands-on Lab
Size: 127 KB - Last synced at: over 2 years ago - Pushed at: about 5 years ago - Stars: 0 - Forks: 0
msjgriffiths/rAthena
dplyr backend for Amazon Athena
Language: R - Size: 7.9 MB - Last synced at: almost 2 years ago - Pushed at: over 8 years ago - Stars: 8 - Forks: 1
macbury/saint_seiya
Example app showing how to interact with amazon athena
Language: Ruby - Size: 5.86 KB - Last synced at: about 1 year ago - Pushed at: over 6 years ago - Stars: 0 - Forks: 0
richiverse/fiat
Firehose to Athena
Language: Python - Size: 9.77 KB - Last synced at: 10 days ago - Pushed at: over 7 years ago - Stars: 6 - Forks: 0
zkan/hello-amazon-athena
Hello, Amazon Athena!
Language: Python - Size: 4.88 KB - Last synced at: 3 months ago - Pushed at: over 7 years ago - Stars: 0 - Forks: 0
tearoom6/postal_search_jp
Postal search engine in Japan using Amazon Athena.
Language: Ruby - Size: 12.7 KB - Last synced at: 21 days ago - Pushed at: over 8 years ago - Stars: 0 - Forks: 0
