Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: kinesis-firehose

Repositories

SumoLogic/sumologic-aws-lambda

A collection of lambda functions to collect data from Cloudwatch, Kinesis, VPC Flow logs, S3, security-hub and AWS Inspector

Language: Python - Size: 72.4 MB - Last synced: 1 day ago - Pushed: 1 day ago - Stars: 148 - Forks: 133

aws-samples/opensearch-serverless-common-usage-patterns

Typical use cases of opensearch serverelss: search, time-series, kinesis firehose integration, securing with VPC

Language: Python - Size: 2.31 MB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 12 - Forks: 2

muhlba91/telegraf-output-kinesis-data-firehose

Telegraf Output Plugin for Amazon Kinesis Data Firehose

Language: Go - Size: 450 KB - Last synced: 2 days ago - Pushed: 2 days ago - Stars: 0 - Forks: 2

awslabs/aws-fluent-plugin-kinesis

Amazon Kinesis output plugin for Fluentd

Language: Ruby - Size: 466 KB - Last synced: 14 days ago - Pushed: about 1 month ago - Stars: 292 - Forks: 97

Moulick/kinesis2elastic

Receives data from firehose and puts it to ElasticSearch

Language: Go - Size: 1.33 MB - Last synced: 16 days ago - Pushed: 16 days ago - Stars: 3 - Forks: 0

aws-samples/web-analytics-on-aws

This web analytics demo shows how to collect web logs with API Gateway and store them into S3 through Amazon Kinesis. Then this project shows how to analyze web logs with Amazon Athena.

Language: Python - Size: 305 KB - Last synced: 21 days ago - Pushed: 21 days ago - Stars: 13 - Forks: 2

rewardStyle/kinetic

High-Performance AWS Kinesis Client for Go

Language: Go - Size: 1.77 MB - Last synced: 22 days ago - Pushed: 22 days ago - Stars: 20 - Forks: 10

miztiik/sensitive-data-filter-instream

Scrub Sensitive Data from Event Streams - Kinesis

Language: Python - Size: 155 KB - Last synced: 30 days ago - Pushed: over 3 years ago - Stars: 0 - Forks: 1

miztiik/kinesis-tumbling-window-analytics

Streaming analytics using Kinesis Data Analytics using SQL

Language: Python - Size: 563 KB - Last synced: 30 days ago - Pushed: about 3 years ago - Stars: 1 - Forks: 4

spoqa/firefighter

AWS Kinesis Data Firehose logging handler for Python

Language: Python - Size: 20.5 KB - Last synced: about 1 month ago - Pushed: about 5 years ago - Stars: 1 - Forks: 1

QuiNovas/dynamo-firehose-adapter

A simple adapter that takes records from a DynamoDB stream and sends them to a Kinesis Firehose for processing.

Language: Python - Size: 16.6 KB - Last synced: about 1 month ago - Pushed: over 1 year ago - Stars: 0 - Forks: 1

fdmsantos/terraform-aws-kinesis-firehose

Dynamic Terraform module, which creates a Kinesis Firehose Stream and others resources like Cloudwatch, IAM Roles and Security Groups that integrate with Kinesis Firehose. Supports all destinations and all Kinesis Firehose Features.

Language: HCL - Size: 323 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 6 - Forks: 6

having-fun-serverless/backup-twitter

Backup your twitter account on a daily basis

Language: Python - Size: 1.46 MB - Last synced: about 2 months ago - Pushed: over 1 year ago - Stars: 1 - Forks: 0

ZeroTwoDataRW/Ecommerce-DE-On-AWS

An end-to-end AWS data engineering pipeline that orchestrates data ingestion, processing, and storage using Kinesis, S3, Redshift, Lambda, DynamoDB

Language: Python - Size: 81.1 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 1 - Forks: 0

sdabhi23/streaming-data-pipeline

A streaming data pipeline leveraging managed services from AWS & the serverless architecture

Language: Python - Size: 105 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

aws-samples/serverless-streaming-data-application

Code companion to AWS Compute Series on building a serverless backend for a streaming application. Questions? Contact @jbesw.

Language: JavaScript - Size: 467 KB - Last synced: 3 months ago - Pushed: 8 months ago - Stars: 42 - Forks: 22

meni432/redis-article

Embracing Modern Data Solutions: Transitioning from Apache Flink to Redis and AWS Lambda with SQS and Kinesis

Language: JavaScript - Size: 71.4 MB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

gakas14/DynamoDb-Stream-Data-Capture

Using DynamoDb Stream to capture item-level changes in the DynamoDb table, then use kinesis data stream and kinesis firehose to save the changes into a s3 bucket.

Size: 6.84 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

aws-samples/aws-serverless-centralized-logs

The purpose of this sample is to show how to create a serverless and centralized architecture to ingest, store and query logs utilizing Amazon Kinesis Data Firehose, Amazon S3 and Amazon Athena.

Size: 750 KB - Last synced: 3 months ago - Pushed: 4 months ago - Stars: 17 - Forks: 6

aws-samples/saas-metering-system-on-aws

This project shows how to implement a simple SaaS metering system on AWS

Language: Python - Size: 461 KB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 7 - Forks: 1

pranav-patil/aws-kinesis-analytics

AWS Kinesis Analytics gather metrics from various computers (cpu, memory), perform aggregation on Kinesis stream data using Kinesis Analytics (with flink) and store the stream data into AWS S3 bucket which is used by Amazon Athena for running various Analytics queries and rending charts using Grafana.

Language: Java - Size: 1.29 MB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

aws-samples/aws-greengrass-custom-components

This repository hosts custom Greengrass components

Language: Python - Size: 91.8 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

gakas14/Serverless-data-stream-with-kinesis-data-stream-kinesis-firehouse-S3-and-snowflake

Serverless Data pipeline with Api Gateway, AWS lambda, Amazon Kinesis, S3 and Snoflake

Size: 113 KB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

abideenml/RealTime-StarRatingPrediction-with-AWSKinesis

This repository contains an End to End Real time 🕰️ Machine Learning Pipeline to predict star ⭐️ rating of product reviews. This project uses AWS Sagemaker, Kinesis, Lambda, S3, Redshift, Athena, and Step functions. Deployment of multiple models for AB testing and Bandit testing is also included.

Language: Jupyter Notebook - Size: 15.1 MB - Last synced: 5 months ago - Pushed: 6 months ago - Stars: 0 - Forks: 0

gakas14/AWS-Serverless-Data-Lake

This workshop is to build a serverless data lake architecture using Amazon Kinesis Firehose for streaming data ingestion, AWS Glue for Data Integration (ETL, Catalogue Management), Amazon S3 for data lake storage, Amazon Athena for SQL big data analytics.

Language: Jupyter Notebook - Size: 56.6 KB - Last synced: 6 months ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

shirou/prometheus_remote_kinesis 📦

prometheus remote write integration for AWS kinesis

Language: Go - Size: 52.7 KB - Last synced: 21 days ago - Pushed: about 1 year ago - Stars: 9 - Forks: 7

coryodaniel/docker-kinesis-agent

AWS Kinesis Agent on Debian Slim 9

Language: Makefile - Size: 2.93 KB - Last synced: 22 days ago - Pushed: over 4 years ago - Stars: 8 - Forks: 2

AtomGraph/XSLTTransform

XSLT transformation Lambda function for AWS Kinesis Firehose records

Language: Java - Size: 21.5 KB - Last synced: about 1 month ago - Pushed: 8 months ago - Stars: 7 - Forks: 2

alexandremcastro/Data-Lake-Serverless-AWS

Projeto de Big Data, implementando um Data Lake completo em nuvem (Cloud), Serverless, no ecossistema da AWS, utilizando as principais ferramentas (Cloud9, Kinesis, DynamoDB, IAM, S3, Lambda, Glue, Athena).

Size: 5.63 MB - Last synced: 7 months ago - Pushed: about 1 year ago - Stars: 1 - Forks: 0

gerardodavidlopezcastillo/Cloud9KinesisAthena_Public

Streaming data analysis using AWS tools such as Cloud9 to generate events in the cloud, using boto3 to send records to Kinesis Data Firehose to connect to the S3 bucket destination, saving files in .parquet format. With the help of Glue, a data catalog will be created to enable real-time querying of all records with Athena.

Language: Python - Size: 1.3 MB - Last synced: 8 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

leonardohss0/data-streaming-with-kinesis

Keywords: Python, AWS Kinesis, AWS Firehose, AWS Lambda, Snowflake, AWS S3, AWS Athena

Language: Python - Size: 363 KB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 1 - Forks: 0

richhuwtaylor/transforming-streaming-data-for-analytics

A project using AWS to combine and transform streaming data for analytics.

Language: Python - Size: 456 KB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 0 - Forks: 0

thomasdesr/firehose-batcher

Provide a lightweight wrapper to batch record sends to AWS Kinesis Firehose

Language: Go - Size: 1.63 MB - Last synced: 10 months ago - Pushed: over 6 years ago - Stars: 1 - Forks: 1

chrisammon3000/serverless-streaming-reddit-pipeline

Serverless big data streaming application using a fan-out architecture with Lambda and SQS to fetch data from Reddit forums, stream to Kinesis Firehose and store in an S3 data lake.

Language: Python - Size: 994 KB - Last synced: 11 months ago - Pushed: 11 months ago - Stars: 1 - Forks: 0

gs-costa/nearRealTime

Exciting AWS project that combines a streamlined pipeline for near real-time updates. The architecture is powered by essential AWS services like IAM, VPC, RDS, DMS, Kinesis, S3, and Redshift

Language: Python - Size: 38.1 KB - Last synced: 11 months ago - Pushed: 11 months ago - Stars: 1 - Forks: 0

garystafford/kinesis-redshift-streaming-demo

Language: Python - Size: 251 KB - Last synced: 10 months ago - Pushed: over 3 years ago - Stars: 17 - Forks: 5

HsiehShuJeng/dynamodb-streaming-datalake

A demo of DynamoDB CDC into data lake with AWS CDK v2

Language: TypeScript - Size: 256 KB - Last synced: 10 months ago - Pushed: 10 months ago - Stars: 0 - Forks: 0

aws-samples/unified-log-aggregation-and-analytics

The proposed solution shows and approach to unify and centralize logs across different compute platforms like EC2, ECS, EKS and Lambda with Kinesis Data Firehose using log collection agents (EC2 Kinesis agent), log routers (Fluentbit and Firelens) and lambda extension. We can easily deploy the solution presented here on the customer site using the CDK scripts packaged part of this article. Amazon managed Elastic search with Kibana is used to visualize the logs collected across different applications running on different compute platforms.

Language: TypeScript - Size: 8.06 MB - Last synced: 10 months ago - Pushed: 10 months ago - Stars: 8 - Forks: 2

sazzrahman/kinesis-dstream

kinesis data streaming pipelines

Language: Python - Size: 8.79 KB - Last synced: about 1 year ago - Pushed: about 1 year ago - Stars: 0 - Forks: 0

aws-samples/amazon-ecs-fluent-bit-daemon-service

Fluent Bit plugin-based centralized log analysis across Amazon ECS & EKS clusters

Language: Shell - Size: 20.5 KB - Last synced: about 1 year ago - Pushed: over 3 years ago - Stars: 50 - Forks: 23

eduardo3g/twitter

A serverless Twitter built with some cool stuff, such as the Serverless Framework, AppSync (GraphQL), Lambda, DynamoDB, Cognito, Kinesis Firehose, and Algolia ☁️

Language: JavaScript - Size: 1.54 MB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 40 - Forks: 6

homeruntech/target-kinesis 📦

Singer target for Amazon Kinesis and Kinesis Firehose

Language: Python - Size: 42 KB - Last synced: about 1 year ago - Pushed: almost 4 years ago - Stars: 2 - Forks: 3

aws-samples/analyzing-reddit-sentiment-with-aws

Learn how to use Kinesis Firehose, AWS Glue, S3, and Amazon Athena by streaming and analyzing reddit comments in realtime. 100-200 level tutorial.

Language: Python - Size: 3.48 MB - Last synced: about 1 year ago - Pushed: about 3 years ago - Stars: 41 - Forks: 16

kyopark2014/iot-analytics-for-thermometer

It shows IoT analytics infrastructure to gather sensor information using M5Stack, AWS IoT Edukit.

Language: C - Size: 274 MB - Last synced: about 1 month ago - Pushed: 11 months ago - Stars: 8 - Forks: 0

harmatiuk/ingestion_with_kinesis_firehose

The purpose of this repository is to apply a data ingestion with Amazon Kinesis Firehose saving that data to S3 using the boto3. After that I use AWS Glue to catalog and Athena to query the data.

Language: Python - Size: 9.77 KB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

jonrau1/AWS-ComplianceMachineDontStop

Proof of Value Terraform Scripts to utilize Amazon Web Services (AWS) Security, Identity & Compliance Services to Support your AWS Account Security Posture.

Language: HCL - Size: 95.7 KB - Last synced: about 1 year ago - Pushed: about 4 years ago - Stars: 13 - Forks: 12

felipefrizzo/terraform-aws-kinesis-firehose

This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

Language: HCL - Size: 20.5 KB - Last synced: about 1 year ago - Pushed: almost 3 years ago - Stars: 25 - Forks: 22

pedro-tofani/Messaging

IaC to simulate a microsservice hosted in AWS based on messages

Language: Python - Size: 369 KB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

cxcloud/terraform-kinesis-firehose-elasticsearch

Language: HCL - Size: 13.7 KB - Last synced: about 1 year ago - Pushed: over 3 years ago - Stars: 4 - Forks: 4

aws-samples/serverless-realtime-analytics

Demo application on how to create a serverless realtime analytics application using Kinesis Data Streams, Kinesis Firehose, DynamoDB and Lambda, using Cognito to secure the App.

Language: TypeScript - Size: 145 KB - Last synced: about 1 year ago - Pushed: over 3 years ago - Stars: 8 - Forks: 10

bquintas/cross_account_log_exports

Cross account log exports of S3 access logs, CWLogs, Cloudtrail, GuardDuty and Ec2 custom log upload to CWLogs

Size: 193 KB - Last synced: about 1 year ago - Pushed: about 2 years ago - Stars: 0 - Forks: 0

gordonmurray/terraform_aws_ec2_cloudwatch_logs

Creates an EC2 webserver instance in a VPC with logs sent to Cloudwatch and to s3 for long term storage

Language: HCL - Size: 290 KB - Last synced: about 1 year ago - Pushed: about 2 years ago - Stars: 0 - Forks: 0

AtaruOhto/terraform-firelens-firehose-datadog-example

Language: HCL - Size: 4.88 KB - Last synced: about 1 year ago - Pushed: about 2 years ago - Stars: 0 - Forks: 0

aws-samples/aws-cdk-centralize-logs-typescript

This is a CDK project that set up centralized logging to an S3 bucket via a Kinesis Firehose. The solution allows you to specify trusted accounts for different regions and then will configure the CloudWatch Log Delivery Endpoint with proper permission in those regions Solution is based heavily on Centralize Cloudwatch Log with CDK https://github.com/aws-samples/aws-centralize-logs-using-cdk

Language: TypeScript - Size: 338 KB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 1 - Forks: 2