apache-beam | Topic | Ecosyste.ms: Repos

Topic: "apache-beam"

tensorflow/tfx

TFX is an end-to-end platform for deploying production ML pipelines

Language: Python - Size: 234 MB - Last synced at: 3 days ago - Pushed at: 9 days ago - Stars: 2,141 - Forks: 724

GoogleCloudPlatform/DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

Language: Java - Size: 23.8 MB - Last synced at: about 17 hours ago - Pushed at: about 18 hours ago - Stars: 1,199 - Forks: 1,009

nielsbasjes/yauaa

Yet Another UserAgent Analyzer

Language: Java - Size: 81.8 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 816 - Forks: 138

GoogleCloudPlatform/flink-on-k8s-operator 📦

[DEPRECATED] Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

Language: Go - Size: 1.67 MB - Last synced at: 14 days ago - Pushed at: over 2 years ago - Stars: 658 - Forks: 266

blockchain-etl/bitcoin-etl

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Language: Python - Size: 354 KB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 421 - Forks: 127

google/weather-tools

Tools to make weather data accessible and useful.

Language: Python - Size: 196 MB - Last synced at: about 23 hours ago - Pushed at: 1 day ago - Stars: 227 - Forks: 43

spotify/flink-on-k8s-operator

Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

Language: Go - Size: 3.08 MB - Last synced at: 10 days ago - Pushed at: 10 days ago - Stars: 212 - Forks: 73

google/tensorflow-recorder 📦

TFRecorder makes it easy to create TensorFlow records (TFRecords) from Pandas DataFrames and CSVs files containing images or structured data.

Language: Python - Size: 6.54 MB - Last synced at: 14 days ago - Pushed at: almost 3 years ago - Stars: 183 - Forks: 32

google/fhir-data-pipes

A collection of tools for extracting FHIR resources and analytics services on top of that data.

Language: Java - Size: 386 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 176 - Forks: 98

ngrunwald/datasplash

Clojure API for a more dynamic Google Dataflow

Language: Clojure - Size: 853 KB - Last synced at: 8 days ago - Pushed at: 16 days ago - Stars: 131 - Forks: 32

mohaseeb/beam-nuggets

Collection of transforms for the Apache beam python SDK.

Language: Python - Size: 6.19 MB - Last synced at: 6 days ago - Pushed at: over 1 year ago - Stars: 89 - Forks: 39

blockchain-etl/blockchain-etl-streaming

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Language: Python - Size: 64.5 KB - Last synced at: about 21 hours ago - Pushed at: over 3 years ago - Stars: 80 - Forks: 22

tosun-si/asgarde

Asgarde allows simplifying error handling with Apache Beam Java, with less code, more concise and expressive code.

Language: Java - Size: 159 KB - Last synced at: 3 months ago - Pushed at: 9 months ago - Stars: 74 - Forks: 5

mercari/pipeline

Tool to define Apache Beam pipeline in YAML or JSON

Language: Java - Size: 3.44 MB - Last synced at: 1 day ago - Pushed at: 2 days ago - Stars: 72 - Forks: 22

Fematich/mlengine-boilerplate

Repository to quickly get you started with new Machine Learning projects on Google Cloud Platform. More info(slides):

Language: Python - Size: 54.7 KB - Last synced at: over 1 year ago - Pushed at: over 6 years ago - Stars: 63 - Forks: 18

yu-iskw/bigquery-to-datastore 📦

Export a whole BigQuery table to Google Datastore with Apache Beam/Google Dataflow

Language: Java - Size: 93.8 KB - Last synced at: about 1 year ago - Pushed at: over 4 years ago - Stars: 58 - Forks: 18

xmlking/micro-apps

Microservices in Post-Kubernetes Era. A polyglot monorepo

Language: Kotlin - Size: 5.77 MB - Last synced at: 30 days ago - Pushed at: over 1 year ago - Stars: 53 - Forks: 10

luisbelloch/data_processing_course

Some class materials for a data processing course using PySpark

Language: Python - Size: 563 KB - Last synced at: almost 2 years ago - Pushed at: over 2 years ago - Stars: 51 - Forks: 24

blockchain-etl/blockchain-etl-architecture

Blockchain ETL Architecture

Size: 101 KB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 47 - Forks: 14

doitintl/banias

Opinionated serverless event analytics pipeline

Language: Go - Size: 416 KB - Last synced at: 3 days ago - Pushed at: about 2 years ago - Stars: 43 - Forks: 16

tosun-si/pasgarde

Asgarde allows simplifying error handling with Apache Beam Python, with less code, more concise and expressive code.

Language: Python - Size: 61.5 KB - Last synced at: 5 days ago - Pushed at: almost 3 years ago - Stars: 31 - Forks: 1

asaharland/beam-pipeline-examples

Apache Beam examples for running on Google Cloud Dataflow.

Language: Java - Size: 59.5 MB - Last synced at: almost 2 years ago - Pushed at: over 6 years ago - Stars: 29 - Forks: 14

NucleusEngineering/hack-your-pipe

Efficient streaming data ingestion, transformation & activation

Language: Python - Size: 3.01 MB - Last synced at: 9 months ago - Pushed at: about 2 years ago - Stars: 28 - Forks: 3

google-parfait/dataset_grouper

Libraries for efficient and scalable group-structured dataset pipelines.

Language: Python - Size: 59.6 KB - Last synced at: 7 days ago - Pushed at: 5 months ago - Stars: 25 - Forks: 3

mozilla-services/foxsec-pipeline

Log analysis pipeline utilizing Apache Beam

Language: Java - Size: 13.5 MB - Last synced at: 22 days ago - Pushed at: almost 2 years ago - Stars: 25 - Forks: 6

mercari/DataflowTemplates

Convenient Dataflow pipelines for transforming data between cloud data sources

Language: Java - Size: 97.7 KB - Last synced at: 9 days ago - Pushed at: about 1 year ago - Stars: 24 - Forks: 12

sayakpaul/count-tokens-hf-datasets

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

Language: Python - Size: 19.5 KB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 24 - Forks: 1

pysql-beam-org/pysql-beam

Language: Python - Size: 2.13 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 23 - Forks: 12

google/consent-based-conversion-adjustments

Code to statistically up-weight conversion values of consenting customers to feed up to 100% of the factual conversion values back into Google Ads.

Language: Python - Size: 62.5 KB - Last synced at: 14 days ago - Pushed at: 9 months ago - Stars: 22 - Forks: 8

janaom/gcp-data-engineering-etl-with-composer-dataflow

This project leverages GCS, Composer, Dataflow, BigQuery, and Looker on Google Cloud Platform (GCP) to build a robust data engineering solution for processing, storing, and reporting daily transaction data in the online food delivery industry.

Language: Python - Size: 290 KB - Last synced at: 21 days ago - Pushed at: over 1 year ago - Stars: 22 - Forks: 13

O2-Czech-Republic/proxima-platform

The Proxima platform.

Language: Java - Size: 9.3 MB - Last synced at: 9 days ago - Pushed at: 9 days ago - Stars: 21 - Forks: 7

esakik/beam-mysql-connector

An Apache Beam I/O connector for seamless integration with MySQL database 🔗 https://beam.apache.org/documentation/io/connectors/#other-io-connectors-for-apache-beam

Language: Python - Size: 188 KB - Last synced at: 19 days ago - Pushed at: 10 months ago - Stars: 21 - Forks: 19

datastacktv/apache-beam-explained

Source code for the YouTube video, Apache Beam Explained in 12 Minutes

Language: Python - Size: 2.93 KB - Last synced at: 3 days ago - Pushed at: over 4 years ago - Stars: 21 - Forks: 14

Ganeshsivakumar/langchain-beam

Integrates LLMs as PTransform in Apache Beam pipelines using LangChain

Language: Java - Size: 1.35 MB - Last synced at: 5 days ago - Pushed at: 2 months ago - Stars: 20 - Forks: 1

carted/processing-text-data

Presents an optimized Apache Beam pipeline for generating sentence embeddings (runnable on Cloud Dataflow).

Language: Python - Size: 16.6 KB - Last synced at: about 2 years ago - Pushed at: about 3 years ago - Stars: 20 - Forks: 6

cassiobolba/Data-Engineering

Projects and studies regarding Data Engineering Area

Language: HTML - Size: 13.1 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 18 - Forks: 3

datastacktv/apache-beam-batch-processing

Public source code for the Batch Processing with Apache Beam (Python) online course

Language: Python - Size: 81.1 KB - Last synced at: 3 days ago - Pushed at: over 4 years ago - Stars: 18 - Forks: 9

delftdata/stateflow

Prototype which extracts stateful dataflows by analysing Python code.

Language: Python - Size: 3.99 MB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 17 - Forks: 4

GoogleCloudPlatform/dataflow-pubsub-dedup

Language: Java - Size: 67.4 KB - Last synced at: 14 days ago - Pushed at: 7 months ago - Stars: 16 - Forks: 10

tosun-si/midgard

Midgard is a wrapper on Beam Kotlin, allowing more concise and expressive code. It removes Beam boilerplate code and proposes more Functional Programming style

Language: Kotlin - Size: 97.7 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 16 - Forks: 1

medzin/beam-postgres

Light IO transforms for Postgres read/write in Apache Beam pipelines.

Language: Python - Size: 46.9 KB - Last synced at: 28 days ago - Pushed at: 3 months ago - Stars: 14 - Forks: 4

yu-iskw/kuromoji-for-bigquery 📦

Tokenize Japanese text on BigQuery with Kuromoji in Apache Beam/Google Dataflow at scale

Language: Java - Size: 11 MB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 14 - Forks: 3

RajeshHegde/apache-beam-example

Apache Beam example project

Language: Python - Size: 8.79 KB - Last synced at: over 1 year ago - Pushed at: over 5 years ago - Stars: 13 - Forks: 14

mmziyad/beam-scala-examples

Scala examples for using Apache Beam Java API (2.1.0)

Language: Scala - Size: 10.7 KB - Last synced at: over 1 year ago - Pushed at: over 7 years ago - Stars: 13 - Forks: 6

chermenin/kio

Kotlin extensions for Apache Beam

Language: Kotlin - Size: 1.28 MB - Last synced at: 18 days ago - Pushed at: 5 months ago - Stars: 12 - Forks: 1

ntakouris/fluent-tfx

A fluent API layer for tensorflow extended e2e machine learning pipelines

Language: Python - Size: 534 KB - Last synced at: 7 days ago - Pushed at: over 4 years ago - Stars: 12 - Forks: 0

japila-books/apache-beam-internals

The Internals of Apache Beam

Size: 795 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 12 - Forks: 4

GianlucaMancusi/COVID-19-Apache-Beam-Statistics

Statistical processing of COVID-19 data using Apache Beam for Google Cloud Dataflow in Python. Project for the exam of "Sistemi ed Applicazioni Cloud" (2019-20), Magistrale di Ingegneria Informatica at the Dipartimento di Ingegneria Enzo Ferrari.

Language: Python - Size: 322 KB - Last synced at: about 2 years ago - Pushed at: about 5 years ago - Stars: 12 - Forks: 0

janaom/gcp-de-project-streaming-pubsub-beam-dataflow

This project demonstrates an end-to-end solution for processing and analyzing real-time conversations data from a JSON file using GCP services and infrastructure automation, showcasing data storage, streaming, processing, and analysis at scale.

Language: Python - Size: 172 KB - Last synced at: 21 days ago - Pushed at: about 1 year ago - Stars: 10 - Forks: 1

prakashdontaraju/google-cloud-ecommerce

ecommerce GCP Streaming pipeline ― Cloud Storage, Compute Engine, Pub/Sub, Dataflow, Apache Beam, BigQuery and Tableau; GCP Batch pipeline ― Cloud Storage, Dataproc, PySpark, Cloud Spanner and Tableau

Language: Python - Size: 4.38 MB - Last synced at: 9 months ago - Pushed at: about 3 years ago - Stars: 10 - Forks: 3

sfeir-open-source/kbeam

Idiomatic Kotlin Pipelines for Apache Beam

Language: Kotlin - Size: 57.6 KB - Last synced at: about 1 year ago - Pushed at: almost 4 years ago - Stars: 10 - Forks: 0

blockchain-etl/eos-etl

ETL scripts for EOS.

Language: Python - Size: 226 KB - Last synced at: 11 days ago - Pushed at: almost 2 years ago - Stars: 9 - Forks: 7

mkuthan/example-beam

Playground for Apache Beam and Scio experiments, driven by real-world use cases.

Language: Scala - Size: 190 KB - Last synced at: 27 days ago - Pushed at: over 4 years ago - Stars: 9 - Forks: 3

mkuthan/stream-processing

Learn how to develop and test stateful streaming and batch data pipelines

Language: Scala - Size: 24.9 MB - Last synced at: 27 days ago - Pushed at: 2 months ago - Stars: 8 - Forks: 2

ksalama/data2cooc2emb2ann

Learning embeddings from item co-occurrence statistics, and building an approx. nearest neighbour index

Language: Jupyter Notebook - Size: 1.27 MB - Last synced at: 28 days ago - Pushed at: about 2 years ago - Stars: 8 - Forks: 1

blockchain-etl/hedera-etl

ETL scripts for Hedera Hashgraph

Language: Java - Size: 293 KB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 8 - Forks: 3

dsnair/GCP

Tutorials on Google Cloud Platform

Language: Jupyter Notebook - Size: 21.6 MB - Last synced at: 4 months ago - Pushed at: almost 7 years ago - Stars: 8 - Forks: 1

sanderploegsma/beam-scheduling-kubernetes

Scheduled Dataflow pipelines using Kubernetes Cronjobs

Language: Kotlin - Size: 59.6 KB - Last synced at: 4 days ago - Pushed at: about 7 years ago - Stars: 8 - Forks: 2

beamsummit/beam-intro-las-vegas-2019 📦

Hands-on "Workshop" Offered on 11 September 2019 at Beam Summit (Las Vegas).

Language: Java - Size: 1.31 MB - Last synced at: about 2 years ago - Pushed at: almost 4 years ago - Stars: 7 - Forks: 1

rrmerugu-archive/apache-beam-io-extras

The missing I/O PTransforms of Apache Beam in python; which already exist in Java SDK based but not yet supported in the official apache-beam module.

Language: Python - Size: 12.7 KB - Last synced at: 7 days ago - Pushed at: over 6 years ago - Stars: 7 - Forks: 6

redvg/streaming-pipeline-IoT-simulator

ETL pipeline on GCP

Language: Jupyter Notebook - Size: 1.14 MB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 7 - Forks: 3

mkuthan/gcp-dataflow-tampermonkey

Tampermonkey script for GCP Dataflow console with enhanced view for finding job bottlenecks

Language: JavaScript - Size: 1.17 MB - Last synced at: 27 days ago - Pushed at: over 4 years ago - Stars: 6 - Forks: 0

regadas/scio-cats

leverage cats type classes and data types in scio pipelines

Language: Scala - Size: 1.09 MB - Last synced at: about 1 month ago - Pushed at: about 2 months ago - Stars: 5 - Forks: 2

thecodemancer/study-with-me

Lots of code, resources, examples, some graphs and so much fun ahead!

Language: Jupyter Notebook - Size: 10.3 MB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 5 - Forks: 0

johannaojeling/go-beam-pipeline

Data pipeline built with the Apache Beam Go SDK

Language: Go - Size: 887 KB - Last synced at: 26 days ago - Pushed at: about 2 years ago - Stars: 5 - Forks: 1

arkady-emelyanov/toy-data-platform

Toy data platform for a company that provides web analytics

Language: HCL - Size: 485 KB - Last synced at: about 1 month ago - Pushed at: about 4 years ago - Stars: 5 - Forks: 0

picnicml/doddle-beam-example

:icecream: Serve doddle-model in a pipeline implemented with Apache Beam

Language: Scala - Size: 50.8 KB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 5 - Forks: 0

vicenteg/dataflow-example

Language: Java - Size: 250 KB - Last synced at: over 1 year ago - Pushed at: over 6 years ago - Stars: 5 - Forks: 6

davideanastasia/apache-beam-getting-started

Getting Started with Apache Beam: inverted index

Language: Java - Size: 15.6 KB - Last synced at: about 1 year ago - Pushed at: almost 7 years ago - Stars: 5 - Forks: 2

miseyu/luigi-etl

luigi app for own

Language: Python - Size: 17.6 KB - Last synced at: over 1 year ago - Pushed at: over 8 years ago - Stars: 5 - Forks: 0

SolaceProducts/solace-apache-beam

Solace connector for Apache Beam / Google Cloud Dataflow

Language: Java - Size: 3.3 MB - Last synced at: 28 days ago - Pushed at: about 1 year ago - Stars: 4 - Forks: 14

mitchelllisle/beam-sink 📦

☀️ An Apache Beam Sink Library for Databases and other Sinks

Language: Python - Size: 96.7 KB - Last synced at: 8 days ago - Pushed at: about 2 years ago - Stars: 4 - Forks: 0

Rajeshwari-Rudra/apache_beam-python

A demo project on batch data-parallel processing using Apache Beam and Python

Language: Jupyter Notebook - Size: 4.08 MB - Last synced at: over 1 year ago - Pushed at: about 4 years ago - Stars: 4 - Forks: 3

sb2nov/beam Fork of apache/beam

Mirror of Apache Beam

Language: Java - Size: 40.9 MB - Last synced at: about 1 year ago - Pushed at: over 7 years ago - Stars: 4 - Forks: 0

amitsela/beam Fork of apache/beam

Mirror of Apache Beam

Language: Java - Size: 30 MB - Last synced at: about 2 years ago - Pushed at: about 8 years ago - Stars: 4 - Forks: 0

BeamStackProj/beamstack-cli

Language: Go - Size: 95.3 MB - Last synced at: 4 months ago - Pushed at: 8 months ago - Stars: 3 - Forks: 0

tosun-si/world-cup-qatar-team-stats-kotlin-midgard

This application shows a full Apache Beam pipeline with Kotlin and Midgard library. The use case works on the last Qatar FIFA world cup data and calculate players statistics per team. This application will be presented at Beam Summit 2023 in New York

Language: Kotlin - Size: 851 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 3 - Forks: 1

arthurpessoa/kotlin-beam-starter

Kotlin Apache-Beam Starter project

Language: Kotlin - Size: 1.15 MB - Last synced at: 12 months ago - Pushed at: 12 months ago - Stars: 3 - Forks: 0

eliias/gleam

Fun DSL for Apache Beam and Kotlin.

Language: Kotlin - Size: 2.72 MB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 0

ashleycheng/house-price-etl-pipeline

Building house price data pipelines with Apache Beam and Spark on GCP

Language: Python - Size: 14.6 KB - Last synced at: 10 months ago - Pushed at: over 3 years ago - Stars: 3 - Forks: 1

blockchain-etl/anomalous-transactions-detector-dataflow

Dataflow pipeline for detecting anomalous transactions on the Ethereum and Bitcoin blockchains

Language: Java - Size: 41 KB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 3 - Forks: 1

goatcheesesaladwithpeanutoildressing/hashtagsbattle 📦

IIM-DEVOPS - (demo) Real-time Twitter's hashtags analytics... fully managed by Google Cloud

Language: Python - Size: 827 KB - Last synced at: 8 days ago - Pushed at: almost 5 years ago - Stars: 3 - Forks: 0

gjbae1212/go-apachebeam-gzipio

To transform for reading and writing gzip files in apache beam using Golang.

Language: Go - Size: 32.2 KB - Last synced at: 3 months ago - Pushed at: about 5 years ago - Stars: 3 - Forks: 1

goatcheesesaladwithpeanutoildressing/beam-amazon-batch-example

A practical example of batch processing on Google Cloud Dataflow using the Go SDK for Apache Beam :fire:

Language: Go - Size: 455 KB - Last synced at: 8 days ago - Pushed at: over 5 years ago - Stars: 3 - Forks: 0

sfujiwara/dataflow-gcs2gdrive

An implementation of Dataflow Template copying files from Google Cloud Storage to Google Drive

Language: Python - Size: 6.84 KB - Last synced at: about 1 year ago - Pushed at: over 6 years ago - Stars: 3 - Forks: 1

ryanmcdowell/dataflow-pubsub-event-router

An example pipeline which re-publishes events to different topics based a message attribute.

Language: Java - Size: 11.7 KB - Last synced at: about 2 months ago - Pushed at: over 6 years ago - Stars: 3 - Forks: 0

mbari-org/aipipeline

Library for running detection, clustering or classification ai pipelines plus performance monitoring using ApacheBeam

Language: Jupyter Notebook - Size: 77.4 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 2 - Forks: 0

beam-pyio/firehose_pyio

Apache Beam Python I/O connector for Amazon Data Firehose

Language: Python - Size: 2.83 MB - Last synced at: 5 days ago - Pushed at: 8 months ago - Stars: 2 - Forks: 0

GoogleCloudPlatform/dataflow-metrics-exporter

CLI tool to collect dataflow resource & execution metrics and export to either BigQuery or Google Cloud Storage. Tool will be useful to compare & visualize the metrics while benchmarking the dataflow pipelines using various data formats, resource configurations etc

Language: Java - Size: 65.4 KB - Last synced at: 14 days ago - Pushed at: 12 months ago - Stars: 2 - Forks: 3

tosun-si/dataflow-java-ci-cd

Project showing a CI CD pipeline for Dataflow Java with Flex Template and Cloud Build

Language: Java - Size: 3.31 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 2 - Forks: 0

hwf87/hwf-search-etl

This is a data scraping project that sources data from the Houzz e-commerce platform, the CNN YouTube channel, and the TedTalk official website. The implementation uses the Apache Beam framework to build an ETL pipeline and write the results into an Elasticsearch database. The final step visualizes the crawler results using Kibana.

Language: HTML - Size: 951 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 2 - Forks: 0