Topic: "stream-processing"
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Language: Python - Size: 133 MB - Last synced at: 4 days ago - Pushed at: 5 days ago - Stars: 50,738 - Forks: 1,454
google-ai-edge/mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
Language: C++ - Size: 578 MB - Last synced at: 4 days ago - Pushed at: 5 days ago - Stars: 32,560 - Forks: 5,666
vectordotdev/vector
A high-performance observability data pipeline.
Language: Rust - Size: 124 MB - Last synced at: 3 days ago - Pushed at: 5 days ago - Stars: 20,946 - Forks: 1,950
zhisheng17/flink-learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Language: Java - Size: 41.6 MB - Last synced at: 7 months ago - Pushed at: 10 months ago - Stars: 14,800 - Forks: 3,940
oxnr/awesome-bigdata
A curated list of awesome big data frameworks, ressources and other awesomeness.
Size: 845 KB - Last synced at: 5 days ago - Pushed at: 28 days ago - Stars: 14,102 - Forks: 2,589
redpanda-data/redpanda
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
Language: C++ - Size: 133 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 11,469 - Forks: 704
madd86/awesome-system-design
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
Size: 1.71 MB - Last synced at: 7 days ago - Pushed at: over 1 year ago - Stars: 11,304 - Forks: 1,235
ThreeDotsLabs/watermill
Building event-driven applications the easy way in Go.
Language: Go - Size: 26.7 MB - Last synced at: 24 days ago - Pushed at: 26 days ago - Stars: 9,280 - Forks: 479
risingwavelabs/risingwave
Streaming data platform. Real-time stream processing, low-latency serving, and Iceberg table management.
Language: Rust - Size: 186 MB - Last synced at: 13 days ago - Pushed at: 13 days ago - Stars: 8,592 - Forks: 714
redpanda-data/connect
Fancy stream processing made operationally mundane
Language: Go - Size: 36.6 MB - Last synced at: 4 days ago - Pushed at: 5 days ago - Stars: 8,536 - Forks: 901
fluent/fluent-bit
Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows
Language: C - Size: 128 MB - Last synced at: 4 days ago - Pushed at: 5 days ago - Stars: 7,521 - Forks: 1,843
robinhood/faust
Python Stream Processing
Language: Python - Size: 8.3 MB - Last synced at: about 2 months ago - Pushed at: over 1 year ago - Stars: 6,823 - Forks: 536
hazelcast/hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
Language: Java - Size: 389 MB - Last synced at: about 8 hours ago - Pushed at: 2 days ago - Stars: 6,540 - Forks: 1,874
MaterializeInc/materialize
The live data layer for apps and AI agents Create up-to-the-second views into your business, just using SQL
Language: Rust - Size: 288 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 6,192 - Forks: 487
apache/hudi
Upserts, Deletes And Incremental Processing on Big Data.
Language: Java - Size: 2.36 GB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 6,047 - Forks: 2,454
online-ml/river
🌊 Online machine learning in Python
Language: Python - Size: 325 MB - Last synced at: 12 days ago - Pushed at: 13 days ago - Stars: 5,641 - Forks: 603
fluvio-community/fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Language: Rust - Size: 34.6 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 5,115 - Forks: 522
javascriptdata/danfojs
Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.
Language: TypeScript - Size: 79.1 MB - Last synced at: 9 days ago - Pushed at: 5 months ago - Stars: 5,034 - Forks: 222
ag2ai/faststream
FastStream is a powerful and easy-to-use asynchronous Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.
Language: Python - Size: 48.3 MB - Last synced at: 1 day ago - Pushed at: 3 days ago - Stars: 4,846 - Forks: 309
ArroyoSystems/arroyo
Distributed stream processing engine in Rust
Language: Rust - Size: 15.6 MB - Last synced at: 12 days ago - Pushed at: 13 days ago - Stars: 4,705 - Forks: 325
memgraph/memgraph
Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.
Language: C++ - Size: 83.4 MB - Last synced at: 11 days ago - Pushed at: 11 days ago - Stars: 3,531 - Forks: 193
manuzhang/awesome-streaming
a curated list of awesome streaming frameworks, applications, etc
Size: 2.72 MB - Last synced at: 6 days ago - Pushed at: 5 months ago - Stars: 2,927 - Forks: 311
PeerDB-io/peerdb
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
Language: Go - Size: 19.9 MB - Last synced at: 3 days ago - Pushed at: 5 days ago - Stars: 2,850 - Forks: 163
douban/dpark 📦
Python clone of Spark, a MapReduce alike framework in Python
Language: Python - Size: 2.65 MB - Last synced at: 4 days ago - Pushed at: almost 5 years ago - Stars: 2,679 - Forks: 530
pipelinedb/pipelinedb
High-performance time-series aggregation for PostgreSQL
Language: C - Size: 46.7 MB - Last synced at: 9 months ago - Pushed at: almost 4 years ago - Stars: 2,643 - Forks: 243
numaproj/numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
Language: Rust - Size: 53.8 MB - Last synced at: 8 days ago - Pushed at: 8 days ago - Stars: 2,419 - Forks: 147
francoispqt/gojay
high performance JSON encoder/decoder with stream API for Golang
Language: Go - Size: 5.61 MB - Last synced at: 7 months ago - Pushed at: about 2 years ago - Stars: 2,130 - Forks: 116
reugn/go-streams
A lightweight stream processing library for Go
Language: Go - Size: 561 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 2,125 - Forks: 172
timeplus-io/proton
⚡ Fastest SQL ETL pipeline in a single C++ binary, built for stream processing, observability, analytics and AI/ML
Language: C++ - Size: 566 MB - Last synced at: 4 days ago - Pushed at: 7 days ago - Stars: 2,103 - Forks: 97
yomorun/yomo
🦖 Serverless AI Agent Framework with Geo-distributed Edge AI Infra.
Language: Go - Size: 66 MB - Last synced at: 18 days ago - Pushed at: 21 days ago - Stars: 1,873 - Forks: 139
bytewax/bytewax
Python Stream Processing
Language: Python - Size: 12 MB - Last synced at: 2 months ago - Pushed at: 9 months ago - Stars: 1,861 - Forks: 96
nerevu/riko
A Python stream processing engine modeled after Yahoo! Pipes
Language: Python - Size: 2.58 MB - Last synced at: 4 months ago - Pushed at: almost 4 years ago - Stars: 1,599 - Forks: 75
siddhi-io/siddhi
Stream Processing and Complex Event Processing Engine
Language: Java - Size: 33.6 MB - Last synced at: 5 months ago - Pushed at: 7 months ago - Stars: 1,554 - Forks: 531
quixio/quix-streams
Python Streaming DataFrames for Kafka
Language: Python - Size: 9.64 MB - Last synced at: 22 days ago - Pushed at: about 1 month ago - Stars: 1,494 - Forks: 91
WallarooLabs/wally
Distributed Stream Processing
Language: Pony - Size: 268 MB - Last synced at: 7 months ago - Pushed at: over 4 years ago - Stars: 1,481 - Forks: 68
halaxa/json-machine
Efficient, easy-to-use, and fast PHP JSON stream parser
Language: PHP - Size: 1.51 MB - Last synced at: 16 days ago - Pushed at: 20 days ago - Stars: 1,271 - Forks: 73
spring-attic/spring-cloud-dataflow 📦
A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
Language: Java - Size: 68.3 MB - Last synced at: 5 months ago - Pushed at: 8 months ago - Stars: 1,136 - Forks: 591
hazelcast/hazelcast-jet 📦
Distributed Stream and Batch Processing
Language: Java - Size: 171 MB - Last synced at: 2 months ago - Pushed at: about 1 year ago - Stars: 1,110 - Forks: 207
uhop/stream-json
The micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API.
Language: JavaScript - Size: 899 KB - Last synced at: 3 months ago - Pushed at: 11 months ago - Stars: 1,108 - Forks: 50
spring-cloud/spring-cloud-stream
Framework for building Event-Driven Microservices
Language: Java - Size: 19.3 MB - Last synced at: 3 days ago - Pushed at: 7 days ago - Stars: 1,056 - Forks: 629
clj-commons/manifold
A compatibility layer for event-driven abstractions
Language: Clojure - Size: 1.5 MB - Last synced at: 12 days ago - Pushed at: about 1 month ago - Stars: 1,039 - Forks: 109
edp963/wormhole
Wormhole is a SPaaS (Stream Processing as a Service) Platform
Language: JavaScript - Size: 45.5 MB - Last synced at: 7 months ago - Pushed at: about 3 years ago - Stars: 978 - Forks: 479
ververica/flink-sql-cookbook
The Apache Flink SQL Cookbook is a curated collection of examples, patterns, and use cases of Apache Flink SQL. Many of the recipes are completely self-contained and can be run in Ververica Platform as is.
Language: Dockerfile - Size: 14.1 MB - Last synced at: 3 months ago - Pushed at: about 3 years ago - Stars: 907 - Forks: 209
nodefluent/kafka-streams
equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:
Language: TypeScript - Size: 4.14 MB - Last synced at: 2 months ago - Pushed at: almost 3 years ago - Stars: 839 - Forks: 111
vladimirvivien/automi
A stream processing API for Go (now with generic type support)
Language: Go - Size: 2.72 MB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 799 - Forks: 63
gazette/core
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
Language: Go - Size: 5.02 MB - Last synced at: 3 days ago - Pushed at: 5 days ago - Stars: 780 - Forks: 57
pipeless-ai/pipeless
An open-source computer vision framework to build and deploy apps in minutes
Language: Rust - Size: 142 MB - Last synced at: 9 days ago - Pushed at: over 1 year ago - Stars: 774 - Forks: 41
gearpump/gearpump
Lightweight real-time big data streaming engine over Akka
Language: Scala - Size: 26.2 MB - Last synced at: 5 days ago - Pushed at: almost 4 years ago - Stars: 758 - Forks: 152
lw-lin/streaming-readings
Streaming System 相关的论文读物
Size: 6.84 KB - Last synced at: 7 months ago - Pushed at: almost 4 years ago - Stars: 733 - Forks: 154
hstreamdb/hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
Language: Haskell - Size: 6.28 MB - Last synced at: about 2 months ago - Pushed at: 12 months ago - Stars: 727 - Forks: 55
TouK/nussknacker
Low-code tool for automating actions on real time data | Stream processing for the users.
Language: Scala - Size: 206 MB - Last synced at: about 6 hours ago - Pushed at: 1 day ago - Stars: 703 - Forks: 103
apache/streampipes
Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.
Language: Java - Size: 315 MB - Last synced at: 5 days ago - Pushed at: 6 days ago - Stars: 695 - Forks: 215
infoslack/awesome-kafka
A list about Apache Kafka
Size: 96.7 KB - Last synced at: about 1 month ago - Pushed at: 10 months ago - Stars: 583 - Forks: 165
microsoft/psi
Platform for Situated Intelligence
Language: C# - Size: 37.2 MB - Last synced at: 13 days ago - Pushed at: about 1 month ago - Stars: 565 - Forks: 103
leesf/hudi-resources
汇总Apache Hudi相关资料
Size: 32.2 MB - Last synced at: 13 days ago - Pushed at: 14 days ago - Stars: 561 - Forks: 160
deep5050/radio-active
Play any radios around the globe right from the terminal :zap:
Language: Python - Size: 553 KB - Last synced at: 3 months ago - Pushed at: 11 months ago - Stars: 557 - Forks: 29
constellation-rs/amadeus
Harmonious distributed data analysis in Rust.
Language: Rust - Size: 2.46 MB - Last synced at: about 2 months ago - Pushed at: over 4 years ago - Stars: 482 - Forks: 25
nmaquet/kasper 📦
Kasper is a lightweight library for processing Kafka topics.
Language: Go - Size: 292 KB - Last synced at: 6 days ago - Pushed at: 10 months ago - Stars: 438 - Forks: 23
FutureSDR/FutureSDR
An Async SDR Runtime for Heterogeneous Architectures
Language: Rust - Size: 28.1 MB - Last synced at: 10 days ago - Pushed at: 12 days ago - Stars: 412 - Forks: 67
Kurento/kurento
Kurento WebRTC Media Server
Language: C - Size: 132 MB - Last synced at: 27 days ago - Pushed at: 27 days ago - Stars: 412 - Forks: 76
smooks/smooks
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
Language: Java - Size: 29.2 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 406 - Forks: 360
RedisGears/RedisGears
Dynamic execution framework for your Redis data
Language: Rust - Size: 4.92 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 380 - Forks: 67
Point72/csp
csp is a high performance reactive stream processing library, written in C++ and Python
Language: Python - Size: 5.36 MB - Last synced at: 5 days ago - Pushed at: 6 days ago - Stars: 373 - Forks: 75
graphform/swim-rust
Self-contained distributed software platform for building stateful, massively real-time streaming applications in Rust.
Language: Rust - Size: 25.2 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 352 - Forks: 12
p2panda/p2panda
All the things a panda needs
Language: Rust - Size: 3.94 MB - Last synced at: 3 days ago - Pushed at: 4 days ago - Stars: 348 - Forks: 25
AICoE/log-anomaly-detector 📦
Log Anomaly Detection - Machine learning to detect abnormal events logs
Language: Jupyter Notebook - Size: 18.7 MB - Last synced at: 28 days ago - Pushed at: over 2 years ago - Stars: 339 - Forks: 137
memgraph/mage
MAGE - Memgraph Advanced Graph Extensions :crystal_ball:
Language: C++ - Size: 22.5 MB - Last synced at: 7 days ago - Pushed at: 9 days ago - Stars: 325 - Forks: 34
AxisCommunications/media-stream-library-js 📦
JavaScript library to handle media streams on the command line (Node.js) and in the browser.
Language: TypeScript - Size: 872 MB - Last synced at: 2 months ago - Pushed at: 3 months ago - Stars: 311 - Forks: 105
vectorengine/vectorsql 📦
VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.
Language: Go - Size: 1.36 MB - Last synced at: 8 months ago - Pushed at: about 4 years ago - Stars: 292 - Forks: 53
Chulong-Li/Real-time-Sentiment-Tracking-on-Twitter-for-Brand-Improvement-and-Trend-Recognition
A real-time interactive web app based on data pipelines using streaming Twitter data, automated sentiment analysis, and MySQL&PostgreSQL database (Deployed on Heroku)
Language: Jupyter Notebook - Size: 5.78 MB - Last synced at: about 1 year ago - Pushed at: over 5 years ago - Stars: 283 - Forks: 127
streamnative/pulsar-flink 📦
Elastic data processing with Apache Pulsar and Apache Flink
Language: Java - Size: 2.16 MB - Last synced at: 5 months ago - Pushed at: about 3 years ago - Stars: 279 - Forks: 120
raystack/dagger
Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.
Language: Java - Size: 12 MB - Last synced at: 22 days ago - Pushed at: over 2 years ago - Stars: 277 - Forks: 42
confluentinc/ksql
The database purpose-built for stream processing applications.
Language: Java - Size: 226 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 273 - Forks: 1,042
bbejeck/kafka-streams-in-action
Source code for the Kafka Streams in Action Book
Language: Java - Size: 176 MB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 251 - Forks: 177
cross-platform/dspatch
The Refreshingly Simple Cross-Platform C++ Dataflow / Patching / Pipelining / Graph Processing / Stream Processing / Reactive Programming Framework
Language: C++ - Size: 6.15 MB - Last synced at: 14 days ago - Pushed at: 15 days ago - Stars: 249 - Forks: 45
talariadb/talaria
TalariaDB is a distributed, highly available, and low latency time-series database for Presto
Language: Go - Size: 12.8 MB - Last synced at: 3 months ago - Pushed at: almost 2 years ago - Stars: 227 - Forks: 32
streamnative/function-mesh
The serverless framework purpose-built for event streaming applications.
Language: Go - Size: 55.8 MB - Last synced at: 15 days ago - Pushed at: 15 days ago - Stars: 224 - Forks: 29
schollz/pluck
Pluck text in a fast and intuitive way :rooster:
Language: Go - Size: 6.55 MB - Last synced at: 2 months ago - Pushed at: over 6 years ago - Stars: 215 - Forks: 6
TomasMikula/libretto
Declarative concurrency and stream processing library for Scala
Language: Scala - Size: 15.5 MB - Last synced at: 5 days ago - Pushed at: 8 days ago - Stars: 213 - Forks: 7
logrange/logrange
High performance data aggregating storage
Language: Go - Size: 69.2 MB - Last synced at: 9 months ago - Pushed at: almost 2 years ago - Stars: 191 - Forks: 16
miroslavpejic85/mirotalkbro
📡 MiroTalk WebRTC Live Broadcast allows to broadcast live video, audio and screen stream to all connected users (viewers).
Language: JavaScript - Size: 1.89 MB - Last synced at: 10 days ago - Pushed at: 11 days ago - Stars: 186 - Forks: 47
neo4j-contrib/neo4j-streams
Neo4j Kafka Connector
Language: Kotlin - Size: 3.82 MB - Last synced at: 27 days ago - Pushed at: about 1 month ago - Stars: 179 - Forks: 74
bytewatch/dolphinbeat
A server that pulls and parses MySQL binlog, pushs change data into different sinks like Kafka.
Language: Go - Size: 2.97 MB - Last synced at: about 1 year ago - Pushed at: almost 7 years ago - Stars: 176 - Forks: 30
Claviz/xlstream
Turns XLSX into a readable stream.
Language: TypeScript - Size: 641 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 175 - Forks: 19
pemrouz/fero
light, fast, scalable, streaming microservices made easy
Language: JavaScript - Size: 1.05 MB - Last synced at: 5 months ago - Pushed at: over 7 years ago - Stars: 174 - Forks: 4
akka/akka-stream-contrib
Add-ons to Akka Stream
Language: Scala - Size: 315 KB - Last synced at: 4 months ago - Pushed at: over 2 years ago - Stars: 173 - Forks: 78
vsouza/go-kafka-example
Golang Kafka consumer and producer example
Language: Go - Size: 505 KB - Last synced at: about 2 months ago - Pushed at: about 2 years ago - Stars: 172 - Forks: 59
fluxus-labs/fluxus
Fluxus Stream Processing Engine
Language: Rust - Size: 5.07 MB - Last synced at: 28 days ago - Pushed at: 3 months ago - Stars: 169 - Forks: 22
GridProtectionAlliance/gsf
Grid Solutions Framework
Language: C# - Size: 249 MB - Last synced at: 12 days ago - Pushed at: 13 days ago - Stars: 168 - Forks: 73
mariomac/gostream
A Go port of the Java Streams API. Type-safe and functional Go Streams processing
Language: Go - Size: 297 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 167 - Forks: 11
whitaker-io/machine
Machine is a workflow/pipeline library for processing data
Language: Go - Size: 1.39 MB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 164 - Forks: 12
LearningJournal/Kafka-Streams-Real-time-Stream-Processing
This is the central repository for all materials related to Kafka Streams : Real-time Stream Processing! Book by Prashant Pandey.
Language: Java - Size: 5.25 MB - Last synced at: 9 months ago - Pushed at: over 5 years ago - Stars: 164 - Forks: 163
synacker/daggy
Daggy - Data Aggregation Utility and C/C++ developer library for data streams catching
Language: C++ - Size: 7.36 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 157 - Forks: 16
psila-ai/callysto
Stream processing & Service framework.
Language: Rust - Size: 3.67 MB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 156 - Forks: 9
GridProtectionAlliance/openPDC
Open Source Phasor Data Concentrator
Language: C# - Size: 2.66 GB - Last synced at: 15 days ago - Pushed at: 16 days ago - Stars: 150 - Forks: 58
ElAlev/Wayeb
Wayeb is a Complex Event Processing and Forecasting (CEP/F) engine written in Scala.
Language: Scala - Size: 8.17 MB - Last synced at: almost 2 years ago - Pushed at: about 2 years ago - Stars: 147 - Forks: 6
samsara/samsara
Samsara is a real-time analytics platform
Language: Clojure - Size: 22.8 MB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 147 - Forks: 17
runreveal/kawa
A blazingly fast event stream processing library powering the reveald event processing daemon.
Language: Go - Size: 1 MB - Last synced at: 27 days ago - Pushed at: about 1 month ago - Stars: 144 - Forks: 3
zakariamaaraki/RemoteCodeCompiler
An online code compiler supporting 11 programming languages (Java, Kotlin, Scala, C, C++, C#, Golang, Python, Ruby, Rust and Haskell) for competitive programming and coding interviews.
Language: Java - Size: 3.34 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 135 - Forks: 47
mjwestcott/runnel
Distributed event processing for Python based on Redis Streams
Language: Python - Size: 6.58 MB - Last synced at: 10 months ago - Pushed at: over 5 years ago - Stars: 133 - Forks: 11
