Topic: "stream-processing"
google-ai-edge/mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
Language: C++ - Size: 576 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 30,373 - Forks: 5,419
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Language: Python - Size: 133 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 28,469 - Forks: 627
vectordotdev/vector
A high-performance observability data pipeline.
Language: Rust - Size: 117 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 19,835 - Forks: 1,780
zhisheng17/flink-learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Language: Java - Size: 41.6 MB - Last synced at: about 1 month ago - Pushed at: 4 months ago - Stars: 14,800 - Forks: 3,940
oxnr/awesome-bigdata
A curated list of awesome big data frameworks, ressources and other awesomeness.
Size: 843 KB - Last synced at: 7 days ago - Pushed at: 5 months ago - Stars: 13,673 - Forks: 2,573
redpanda-data/redpanda
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
Language: C++ - Size: 112 MB - Last synced at: about 7 hours ago - Pushed at: about 8 hours ago - Stars: 10,509 - Forks: 644
madd86/awesome-system-design
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
Size: 1.71 MB - Last synced at: 6 days ago - Pushed at: about 1 year ago - Stars: 10,464 - Forks: 1,174
ThreeDotsLabs/watermill
Building event-driven applications the easy way in Go.
Language: Go - Size: 25.5 MB - Last synced at: 13 days ago - Pushed at: about 1 month ago - Stars: 8,563 - Forks: 437
redpanda-data/connect
Fancy stream processing made operationally mundane
Language: Go - Size: 35.3 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 8,392 - Forks: 878
risingwavelabs/risingwave
Stream processing and management platform.
Language: Rust - Size: 167 MB - Last synced at: about 11 hours ago - Pushed at: about 11 hours ago - Stars: 7,934 - Forks: 654
fluent/fluent-bit
Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows
Language: C - Size: 104 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 7,000 - Forks: 1,715
robinhood/faust
Python Stream Processing
Language: Python - Size: 8.3 MB - Last synced at: 15 days ago - Pushed at: 11 months ago - Stars: 6,797 - Forks: 533
hazelcast/hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
Language: Java - Size: 398 MB - Last synced at: 5 days ago - Pushed at: 6 days ago - Stars: 6,347 - Forks: 1,854
MaterializeInc/materialize
Real-time Data Integration and Transformation: use SQL to transform, deliver, and act on fast-changing data.
Language: Rust - Size: 261 MB - Last synced at: about 7 hours ago - Pushed at: about 7 hours ago - Stars: 6,032 - Forks: 471
apache/hudi
Upserts, Deletes And Incremental Processing on Big Data.
Language: Java - Size: 1.82 GB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 5,846 - Forks: 2,422
online-ml/river
🌊 Online machine learning in Python
Language: Python - Size: 317 MB - Last synced at: about 7 hours ago - Pushed at: about 9 hours ago - Stars: 5,429 - Forks: 580
javascriptdata/danfojs
Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.
Language: TypeScript - Size: 79.1 MB - Last synced at: 4 days ago - Pushed at: 9 days ago - Stars: 4,949 - Forks: 217
infinyon/fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Language: Rust - Size: 34.3 MB - Last synced at: 9 days ago - Pushed at: 9 days ago - Stars: 4,938 - Forks: 514
ArroyoSystems/arroyo
Distributed stream processing engine in Rust
Language: Rust - Size: 14.7 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 4,390 - Forks: 283
ag2ai/faststream
FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.
Language: Python - Size: 496 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 4,080 - Forks: 241
memgraph/memgraph
Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.
Language: C++ - Size: 43.8 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 3,027 - Forks: 165
manuzhang/awesome-streaming
a curated list of awesome streaming frameworks, applications, etc
Size: 2.71 MB - Last synced at: 4 days ago - Pushed at: about 2 months ago - Stars: 2,845 - Forks: 308
douban/dpark 📦
Python clone of Spark, a MapReduce alike framework in Python
Language: Python - Size: 2.65 MB - Last synced at: 10 days ago - Pushed at: over 4 years ago - Stars: 2,680 - Forks: 530
pipelinedb/pipelinedb
High-performance time-series aggregation for PostgreSQL
Language: C - Size: 46.7 MB - Last synced at: 4 months ago - Pushed at: over 3 years ago - Stars: 2,643 - Forks: 243
PeerDB-io/peerdb
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
Language: Go - Size: 15 MB - Last synced at: 7 days ago - Pushed at: 7 days ago - Stars: 2,570 - Forks: 119
francoispqt/gojay
high performance JSON encoder/decoder with stream API for Golang
Language: Go - Size: 5.61 MB - Last synced at: about 1 month ago - Pushed at: over 1 year ago - Stars: 2,130 - Forks: 116
reugn/go-streams
A lightweight stream processing library for Go
Language: Go - Size: 598 KB - Last synced at: about 1 month ago - Pushed at: about 2 months ago - Stars: 2,036 - Forks: 164
numaproj/numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
Language: Go - Size: 46.4 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 1,888 - Forks: 135
timeplus-io/proton
High-performance, low-footprint SQL database written in C++. Process millions of rows per second from Kafka/Pulsar, Iceberg, or ClickHouse, and seamlessly write results back. Supports powerful features like JOIN, CDC, UPSERT, and LOOKUP, enabling real-time analytics and ETL at scale.
Language: C++ - Size: 467 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1,801 - Forks: 83
yomorun/yomo
🦖 Stateful Serverless Framework for Geo-distributed Edge AI Infra. with function calling support, write once, run on any model.
Language: Go - Size: 66 MB - Last synced at: about 13 hours ago - Pushed at: about 14 hours ago - Stars: 1,767 - Forks: 131
bytewax/bytewax
Python Stream Processing
Language: Python - Size: 12 MB - Last synced at: 3 days ago - Pushed at: 3 months ago - Stars: 1,766 - Forks: 83
nerevu/riko
A Python stream processing engine modeled after Yahoo! Pipes
Language: Python - Size: 2.58 MB - Last synced at: 9 days ago - Pushed at: over 3 years ago - Stars: 1,600 - Forks: 75
siddhi-io/siddhi
Stream Processing and Complex Event Processing Engine
Language: Java - Size: 33.6 MB - Last synced at: 24 days ago - Pushed at: 24 days ago - Stars: 1,547 - Forks: 530
WallarooLabs/wally
Distributed Stream Processing
Language: Pony - Size: 268 MB - Last synced at: about 1 month ago - Pushed at: about 4 years ago - Stars: 1,481 - Forks: 68
quixio/quix-streams
Python Streaming DataFrames for Kafka
Language: Python - Size: 8.76 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 1,393 - Forks: 83
halaxa/json-machine
Efficient, easy-to-use, and fast PHP JSON stream parser
Language: PHP - Size: 1.58 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 1,223 - Forks: 68
spring-attic/spring-cloud-dataflow 📦
A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
Language: Java - Size: 68.3 MB - Last synced at: 23 days ago - Pushed at: 2 months ago - Stars: 1,133 - Forks: 590
hazelcast/hazelcast-jet 📦
Distributed Stream and Batch Processing
Language: Java - Size: 171 MB - Last synced at: 3 months ago - Pushed at: 7 months ago - Stars: 1,101 - Forks: 207
uhop/stream-json
The micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API.
Language: JavaScript - Size: 899 KB - Last synced at: 2 days ago - Pushed at: 5 months ago - Stars: 1,073 - Forks: 48
clj-commons/manifold
A compatibility layer for event-driven abstractions
Language: Clojure - Size: 1.56 MB - Last synced at: 6 days ago - Pushed at: about 1 year ago - Stars: 1,028 - Forks: 109
spring-cloud/spring-cloud-stream
Framework for building Event-Driven Microservices
Language: Java - Size: 17.9 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 1,027 - Forks: 624
edp963/wormhole
Wormhole is a SPaaS (Stream Processing as a Service) Platform
Language: JavaScript - Size: 45.5 MB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 978 - Forks: 479
ververica/flink-sql-cookbook
The Apache Flink SQL Cookbook is a curated collection of examples, patterns, and use cases of Apache Flink SQL. Many of the recipes are completely self-contained and can be run in Ververica Platform as is.
Language: Dockerfile - Size: 14.1 MB - Last synced at: 4 months ago - Pushed at: over 2 years ago - Stars: 879 - Forks: 202
nodefluent/kafka-streams
equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:
Language: TypeScript - Size: 4.14 MB - Last synced at: 10 days ago - Pushed at: over 2 years ago - Stars: 836 - Forks: 111
vladimirvivien/automi
A stream processing API for Go (now with generic type support)
Language: Go - Size: 2.72 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 799 - Forks: 63
gearpump/gearpump
Lightweight real-time big data streaming engine over Akka
Language: Scala - Size: 26.2 MB - Last synced at: 4 days ago - Pushed at: over 3 years ago - Stars: 761 - Forks: 152
gazette/core
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
Language: Go - Size: 4.89 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 756 - Forks: 53
pipeless-ai/pipeless
An open-source computer vision framework to build and deploy apps in minutes
Language: Rust - Size: 142 MB - Last synced at: 19 days ago - Pushed at: about 1 year ago - Stars: 756 - Forks: 39
lw-lin/streaming-readings
Streaming System 相关的论文读物
Size: 6.84 KB - Last synced at: about 1 month ago - Pushed at: over 3 years ago - Stars: 733 - Forks: 154
hstreamdb/hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
Language: Haskell - Size: 6.28 MB - Last synced at: 8 days ago - Pushed at: 6 months ago - Stars: 721 - Forks: 55
TouK/nussknacker
Low-code tool for automating actions on real time data | Stream processing for the users.
Language: Scala - Size: 187 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 691 - Forks: 99
apache/streampipes
Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.
Language: Java - Size: 312 MB - Last synced at: 2 days ago - Pushed at: 8 days ago - Stars: 659 - Forks: 201
infoslack/awesome-kafka
A list about Apache Kafka
Size: 96.7 KB - Last synced at: 12 days ago - Pushed at: 4 months ago - Stars: 579 - Forks: 164
microsoft/psi
Platform for Situated Intelligence
Language: C# - Size: 37.2 MB - Last synced at: 6 days ago - Pushed at: 3 months ago - Stars: 561 - Forks: 102
leesf/hudi-resources
汇总Apache Hudi相关资料
Size: 23.8 MB - Last synced at: 10 days ago - Pushed at: 10 days ago - Stars: 554 - Forks: 160
deep5050/radio-active
Play any radios around the globe right from the terminal :zap:
Language: Python - Size: 553 KB - Last synced at: 9 days ago - Pushed at: 5 months ago - Stars: 541 - Forks: 26
constellation-rs/amadeus
Harmonious distributed data analysis in Rust.
Language: Rust - Size: 2.46 MB - Last synced at: 1 day ago - Pushed at: almost 4 years ago - Stars: 480 - Forks: 25
nmaquet/kasper 📦
Kasper is a lightweight library for processing Kafka topics.
Language: Go - Size: 292 KB - Last synced at: 2 months ago - Pushed at: 4 months ago - Stars: 439 - Forks: 23
smooks/smooks
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
Language: Java - Size: 29.2 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 404 - Forks: 360
FutureSDR/FutureSDR
An Async SDR Runtime for Heterogeneous Architectures
Language: Rust - Size: 26.8 MB - Last synced at: 8 days ago - Pushed at: 2 months ago - Stars: 375 - Forks: 64
RedisGears/RedisGears
Dynamic execution framework for your Redis data
Language: Rust - Size: 4.76 MB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 373 - Forks: 66
Kurento/kurento
Kurento WebRTC Media Server
Language: C - Size: 132 MB - Last synced at: 2 days ago - Pushed at: 3 months ago - Stars: 368 - Forks: 64
graphform/swim-rust
Self-contained distributed software platform for building stateful, massively real-time streaming applications in Rust.
Language: Rust - Size: 25.1 MB - Last synced at: 10 days ago - Pushed at: 12 days ago - Stars: 341 - Forks: 12
AICoE/log-anomaly-detector 📦
Log Anomaly Detection - Machine learning to detect abnormal events logs
Language: Jupyter Notebook - Size: 18.7 MB - Last synced at: 23 days ago - Pushed at: about 2 years ago - Stars: 329 - Forks: 134
Point72/csp
csp is a high performance reactive stream processing library, written in C++ and Python
Language: Python - Size: 4.8 MB - Last synced at: 7 days ago - Pushed at: 7 days ago - Stars: 326 - Forks: 58
AxisCommunications/media-stream-library-js
JavaScript library to handle media streams on the command line (Node.js) and in the browser.
Language: TypeScript - Size: 872 MB - Last synced at: 7 days ago - Pushed at: 22 days ago - Stars: 313 - Forks: 105
memgraph/mage
MAGE - Memgraph Advanced Graph Extensions :crystal_ball:
Language: C++ - Size: 21.8 MB - Last synced at: 7 days ago - Pushed at: 14 days ago - Stars: 294 - Forks: 31
vectorengine/vectorsql 📦
VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.
Language: Go - Size: 1.36 MB - Last synced at: 3 months ago - Pushed at: over 3 years ago - Stars: 292 - Forks: 53
Chulong-Li/Real-time-Sentiment-Tracking-on-Twitter-for-Brand-Improvement-and-Trend-Recognition
A real-time interactive web app based on data pipelines using streaming Twitter data, automated sentiment analysis, and MySQL&PostgreSQL database (Deployed on Heroku)
Language: Jupyter Notebook - Size: 5.78 MB - Last synced at: 8 months ago - Pushed at: about 5 years ago - Stars: 283 - Forks: 127
streamnative/pulsar-flink 📦
Elastic data processing with Apache Pulsar and Apache Flink
Language: Java - Size: 2.16 MB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 279 - Forks: 120
raystack/dagger
Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.
Language: Java - Size: 12 MB - Last synced at: 6 days ago - Pushed at: almost 2 years ago - Stars: 273 - Forks: 41
bbejeck/kafka-streams-in-action
Source code for the Kafka Streams in Action Book
Language: Java - Size: 176 MB - Last synced at: over 1 year ago - Pushed at: almost 4 years ago - Stars: 251 - Forks: 177
cross-platform/dspatch
The Refreshingly Simple Cross-Platform C++ Dataflow / Patching / Pipelining / Graph Processing / Stream Processing / Reactive Programming Framework
Language: C++ - Size: 6.21 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 238 - Forks: 43
talariadb/talaria
TalariaDB is a distributed, highly available, and low latency time-series database for Presto
Language: Go - Size: 12.8 MB - Last synced at: 17 days ago - Pushed at: over 1 year ago - Stars: 221 - Forks: 31
streamnative/function-mesh
The serverless framework purpose-built for event streaming applications.
Language: Go - Size: 55.6 MB - Last synced at: about 12 hours ago - Pushed at: about 12 hours ago - Stars: 218 - Forks: 28
schollz/pluck
Pluck text in a fast and intuitive way :rooster:
Language: Go - Size: 6.55 MB - Last synced at: 7 days ago - Pushed at: almost 6 years ago - Stars: 215 - Forks: 6
TomasMikula/libretto
Declarative concurrency and stream processing library for Scala
Language: Scala - Size: 15.5 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 211 - Forks: 7
confluentinc/ksql
The database purpose-built for stream processing applications.
Language: Java - Size: 211 MB - Last synced at: about 19 hours ago - Pushed at: about 20 hours ago - Stars: 209 - Forks: 1,047
logrange/logrange
High performance data aggregating storage
Language: Go - Size: 69.2 MB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 191 - Forks: 16
bytewatch/dolphinbeat
A server that pulls and parses MySQL binlog, pushs change data into different sinks like Kafka.
Language: Go - Size: 2.97 MB - Last synced at: 8 months ago - Pushed at: over 6 years ago - Stars: 176 - Forks: 30
miroslavpejic85/mirotalkbro
📡 MiroTalk WebRTC Live Broadcast allows to broadcast live video, audio and screen stream to all connected users (viewers).
Language: JavaScript - Size: 1.39 MB - Last synced at: 10 days ago - Pushed at: 10 days ago - Stars: 175 - Forks: 43
neo4j-contrib/neo4j-streams
Neo4j Kafka Connector
Language: Kotlin - Size: 3.76 MB - Last synced at: 8 days ago - Pushed at: 8 days ago - Stars: 174 - Forks: 73
pemrouz/fero
light, fast, scalable, streaming microservices made easy
Language: JavaScript - Size: 1.05 MB - Last synced at: 7 months ago - Pushed at: about 7 years ago - Stars: 174 - Forks: 10
Claviz/xlstream
Turns XLSX into a readable stream.
Language: TypeScript - Size: 709 KB - Last synced at: 8 days ago - Pushed at: 11 months ago - Stars: 173 - Forks: 17
akka/akka-stream-contrib
Add-ons to Akka Stream
Language: Scala - Size: 315 KB - Last synced at: 12 days ago - Pushed at: almost 2 years ago - Stars: 173 - Forks: 78
vsouza/go-kafka-example
Golang Kafka consumer and producer example
Language: Go - Size: 505 KB - Last synced at: 4 days ago - Pushed at: over 1 year ago - Stars: 169 - Forks: 59
GridProtectionAlliance/gsf
Grid Solutions Framework
Language: C# - Size: 249 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 165 - Forks: 70
LearningJournal/Kafka-Streams-Real-time-Stream-Processing
This is the central repository for all materials related to Kafka Streams : Real-time Stream Processing! Book by Prashant Pandey.
Language: Java - Size: 5.25 MB - Last synced at: 3 months ago - Pushed at: almost 5 years ago - Stars: 164 - Forks: 163
whitaker-io/machine
Machine is a workflow/pipeline library for processing data
Language: Go - Size: 1.42 MB - Last synced at: 8 days ago - Pushed at: 8 days ago - Stars: 162 - Forks: 12
mariomac/gostream
A Go port of the Java Streams API. Type-safe and functional Go Streams processing for Go 1.18+.
Language: Go - Size: 211 KB - Last synced at: 2 months ago - Pushed at: over 2 years ago - Stars: 161 - Forks: 9
synacker/daggy
Daggy - Data Aggregation Utility and C/C++ developer library for data streams catching
Language: C++ - Size: 7.34 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 156 - Forks: 15
lispking/fluxus
Fluxus Stream Processing Engine
Language: Rust - Size: 5.02 MB - Last synced at: 17 days ago - Pushed at: about 1 month ago - Stars: 150 - Forks: 22
psila-ai/callysto
Stream processing & Service framework.
Language: Rust - Size: 3.67 MB - Last synced at: 15 days ago - Pushed at: about 1 year ago - Stars: 149 - Forks: 8
ElAlev/Wayeb
Wayeb is a Complex Event Processing and Forecasting (CEP/F) engine written in Scala.
Language: Scala - Size: 8.17 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 147 - Forks: 6
samsara/samsara
Samsara is a real-time analytics platform
Language: Clojure - Size: 22.8 MB - Last synced at: 7 days ago - Pushed at: about 2 years ago - Stars: 147 - Forks: 17
GridProtectionAlliance/openPDC
Open Source Phasor Data Concentrator
Language: C# - Size: 2.48 GB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 138 - Forks: 57
zakariamaaraki/RemoteCodeCompiler
An online code compiler supporting 11 programming languages (Java, Kotlin, Scala, C, C++, C#, Golang, Python, Ruby, Rust and Haskell) for competitive programming and coding interviews.
Language: Java - Size: 3.34 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 135 - Forks: 47
mjwestcott/runnel
Distributed event processing for Python based on Redis Streams
Language: Python - Size: 6.58 MB - Last synced at: 4 months ago - Pushed at: almost 5 years ago - Stars: 133 - Forks: 11
runreveal/kawa
A blazingly fast event stream processing library powering the reveald event processing daemon.
Language: Go - Size: 1010 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 129 - Forks: 3
devshawn/kafka-shell
⚡A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools.
Language: Python - Size: 82 KB - Last synced at: 15 days ago - Pushed at: over 5 years ago - Stars: 128 - Forks: 12
