Topic: "stream-processing"
google-ai-edge/mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
Language: C++ - Size: 575 MB - Last synced at: about 16 hours ago - Pushed at: about 21 hours ago - Stars: 29,453 - Forks: 5,320
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Language: Python - Size: 132 MB - Last synced at: about 4 hours ago - Pushed at: about 6 hours ago - Stars: 24,616 - Forks: 359
vectordotdev/vector
A high-performance observability data pipeline.
Language: Rust - Size: 115 MB - Last synced at: about 5 hours ago - Pushed at: about 19 hours ago - Stars: 19,209 - Forks: 1,725
zhisheng17/flink-learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Language: Java - Size: 41.6 MB - Last synced at: 15 minutes ago - Pushed at: about 1 month ago - Stars: 14,744 - Forks: 3,931
oxnr/awesome-bigdata
A curated list of awesome big data frameworks, ressources and other awesomeness.
Size: 843 KB - Last synced at: 3 days ago - Pushed at: 2 months ago - Stars: 13,568 - Forks: 2,568
redpanda-data/redpanda
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
Language: C++ - Size: 107 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 10,229 - Forks: 617
madd86/awesome-system-design
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
Size: 1.71 MB - Last synced at: 10 days ago - Pushed at: 10 months ago - Stars: 10,180 - Forks: 1,140
redpanda-data/connect
Fancy stream processing made operationally mundane
Language: Go - Size: 34 MB - Last synced at: 5 days ago - Pushed at: 6 days ago - Stars: 8,309 - Forks: 863
ThreeDotsLabs/watermill
Building event-driven applications the easy way in Go.
Language: Go - Size: 25.4 MB - Last synced at: about 4 hours ago - Pushed at: 4 days ago - Stars: 8,306 - Forks: 430
risingwavelabs/risingwave
Stream processing and management platform.
Language: Rust - Size: 158 MB - Last synced at: about 3 hours ago - Pushed at: about 4 hours ago - Stars: 7,667 - Forks: 626
fluent/fluent-bit
Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows
Language: C - Size: 104 MB - Last synced at: 24 minutes ago - Pushed at: 25 minutes ago - Stars: 6,791 - Forks: 1,668
robinhood/faust
Python Stream Processing
Language: Python - Size: 8.3 MB - Last synced at: 14 days ago - Pushed at: 9 months ago - Stars: 6,783 - Forks: 535
hazelcast/hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
Language: Java - Size: 393 MB - Last synced at: about 4 hours ago - Pushed at: about 5 hours ago - Stars: 6,303 - Forks: 1,861
MaterializeInc/materialize
Real-time Data Integration and Transformation: use SQL to transform, deliver, and act on fast-changing data.
Language: Rust - Size: 256 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 5,954 - Forks: 470
apache/hudi
Upserts, Deletes And Incremental Processing on Big Data.
Language: Java - Size: 1.71 GB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 5,740 - Forks: 2,393
online-ml/river
🌊 Online machine learning in Python
Language: Python - Size: 315 MB - Last synced at: 26 minutes ago - Pushed at: about 2 months ago - Stars: 5,294 - Forks: 568
javascriptdata/danfojs
Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.
Language: TypeScript - Size: 79 MB - Last synced at: 4 days ago - Pushed at: 13 days ago - Stars: 4,898 - Forks: 214
infinyon/fluvio
Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.
Language: Rust - Size: 34.2 MB - Last synced at: about 4 hours ago - Pushed at: about 12 hours ago - Stars: 4,613 - Forks: 511
ag2ai/faststream
FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.
Language: Python - Size: 455 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 3,778 - Forks: 219
ArroyoSystems/arroyo
Distributed stream processing engine in Rust
Language: Rust - Size: 13.6 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 3,732 - Forks: 215
manuzhang/awesome-streaming
a curated list of awesome streaming frameworks, applications, etc
Size: 2.7 MB - Last synced at: 3 days ago - Pushed at: about 1 month ago - Stars: 2,800 - Forks: 305
memgraph/memgraph
Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.
Language: C++ - Size: 42.2 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 2,755 - Forks: 144
douban/dpark 📦
Python clone of Spark, a MapReduce alike framework in Python
Language: Python - Size: 2.65 MB - Last synced at: 3 days ago - Pushed at: over 4 years ago - Stars: 2,682 - Forks: 530
pipelinedb/pipelinedb
High-performance time-series aggregation for PostgreSQL
Language: C - Size: 46.7 MB - Last synced at: about 1 month ago - Pushed at: about 3 years ago - Stars: 2,643 - Forks: 243
PeerDB-io/peerdb
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
Language: Go - Size: 13.8 MB - Last synced at: 4 days ago - Pushed at: 5 days ago - Stars: 2,491 - Forks: 111
francoispqt/gojay
high performance JSON encoder/decoder with stream API for Golang
Language: Go - Size: 5.61 MB - Last synced at: 6 months ago - Pushed at: over 1 year ago - Stars: 2,116 - Forks: 111
reugn/go-streams
A lightweight stream processing library for Go
Language: Go - Size: 575 KB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 2,020 - Forks: 164
numaproj/numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
Language: Go - Size: 39.5 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 1,862 - Forks: 130
timeplus-io/proton
High-performance, low-footprint SQL database written in C++. Process millions of rows per second from Kafka/Pulsar, Iceberg, or ClickHouse, and seamlessly write results back. Supports powerful features like JOIN, CDC, UPSERT, and LOOKUP, enabling real-time analytics and ETL at scale.
Language: C++ - Size: 466 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 1,770 - Forks: 81
yomorun/yomo
🦖 Stateful Serverless Framework for Geo-distributed Edge AI Infra. with function calling support, write once, run on any model.
Language: Go - Size: 65.8 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 1,745 - Forks: 132
bytewax/bytewax
Python Stream Processing
Language: Python - Size: 12 MB - Last synced at: 2 days ago - Pushed at: 26 days ago - Stars: 1,711 - Forks: 78
nerevu/riko
A Python stream processing engine modeled after Yahoo! Pipes
Language: Python - Size: 2.58 MB - Last synced at: 8 days ago - Pushed at: over 3 years ago - Stars: 1,603 - Forks: 76
siddhi-io/siddhi
Stream Processing and Complex Event Processing Engine
Language: Java - Size: 33.6 MB - Last synced at: 4 days ago - Pushed at: 5 days ago - Stars: 1,541 - Forks: 530
WallarooLabs/wally
Distributed Stream Processing
Language: Pony - Size: 268 MB - Last synced at: 13 days ago - Pushed at: about 4 years ago - Stars: 1,479 - Forks: 68
quixio/quix-streams
Python Streaming DataFrames for Kafka
Language: Python - Size: 8.13 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 1,354 - Forks: 77
spring-cloud/spring-cloud-dataflow
A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
Language: Java - Size: 68.3 MB - Last synced at: 4 days ago - Pushed at: 5 days ago - Stars: 1,131 - Forks: 588
hazelcast/hazelcast-jet 📦
Distributed Stream and Batch Processing
Language: Java - Size: 171 MB - Last synced at: 18 days ago - Pushed at: 4 months ago - Stars: 1,101 - Forks: 207
halaxa/json-machine
Efficient, easy-to-use, and fast PHP JSON stream parser
Language: PHP - Size: 1.48 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 1,089 - Forks: 65
uhop/stream-json
The micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API.
Language: JavaScript - Size: 899 KB - Last synced at: 3 days ago - Pushed at: 3 months ago - Stars: 1,041 - Forks: 47
clj-commons/manifold
A compatibility layer for event-driven abstractions
Language: Clojure - Size: 1.56 MB - Last synced at: about 9 hours ago - Pushed at: 11 months ago - Stars: 1,028 - Forks: 108
spring-cloud/spring-cloud-stream
Framework for building Event-Driven Microservices
Language: Java - Size: 18.1 MB - Last synced at: about 1 hour ago - Pushed at: about 2 hours ago - Stars: 1,019 - Forks: 621
edp963/wormhole
Wormhole is a SPaaS (Stream Processing as a Service) Platform
Language: JavaScript - Size: 45.5 MB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 976 - Forks: 482
ververica/flink-sql-cookbook
The Apache Flink SQL Cookbook is a curated collection of examples, patterns, and use cases of Apache Flink SQL. Many of the recipes are completely self-contained and can be run in Ververica Platform as is.
Language: Dockerfile - Size: 14.1 MB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 879 - Forks: 202
nodefluent/kafka-streams
equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:
Language: TypeScript - Size: 4.14 MB - Last synced at: about 1 year ago - Pushed at: about 2 years ago - Stars: 821 - Forks: 111
vladimirvivien/automi
A stream processing API for Go (alpha)
Language: Go - Size: 2.27 MB - Last synced at: 12 days ago - Pushed at: 24 days ago - Stars: 789 - Forks: 62
gearpump/gearpump
Lightweight real-time big data streaming engine over Akka
Language: Scala - Size: 26.2 MB - Last synced at: 3 days ago - Pushed at: about 3 years ago - Stars: 762 - Forks: 152
pipeless-ai/pipeless
An open-source computer vision framework to build and deploy apps in minutes
Language: Rust - Size: 142 MB - Last synced at: 8 days ago - Pushed at: 12 months ago - Stars: 749 - Forks: 38
gazette/core
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
Language: Go - Size: 4.81 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 746 - Forks: 53
lw-lin/streaming-readings
Streaming System 相关的论文读物
Size: 6.84 KB - Last synced at: 18 days ago - Pushed at: about 3 years ago - Stars: 731 - Forks: 155
hstreamdb/hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
Language: Haskell - Size: 6.28 MB - Last synced at: 9 days ago - Pushed at: 4 months ago - Stars: 722 - Forks: 55
TouK/nussknacker
Low-code tool for automating actions on real time data | Stream processing for the users.
Language: Scala - Size: 179 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 683 - Forks: 95
apache/streampipes
Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.
Language: Java - Size: 310 MB - Last synced at: 4 days ago - Pushed at: 5 days ago - Stars: 639 - Forks: 195
infoslack/awesome-kafka
A list about Apache Kafka
Size: 96.7 KB - Last synced at: 3 days ago - Pushed at: about 1 month ago - Stars: 580 - Forks: 163
microsoft/psi
Platform for Situated Intelligence
Language: C# - Size: 37.2 MB - Last synced at: 9 days ago - Pushed at: 29 days ago - Stars: 552 - Forks: 101
leesf/hudi-resources
汇总Apache Hudi相关资料
Size: 23.7 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 550 - Forks: 160
deep5050/radio-active
Play any radios around the globe right from the terminal :zap:
Language: Python - Size: 553 KB - Last synced at: 9 days ago - Pushed at: 2 months ago - Stars: 526 - Forks: 25
constellation-rs/amadeus
Harmonious distributed data analysis in Rust.
Language: Rust - Size: 2.46 MB - Last synced at: 5 months ago - Pushed at: over 3 years ago - Stars: 474 - Forks: 26
nmaquet/kasper
Kasper is a lightweight library for processing Kafka topics.
Language: Go - Size: 288 KB - Last synced at: about 1 year ago - Pushed at: over 7 years ago - Stars: 439 - Forks: 24
smooks/smooks
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
Language: Java - Size: 29.2 MB - Last synced at: 10 days ago - Pushed at: 15 days ago - Stars: 400 - Forks: 360
RedisGears/RedisGears
Dynamic execution framework for your Redis data
Language: Rust - Size: 4.76 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 373 - Forks: 66
AICoE/log-anomaly-detector 📦
Log Anomaly Detection - Machine learning to detect abnormal events logs
Language: Jupyter Notebook - Size: 18.7 MB - Last synced at: 16 days ago - Pushed at: about 2 years ago - Stars: 328 - Forks: 132
AxisCommunications/media-stream-library-js
JavaScript library to handle media streams on the command line (Node.js) and in the browser.
Language: TypeScript - Size: 872 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 311 - Forks: 104
vectorengine/vectorsql 📦
VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.
Language: Go - Size: 1.36 MB - Last synced at: 5 days ago - Pushed at: over 3 years ago - Stars: 292 - Forks: 53
Chulong-Li/Real-time-Sentiment-Tracking-on-Twitter-for-Brand-Improvement-and-Trend-Recognition
A real-time interactive web app based on data pipelines using streaming Twitter data, automated sentiment analysis, and MySQL&PostgreSQL database (Deployed on Heroku)
Language: Jupyter Notebook - Size: 5.78 MB - Last synced at: 6 months ago - Pushed at: almost 5 years ago - Stars: 283 - Forks: 127
memgraph/mage
MAGE - Memgraph Advanced Graph Extensions :crystal_ball:
Language: C++ - Size: 21.7 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 279 - Forks: 29
streamnative/pulsar-flink 📦
Elastic data processing with Apache Pulsar and Apache Flink
Language: Java - Size: 2.16 MB - Last synced at: 5 months ago - Pushed at: over 2 years ago - Stars: 279 - Forks: 119
raystack/dagger
Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.
Language: Java - Size: 12 MB - Last synced at: 3 days ago - Pushed at: over 1 year ago - Stars: 273 - Forks: 41
Point72/csp
csp is a high performance reactive stream processing library, written in C++ and Python
Language: Python - Size: 4.72 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 270 - Forks: 45
FutureSDR/FutureSDR
An Async SDR Runtime for Heterogeneous Architectures
Language: Rust - Size: 25.1 MB - Last synced at: 12 months ago - Pushed at: about 1 year ago - Stars: 262 - Forks: 46
bbejeck/kafka-streams-in-action
Source code for the Kafka Streams in Action Book
Language: Java - Size: 176 MB - Last synced at: over 1 year ago - Pushed at: almost 4 years ago - Stars: 251 - Forks: 177
cross-platform/dspatch
The Refreshingly Simple Cross-Platform C++ Dataflow / Patching / Pipelining / Graph Processing / Stream Processing / Reactive Programming Framework
Language: C++ - Size: 6.14 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 236 - Forks: 43
Kurento/kurento
Kurento WebRTC Media Server
Language: C - Size: 131 MB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 222 - Forks: 49
talariadb/talaria
TalariaDB is a distributed, highly available, and low latency time-series database for Presto
Language: Go - Size: 12.8 MB - Last synced at: 8 days ago - Pushed at: about 1 year ago - Stars: 219 - Forks: 31
streamnative/function-mesh
The serverless framework purpose-built for event streaming applications.
Language: Go - Size: 55.6 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 215 - Forks: 27
schollz/pluck
Pluck text in a fast and intuitive way :rooster:
Language: Go - Size: 6.55 MB - Last synced at: 2 days ago - Pushed at: over 5 years ago - Stars: 215 - Forks: 6
TomasMikula/libretto
Declarative concurrency and stream processing library for Scala
Language: Scala - Size: 15.9 MB - Last synced at: 13 days ago - Pushed at: 13 days ago - Stars: 210 - Forks: 7
logrange/logrange
High performance data aggregating storage
Language: Go - Size: 69.2 MB - Last synced at: 30 days ago - Pushed at: about 1 year ago - Stars: 191 - Forks: 16
confluentinc/ksql
The database purpose-built for stream processing applications.
Language: Java - Size: 205 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 186 - Forks: 1,042
bytewatch/dolphinbeat
A server that pulls and parses MySQL binlog, pushs change data into different sinks like Kafka.
Language: Go - Size: 2.97 MB - Last synced at: 5 months ago - Pushed at: about 6 years ago - Stars: 176 - Forks: 30
pemrouz/fero
light, fast, scalable, streaming microservices made easy
Language: JavaScript - Size: 1.05 MB - Last synced at: 5 months ago - Pushed at: almost 7 years ago - Stars: 174 - Forks: 10
neo4j-contrib/neo4j-streams
Neo4j Kafka Connector
Language: Kotlin - Size: 3.74 MB - Last synced at: 10 days ago - Pushed at: 12 days ago - Stars: 173 - Forks: 72
akka/akka-stream-contrib
Add-ons to Akka Stream
Language: Scala - Size: 315 KB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 172 - Forks: 77
Claviz/xlstream
Turns XLSX into a readable stream.
Language: TypeScript - Size: 709 KB - Last synced at: 18 days ago - Pushed at: 9 months ago - Stars: 171 - Forks: 17
miroslavpejic85/mirotalkbro
📡 MiroTalk WebRTC Live Broadcast allows to broadcast live video, audio and screen stream to all connected users (viewers).
Language: JavaScript - Size: 1.37 MB - Last synced at: 8 days ago - Pushed at: 8 days ago - Stars: 169 - Forks: 43
vsouza/go-kafka-example
Golang Kafka consumer and producer example
Language: Go - Size: 505 KB - Last synced at: about 11 hours ago - Pushed at: over 1 year ago - Stars: 169 - Forks: 59
LearningJournal/Kafka-Streams-Real-time-Stream-Processing
This is the central repository for all materials related to Kafka Streams : Real-time Stream Processing! Book by Prashant Pandey.
Language: Java - Size: 5.25 MB - Last synced at: 9 days ago - Pushed at: over 4 years ago - Stars: 164 - Forks: 163
GridProtectionAlliance/gsf
Grid Solutions Framework
Language: C# - Size: 248 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 163 - Forks: 70
mariomac/gostream
A Go port of the Java Streams API. Type-safe and functional Go Streams processing for Go 1.18+.
Language: Go - Size: 211 KB - Last synced at: 19 days ago - Pushed at: over 2 years ago - Stars: 161 - Forks: 9
whitaker-io/machine
Machine is a workflow/pipeline library for processing data
Language: Go - Size: 1.39 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 157 - Forks: 12
synacker/daggy
Daggy - Data Aggregation Utility and C/C++ developer library for data streams catching
Language: C++ - Size: 7.28 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 154 - Forks: 15
Jet-Engine/callysto
Stream processing & Service framework.
Language: Rust - Size: 3.67 MB - Last synced at: 7 days ago - Pushed at: 12 months ago - Stars: 147 - Forks: 8
ElAlev/Wayeb
Wayeb is a Complex Event Processing and Forecasting (CEP/F) engine written in Scala.
Language: Scala - Size: 8.17 MB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 147 - Forks: 6
samsara/samsara
Samsara is a real-time analytics platform
Language: Clojure - Size: 22.8 MB - Last synced at: 17 days ago - Pushed at: almost 2 years ago - Stars: 147 - Forks: 17
zakariamaaraki/RemoteCodeCompiler
An online code compiler supporting 11 programming languages (Java, Kotlin, Scala, C, C++, C#, Golang, Python, Ruby, Rust and Haskell) for competitive programming and coding interviews.
Language: Java - Size: 3.34 MB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 135 - Forks: 47
GridProtectionAlliance/openPDC
Open Source Phasor Data Concentrator
Language: C# - Size: 2.47 GB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 134 - Forks: 56
mjwestcott/runnel
Distributed event processing for Python based on Redis Streams
Language: Python - Size: 6.58 MB - Last synced at: about 1 month ago - Pushed at: over 4 years ago - Stars: 133 - Forks: 11
runreveal/kawa
A blazingly fast event stream processing library powering the reveald event processing daemon.
Language: Go - Size: 1010 KB - Last synced at: 13 days ago - Pushed at: 13 days ago - Stars: 129 - Forks: 3
devshawn/kafka-shell
⚡A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools.
Language: Python - Size: 82 KB - Last synced at: 11 days ago - Pushed at: about 5 years ago - Stars: 126 - Forks: 11
ExpediaGroup/stream-registry
Stream Discovery and Stream Orchestration
Language: Java - Size: 5.52 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 122 - Forks: 54
xerial/streamdb-readings
Readings in Stream Processing
Size: 265 KB - Last synced at: 10 days ago - Pushed at: 5 months ago - Stars: 122 - Forks: 13
