GitHub topics: map-reduce

Repositories

vikram-ayaluri/simple-map-reduce-distributed

A simulation of the distributed MapReduce model that processes large data by parallelizing map and reduce tasks across virtual nodes.

Language: JavaScript - Size: 3.13 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 0 - Forks: 0

xReniar/US-used-cars-analysis

US used cars analysis with Map-Reduce, Hive, Spark core and Spark SQL

Language: Python - Size: 2.25 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 0 - Forks: 0

m4gshm/gollections

slice, map, iter.Seq, iter.Seq2 APIs for transforming, filtering, reducing, summing and other iteration-based tasks.

Language: Go - Size: 875 KB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 4 - Forks: 0

JuliaFolds/Transducers.jl

Efficient transducers for Julia

Language: Julia - Size: 9.04 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 439 - Forks: 25

numaproj/numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

Language: Go - Size: 44.8 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 1,879 - Forks: 130

kalmyk/fox-wamp

Web Application Message Async Server and WAMP/MQTT bridge

Language: JavaScript - Size: 1.23 MB - Last synced at: 9 days ago - Pushed at: 9 days ago - Stars: 14 - Forks: 6

asavinov/prosto

Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

Language: Python - Size: 1.95 MB - Last synced at: 12 days ago - Pushed at: over 3 years ago - Stars: 91 - Forks: 5

Malisha4065/HadoopProject

Map reducing task with apache hadoop.

Language: Java - Size: 17.6 KB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 0 - Forks: 0

xarray-contrib/flox

Fast & furious GroupBy operations for dask.array

Language: Python - Size: 1.75 MB - Last synced at: 8 days ago - Pushed at: 18 days ago - Stars: 131 - Forks: 20

tkf/ThreadsX.jl

Parallelized Base functions

Language: Julia - Size: 1.15 MB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 330 - Forks: 10

tirthajyoti/Spark-with-Python

Fundamentals of Spark with Python (using PySpark), code examples

Language: Jupyter Notebook - Size: 8.97 MB - Last synced at: 25 days ago - Pushed at: over 2 years ago - Stars: 347 - Forks: 271

chrislusf/gleam

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Language: Go - Size: 8.38 MB - Last synced at: 29 days ago - Pushed at: about 2 months ago - Stars: 3,517 - Forks: 291

mjjackey/Python_Practice

The miscellaneous practices for python

Language: Python - Size: 40 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

Varun0157/distributed-super-linear-mst

Implementations of MST calculation in a distributed setting, assuming super-linear memory.

Language: Go - Size: 677 KB - Last synced at: 3 days ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

Qihoo360/poseidon

A search engine which can hold 100 trillion lines of log data.

Language: Go - Size: 9.16 MB - Last synced at: 27 days ago - Pushed at: about 8 years ago - Stars: 1,988 - Forks: 436

LENSS/csce438-spring2025

machine problems for TAMU CSCE438 Distributed Processing Systems, Spring 2025

Language: C++ - Size: 2.68 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 5

gSchool/js-native-array-methods

Language: JavaScript - Size: 18.6 KB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 1

stdlib-js/utils-map-reduce

Perform a single-pass map-reduce operation against each element in an array and return the accumulated result.

Language: JavaScript - Size: 2.74 MB - Last synced at: 14 days ago - Pushed at: 2 months ago - Stars: 2 - Forks: 0

H1ghBre4k3r/rust-map-reduce

A small hobby implementation of MapReduce that I hacked together at 2am.

Language: Rust - Size: 98.6 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

stdlib-js/utils-map-reduce-right

Perform a single-pass map-reduce operation against each element in an array while iterating from right to left and return the accumulated result.

Language: JavaScript - Size: 2.74 MB - Last synced at: 5 days ago - Pushed at: 2 months ago - Stars: 2 - Forks: 0

k-eleftheriou/activity-tracker-backend

A backend system for the Activity Tracker app that processes GPX files using MapReduce and manages user activity data, while integrating with the frontend.

Language: Java - Size: 499 KB - Last synced at: 6 days ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 1

Assifar-Karim/apollo

A lightweight modern map reduce framework brought to k8s

Language: Go - Size: 1.22 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 24 - Forks: 0

gwr3n/jsdp

A Java Stochastic Dynamic Programming Library

Language: Java - Size: 50.4 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 13 - Forks: 4

asuiu/streamerate

Iterable Java8 style Streams for Python

Language: Python - Size: 486 KB - Last synced at: 13 days ago - Pushed at: 3 months ago - Stars: 8 - Forks: 3

miguelTavora/Word-Count 📦

Problem of word count done using Apache Hadoop

Language: Java - Size: 715 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

futureverse/future.mapreduce

[EXPERIMENTAL] R package: future.mapreduce - Utility Functions for Future Map-Reduce API Packages

Language: R - Size: 1.29 MB - Last synced at: 2 months ago - Pushed at: 3 months ago - Stars: 12 - Forks: 1

quinlan-lab/constraint-tools

Tools to discover natural selection given multiple evolved DNA sequences (e.g., gnomad cohort, or multiple tumor samples)

Language: Jupyter Notebook - Size: 601 MB - Last synced at: 3 months ago - Pushed at: 4 months ago - Stars: 1 - Forks: 0

AsadiAhmad/Word-Counter

Word Counter with Haskell for Programming Language Design Course

Language: Haskell - Size: 238 KB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 16 - Forks: 0

0xHumban/marmot-reduce

🦫 - A little distributed system (Map Reduce simplified)

Language: Go - Size: 4.84 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

Susheel-1999/GenAI-youtubeSummarize

App that takes a YouTube video URL, extracts its transcript, and generates a consolidated summary using LangChain’s Map-Reduce strategy and Groq’s Llama model.

Language: Python - Size: 5.86 KB - Last synced at: 3 months ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

mesqueeb/map-anything

Array.map but for objects with good TypeScript support. A small and simple integration.

Language: TypeScript - Size: 1.48 MB - Last synced at: 19 days ago - Pushed at: 4 months ago - Stars: 11 - Forks: 0

w-k-s/Celebrity-Soundbites-MapReduce-Experiment

Using MapReduce to build a dictionary of YouTube celebrity video clips

Language: Python - Size: 60.5 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

AlessandroConti11/Fault-Tolerant_Data_Flow

This project implements a distributed dataflow platform for processing large amount (big-data) of key-value pairs, where keys and values are integers.

Language: Java - Size: 323 KB - Last synced at: 3 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

vasukalariya/MIT-Distributed-Systems-Lab-6.824-6.5840

MIT Distributed Systems CS 6.824/CS 6.5840

Language: Go - Size: 9.02 MB - Last synced at: 3 months ago - Pushed at: about 1 year ago - Stars: 2 - Forks: 1

natelalor/AI_report_generator

A tool that converts long audio files into a thorough, summarized report. Leverages OpenAI and its API (ChatGPT backend), Langchain for text processing, and Pinecone for vector database facilitation.

Language: Python - Size: 15.3 MB - Last synced at: 3 days ago - Pushed at: over 1 year ago - Stars: 8 - Forks: 2

Provides a scaffold to easily build a cluster to query the data from ESA's Gaia satellite. Gaia is an ambitious mission to chart a three-dimensional map of our Galaxy, the Milky Way. Gaia will provide unprecedented positional and radial velocity measurements with the accuracies needed to produce a stereoscopic and kinematic census of about one billion stars in our Galaxy and throughout the Local Group. This amounts to about 1 per cent of the Galactic stellar population.

Language: Java - Size: 186 KB - Last synced at: 3 months ago - Pushed at: over 8 years ago - Stars: 3 - Forks: 0

ishaansathaye/CSC369-IntroDistributedComputing

Cal Poly Fall 2024 CSC 369 Intro to Distributed Computing

Language: Jupyter Notebook - Size: 729 KB - Last synced at: 2 days ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

Adelin-Info/TP_DATACLOUD

Architecture et développement des systèmes distribuées à large echelle

Language: Scala - Size: 37.1 KB - Last synced at: 3 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

aryanGupta-09/Kmeans-using-MapReduce

K-means clustering algorithm using MapReduce.

Language: Python - Size: 23.4 KB - Last synced at: 4 months ago - Pushed at: 7 months ago - Stars: 1 - Forks: 0

sergey-shandar/purelogic-ts 📦

PureLogic for TypeScript

Language: TypeScript - Size: 301 KB - Last synced at: 21 days ago - Pushed at: over 8 years ago - Stars: 3 - Forks: 0

sergey-shandar/purelogic 📦

PureLogic

Language: C# - Size: 45.9 KB - Last synced at: 3 months ago - Pushed at: almost 8 years ago - Stars: 2 - Forks: 0

JuliaFolds/FoldsCUDA.jl

Data-parallelism on CUDA using Transducers.jl and for loops (FLoops.jl)

Language: Julia - Size: 1.3 MB - Last synced at: 2 days ago - Pushed at: about 2 years ago - Stars: 57 - Forks: 5

fangvv/EdgeLD

Code for paper "Locally Distributed Deep Learning Inference on Edge Device Clusters"

Language: Python - Size: 484 KB - Last synced at: 4 months ago - Pushed at: about 2 years ago - Stars: 12 - Forks: 5

nglthu/infoRetrieval

Inverted Indexer, web crawler, sort, search and poster steamer written using Python for information retrieval.

Language: HTML - Size: 2.27 MB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 22 - Forks: 17

rvantonder/hack_parallel

The core parallel and shared memory library used by Hack, Flow, and Pyre

Language: OCaml - Size: 444 KB - Last synced at: 7 days ago - Pushed at: over 4 years ago - Stars: 41 - Forks: 8

Drapegnik/bsuir

🎓Repository for masters labs on FCSN, BSUIR

Language: Python - Size: 41.3 MB - Last synced at: 3 months ago - Pushed at: over 2 years ago - Stars: 8 - Forks: 2

kunalPisolkar24/IR_Lab

Collection of practical codes for Savitribai Phule Pune University's Information Retrieval Lab (410247) .

Language: Jupyter Notebook - Size: 125 KB - Last synced at: 4 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

aneeshmurali-n/Python-Tasks

Here we are going to learn data science and python by doing tasks by task

Language: Jupyter Notebook - Size: 3.21 MB - Last synced at: 4 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

pscosta/go-strm

A rich Map/Reduce API in Go

Language: Go - Size: 1.17 MB - Last synced at: 8 months ago - Pushed at: almost 2 years ago - Stars: 14 - Forks: 0

IA-Programming/websearch_langchain

an implementation of websearch and stuff, map-reduce, refine and map-rerank

Language: Python - Size: 88.9 KB - Last synced at: 3 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

mirzaim/Hadoop-Twitter-Analysis

Hadoop MapReduce analysis of US Election 2020 Tweets.

Language: Java - Size: 3.91 KB - Last synced at: 4 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

SHUBHAMSHARMA0711/CSE530-Distributed-Systems-Assignments

Assignments Completed during CSE530:Distributed Systems Course in IIITD during Winter 2024 Semester. Professor: Dhruv Kumar

Language: Python - Size: 6.88 MB - Last synced at: 2 months ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

francesco-biscaccia-carrara/BigData_Projects Fork of fantinluca/BDC_BSC

Assignment repository for the Big Data Computing course at the University of Padova for the academic year 2023-2024.

Language: Java - Size: 12.5 MB - Last synced at: 5 months ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

ShahMitul-GenAI/ReviewEngine

Extract Amazon customer reviews of select products via a third-party educational web scrapper, analyze them with LLM, and display concise summaries using an interactive user interface.

Language: Jupyter Notebook - Size: 1020 KB - Last synced at: 10 months ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

ranfysvalle02/summwebsite

This repository is not intended to serve as the next 'library' for text summarization. Instead, it is designed to be an educational resource, providing insights into the inner workings of text summarization.

Language: Python - Size: 72.3 KB - Last synced at: 3 months ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

flaviodelgrosso/llm-chain-map-reduce

Simple map reduce LLM chain pattern demo using Ollama and Langchain

Language: Python - Size: 4.88 KB - Last synced at: 4 months ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

eZWALT/Massive-Information-Search-and-Analysis

FIB-CAIM 2023-24 Q1 Massive Information Search and Analysis subject and lab assignments

Language: Jupyter Notebook - Size: 47.7 MB - Last synced at: 10 months ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

k-eleftheriou/activity-tracker

A mobile application that serves as the user interface for the Activity Tracker system, allowing users to add and track their activities.

Language: Java - Size: 354 KB - Last synced at: 6 days ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 1

MollsReis/nsqfun

NSQ Fun

Language: JavaScript - Size: 11.7 KB - Last synced at: 11 months ago - Pushed at: over 7 years ago - Stars: 1 - Forks: 0

iwilare/advanced-programming

Assignments for the "Advanced Programming" 2020/2021 course @ Department of Computer Science, University of Pisa http://pages.di.unipi.it/corradini/Didattica/AP-20/

Language: Java - Size: 32.2 KB - Last synced at: 11 months ago - Pushed at: about 4 years ago - Stars: 0 - Forks: 0

adumitrescu2708/Map-Reduce-Paradigm-Pthreads

This assignment uses the Map-Reduce model for finding the perfect numbers existent in a large set of input files. The Mappers dynamically parse the input files and search for perfect numbers and the Reducers combine for each exponent the results.

Language: C++ - Size: 1.24 MB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

sskender/analysis-of-massive-datasets

Analysis of Massive Datasets FER labs

Language: Python - Size: 19 MB - Last synced at: 3 months ago - Pushed at: about 3 years ago - Stars: 2 - Forks: 0

cbuschka/aws-scatter-gather

Scatter gather with AWS lambda

Language: Python - Size: 592 KB - Last synced at: 4 months ago - Pushed at: over 4 years ago - Stars: 4 - Forks: 0

dragonly/pingcap_interview

pingcap 面试小作业

Language: Go - Size: 477 KB - Last synced at: 12 months ago - Pushed at: almost 5 years ago - Stars: 36 - Forks: 2

cloudposse-archives/terraform-aws-spotinst-mrscaler

Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS using a Spotinst AWS MrScaler resource

Size: 54.7 KB - Last synced at: 7 days ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

ThanosM97/ADBS-NTUA

This repository hosts a pyspark implementation of the k-means algorithm for the course Advanced Database Systems of NTUA ECE.

Language: Python - Size: 632 KB - Last synced at: about 1 year ago - Pushed at: about 5 years ago - Stars: 3 - Forks: 0

MartinCastroAlvarez/hadoop-hdfs-map-reduce-docker

Running Map Reduce in Hadoop using Docker

Language: Python - Size: 3.45 MB - Last synced at: 2 months ago - Pushed at: over 1 year ago - Stars: 2 - Forks: 1

thomasDoukas/NTUA_ATDS

Advanced Topics in Database Systems course of ECE National Technical University of Athens.

Language: Python - Size: 2.2 MB - Last synced at: about 1 year ago - Pushed at: over 3 years ago - Stars: 0 - Forks: 0

andrejanesic/Hadoop-Beginner-Exercise-Streaming-Word-Count-Sorted

Hadoop beginner exercise - using the Streaming API to determine the frequency of words in Wikipedia articles.

Language: Jupyter Notebook - Size: 29.3 KB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

andrejanesic/Hadoop-Beginner-Exercise-Streaming-Word-Count

Exercise of using the Streaming API with Hadoop to determine the word count of Wikipedia articles.

Language: Jupyter Notebook - Size: 31.3 KB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

Arekushi/word-count-map-reduce

Simple Word Count application with Map Reduce.

Language: Java - Size: 2.54 MB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0

97arushisharma/MyNews-Summarization-DesktopApp

Deloitte TechnoUtsav Project

Language: Python - Size: 20.3 MB - Last synced at: about 1 year ago - Pushed at: over 6 years ago - Stars: 0 - Forks: 0

manab-kb/Map-Reduce-Framework

A Map-Reduce framework built via Python - BD Project Sem V

Language: Python - Size: 1.26 MB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

manab-kb/YAMR

A Map-Reduce framework, built via Python.

Language: Python - Size: 143 KB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

vbugaevskii/sphere.info-retrieval-2.hw

Language: Jupyter Notebook - Size: 1.37 MB - Last synced at: about 1 year ago - Pushed at: over 6 years ago - Stars: 0 - Forks: 0

vbugaevskii/sphere.hadoop.hw

Language: Java - Size: 22.5 KB - Last synced at: about 1 year ago - Pushed at: about 8 years ago - Stars: 1 - Forks: 0

vbugaevskii/msu.hadoop.course

Homeworks for "Hadoop" course in CMC MSU

Language: Java - Size: 24.4 KB - Last synced at: about 1 year ago - Pushed at: over 7 years ago - Stars: 1 - Forks: 1

klebermagno/Artificial-Inteligence

The propouse of this project is organize some Artificial Inteligenc techinics and projects. So this project is structured in git submodules to others projects and README.md to documentate some techinics.

Size: 5.86 KB - Last synced at: about 1 year ago - Pushed at: almost 7 years ago - Stars: 1 - Forks: 0

dimits-ts/large-scale-data

Distributed computing for data science tasks, executed on a Ubuntu server.

Language: Java - Size: 27.9 MB - Last synced at: 3 months ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

MaorAssayag/Distributed-System-Programming-Cloud-Computing-and-Map-Reducce

A course from the Computer Science Department in Ben-gurion Unviesity of the Negev, Israel.

Language: Java - Size: 403 KB - Last synced at: about 1 year ago - Pushed at: almost 6 years ago - Stars: 2 - Forks: 2

PeterHUistyping/MIT6.824-6.5840-Distributed-Systems

MIT6.824 Lab implementation, Lecture Note

Language: Go - Size: 2.85 MB - Last synced at: 12 months ago - Pushed at: almost 2 years ago - Stars: 1 - Forks: 0

keshy/PromptSharder

Split large docs for efficient LLM processing. Inspired by MapReduce, it tackles scalability & empowers LLMs to analyze massive text datasets.

Language: Python - Size: 17.6 KB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

RedisGears/redisgears-py

RedisGears python client

Language: Python - Size: 72.3 KB - Last synced at: 9 days ago - Pushed at: almost 2 years ago - Stars: 27 - Forks: 11

skywardpixel/mit-ds-labs

MIT 6.824 Distributed Systems Engineering labs, Spring 2020.

Language: Go - Size: 1.33 MB - Last synced at: 3 months ago - Pushed at: about 5 years ago - Stars: 0 - Forks: 1

commoncrawl/cc-mrjob Fork of Smerity/cc-mrjob

Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

Language: Python - Size: 1020 KB - Last synced at: about 1 year ago - Pushed at: about 3 years ago - Stars: 166 - Forks: 65

Related Keywords

map-reduce 311 hadoop 87 python 49 big-data 46 spark 45 java 38 distributed-systems 24 mapreduce 24 distributed-computing 21 hadoop-mapreduce 18 machine-learning 16 hdfs 14 apache-spark 13 scala 11 python3 11 data-science 11 bigdata 11 pyspark 11 javascript 9 golang 9 reduce 9 parallel 9 sql 9 hive 8 parallel-computing 8 map 8 docker 7 kafka 7 aws 7 word-count 7 go 7 mongodb 7 big-data-analytics 6 data-analysis 6 nodejs 6 tf-idf 6 java-8 6 spark-sql 6 data-visualization 6 data-mining 6 lambda 5 cpp 5 langchain 5 numpy 5 jupyter-notebook 5 search-engine 5 c 5 clustering 5 information-retrieval 5 pandas 5 wordcount 5 inverted-index 5 k-means 4 kmeans 4 functional-programming 4 hpc 4 svm 4 cosine-similarity 4 databases 4 mpi 4 mrjob 4 kmeans-clustering 4 hadoop-hdfs 4 grpc 4 apache-hadoop 4 transducers 4 multithreading 4 julia 4 data-processing 4 parallel-programming 4 high-performance 4 docker-compose 4 mahout 4 rdd 4 k8s 3 stream-processing 3 cassandra 3 cluster 3 transform 3 reduction 3 reducer 3 hadoop-streaming 3 recommender-system 3 typescript 3 pagerank 3 emr 3 iterators 3 aws-emr 3 python-multiprocessing 3 iterator 3 iterate 3 maven 3 gutenberg 3 twitter-api 3 word-counter 3 raft 3 redis 3 hbase 3 algorithms 3 multiprocessing 3