distributed-file-system | Topic | Ecosyste.ms: Repos

seaweedfs/seaweedfs

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

Language: Go - Size: 69.7 MB - Last synced at: 2 days ago - Pushed at: 3 days ago - Stars: 24,619 - Forks: 2,406

ceph/ceph

Ceph is a distributed object, block, and file storage platform

Language: C++ - Size: 797 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 15,068 - Forks: 6,075

happyfish100/fastdfs

FastDFS is an open source high performance distributed file system (DFS). It's major functions include: file storing, file syncing and file accessing, and design for high capacity and load balance. Wechat/Weixin public account (Chinese Language): fastdfs

Language: C - Size: 10.7 MB - Last synced at: 4 days ago - Pushed at: about 1 month ago - Stars: 9,144 - Forks: 1,997

deepseek-ai/3FS

A high-performance distributed file system designed to address the challenges of AI training and inference workloads.

Language: C++ - Size: 5.66 MB - Last synced at: 16 days ago - Pushed at: 16 days ago - Stars: 8,898 - Forks: 885

cubefs/cubefs

cloud-native distributed storage

Language: Go - Size: 245 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 5,076 - Forks: 661

sjqzhang/go-fastdfs

go-fastdfs 是一个简单的分布式文件系统(私有云存储)，具有无中心、高性能，高可靠，免维护等优点，支持断点续传，分块上传，小文件合并，自动同步，自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

Language: Go - Size: 37.7 MB - Last synced at: 16 days ago - Pushed at: 5 months ago - Stars: 4,038 - Forks: 761

moosefs/moosefs

MooseFS Distributed Storage – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System / Software-Defined Storage

Language: C - Size: 6.59 MB - Last synced at: 9 days ago - Pushed at: 2 months ago - Stars: 1,797 - Forks: 218

leo-project/leofs

The LeoFS Storage System

Language: Erlang - Size: 30 MB - Last synced at: 14 days ago - Pushed at: about 5 years ago - Stars: 1,564 - Forks: 156

wenweihu86/raft-java

Raft Java implementation which is simple and easy to understand.

Language: Java - Size: 370 KB - Last synced at: 2 months ago - Pushed at: over 1 year ago - Stars: 1,212 - Forks: 415

roma-glushko/awesome-distributed-system-projects

🚀 List of distributed system projects for inspiration and learning to build distributed services from real world examples

Size: 509 KB - Last synced at: 13 days ago - Pushed at: about 1 year ago - Stars: 872 - Forks: 75

Hexilee/tifs

A distributed POSIX filesystem based on TiKV, with partition tolerance and strict consistency.

Language: Rust - Size: 327 KB - Last synced at: 12 days ago - Pushed at: 11 months ago - Stars: 362 - Forks: 23

hegongshan/File-System-Paper

Must-read Papers for File System (FS)

Size: 18.6 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 176 - Forks: 18

richardschneider/net-ipfs-engine

IPFS Core API implementation in .Net

Language: C# - Size: 110 MB - Last synced at: 11 days ago - Pushed at: almost 4 years ago - Stars: 110 - Forks: 50

freakmaxi/kertish-dos

Kertish-dos is a simple distributed object storage platform, implements object storage on a single distributed computer cluster, and provides interfaces for file/folder handling. Kertish-dos aims primarily for completely distributed operation without a single point of failure, scalable to the exabyte level.

Language: Go - Size: 4.33 MB - Last synced at: 25 days ago - Pushed at: 6 months ago - Stars: 109 - Forks: 9

pitchpoint-solutions/sfs

The legacy distributed object storage server developed by PitchPoint Solutions can store billions of large and small files using minimal resources. Object data is stored in replicated volumes implemented like Facebooks Haystack Object Store. Object metadata which essentially maps an object name to a volume position is stored in an elasticsearch index. (Development by PitchPoint Solutions has been discontinued)

Language: Java - Size: 2.03 MB - Last synced at: 2 months ago - Pushed at: over 2 years ago - Stars: 85 - Forks: 11

IBM/IPFSfB 📦

InterPlanetary File System for Business (IPFSfB) is an enterprise blockchain storage network based on InterPlanetary File System.

Language: Go - Size: 572 KB - Last synced at: about 1 month ago - Pushed at: about 6 years ago - Stars: 79 - Forks: 26

leil-io/saunafs

SaunaFS is a free-and open source, distributed POSIX file system inspired by Google File System.

Language: C++ - Size: 14.9 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 78 - Forks: 6

cholcombe973/rusix

Distributed filesystem in Rust

Language: Rust - Size: 325 KB - Last synced at: about 2 months ago - Pushed at: almost 2 years ago - Stars: 55 - Forks: 11

wenweihu86/distkv

Distributed KV Storage System based on Raft and RocksDB, can be use to store small files, like images.

Language: Java - Size: 312 KB - Last synced at: over 1 year ago - Pushed at: about 5 years ago - Stars: 55 - Forks: 21

fangpin/miniDFS

Simple version of Distributed File System on multiple threads

Language: C++ - Size: 24.4 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 50 - Forks: 21

chengdedeng/yfs

:house:超精简分布式对象存储

Language: Java - Size: 389 KB - Last synced at: 5 months ago - Pushed at: over 6 years ago - Stars: 47 - Forks: 10

mbajer42/ucz-dfs

A distributed file system written in Rust.

Language: Rust - Size: 76.2 KB - Last synced at: 5 months ago - Pushed at: over 4 years ago - Stars: 27 - Forks: 2

UnixJunkie/daft

DAFT Allows File Transfers

Language: OCaml - Size: 469 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 23 - Forks: 1

ks-amit/Distributed-Database

A travel agency app with a distributed database implemented from scratch!

Language: Python - Size: 776 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 17 - Forks: 6

putdotio/efes

A distributed filesystem evolved from MogileFS

Language: Go - Size: 580 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 16 - Forks: 1

MrCirca/zabbix-glusterfs

Monitoring GlusterFS Discovery storage with Zabbix 3.4

Language: Python - Size: 15.6 KB - Last synced at: about 1 year ago - Pushed at: over 6 years ago - Stars: 16 - Forks: 21

foreverbell/yfs

Yet Another File System (yfs) based on Paxos algorithm.

Language: C++ - Size: 186 KB - Last synced at: about 2 months ago - Pushed at: over 7 years ago - Stars: 16 - Forks: 5

abbyssoul/libstyxe

Implementation of 9P parser and message writer.

Language: C++ - Size: 420 KB - Last synced at: 10 months ago - Pushed at: almost 4 years ago - Stars: 14 - Forks: 3

PasaLab/dfs-perf

A general performance test framework for Distributed File System

Language: Java - Size: 2.7 MB - Last synced at: about 1 year ago - Pushed at: over 4 years ago - Stars: 14 - Forks: 4

SyncFree/antidote-fs 📦

A file system backed by AntidoteDB.

Language: Java - Size: 747 KB - Last synced at: about 1 year ago - Pushed at: almost 4 years ago - Stars: 12 - Forks: 0

James-QiuHaoran/distributed-computing-platform-mapreduce

This repository contains a simple Hadoop-like (MapReduce) distributed computing platform implemented in Java. It is extended from a course project at UIUC awarded the best Java version implementation and it's open-sourced for reference.

Language: Java - Size: 454 KB - Last synced at: almost 2 years ago - Pushed at: about 4 years ago - Stars: 12 - Forks: 4

StrumentiResistenti/MagmaFS

Experimental distributed filesystem for Linux

Language: Shell - Size: 1.16 MB - Last synced at: over 2 years ago - Pushed at: over 9 years ago - Stars: 10 - Forks: 2

lmco/ChaordicLedger

The ChaordicLedger is the implementation of a design for a combination of Distributed Ledger Technology (DLT) and a Distributed File System (DFS) to create a secure, enterprise-grade platform for storing interlinked project artifacts.

Language: Shell - Size: 123 MB - Last synced at: 23 days ago - Pushed at: 23 days ago - Stars: 9 - Forks: 5

b4rti/entropy

Distributed Docker Volume System

Language: Go - Size: 7.21 MB - Last synced at: 2 months ago - Pushed at: almost 7 years ago - Stars: 9 - Forks: 1

WilliamX1/cse-2021

A distributed file system similar to Google File System (GFS).

Language: C++ - Size: 158 MB - Last synced at: about 2 months ago - Pushed at: over 3 years ago - Stars: 7 - Forks: 0

BDFD-LearningGround/Coursera_Modern-Big-Data-Analysis-with-SQL-Specialization_OP

Quizzes & Assignment Solutions for Modern Big Data Analysis with SQL Specialization on Coursera. Also included a few resources on side that I found helpful.

Size: 26 MB - Last synced at: 10 months ago - Pushed at: over 3 years ago - Stars: 7 - Forks: 0

chaitanya100100/Google-File-System

A python implementation of Google File System

Language: Python - Size: 24.4 KB - Last synced at: over 1 year ago - Pushed at: almost 6 years ago - Stars: 7 - Forks: 1

shubhamsawantsjsu/Distributed-File-Storage-System

An open-source, scalable, decentralized, robust, heterogeneous file storage solution which is fault tolerant, replicated, distributed and lets you upload, download, and see the catalog of other cluster with low latency and LRU cache capabilities.

Language: Python - Size: 13.5 MB - Last synced at: almost 2 years ago - Pushed at: about 6 years ago - Stars: 7 - Forks: 3

adobe/S3-FileSystem

An implementation of the Hadoop FileSystem contract backed by AWS S3 and DynamoDB

Language: Java - Size: 459 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 6 - Forks: 5

kfrancischen/galaxy

Simple distributed file system based on gRPC

Language: C++ - Size: 813 KB - Last synced at: 4 days ago - Pushed at: about 2 years ago - Stars: 6 - Forks: 1

teimurjan/go-p2p

Peer-to-peer local network

Language: Go - Size: 2.76 MB - Last synced at: 2 months ago - Pushed at: over 2 years ago - Stars: 6 - Forks: 1

hordiales/hdfs-streaming

HTML5 video streaming servlet from Hadoop Distributed File System

Language: Java - Size: 18.6 KB - Last synced at: over 2 years ago - Pushed at: almost 7 years ago - Stars: 6 - Forks: 4

weswu8/bloblib

The core library for blobfs and blobfs-win project, this tool can help you mount the Azure Blob storage as the local disk driver, no matter it is a Linux system or a Windows system.

Language: Java - Size: 1.68 MB - Last synced at: over 1 year ago - Pushed at: about 7 years ago - Stars: 6 - Forks: 4

WenbinZhu/FileStAck

A Distributed File System

Language: Java - Size: 151 KB - Last synced at: about 2 years ago - Pushed at: over 8 years ago - Stars: 6 - Forks: 1

hegongshan/file-system

File System Book

Size: 10.3 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 5 - Forks: 1

last-saiyan/distributed-fs

Distributed file system based on GFS paper

Language: Go - Size: 126 KB - Last synced at: about 1 year ago - Pushed at: almost 4 years ago - Stars: 5 - Forks: 2

codenoid/sdfs

a Connector |> Distributed File System based on Symlinking (that fit author need)

Language: Go - Size: 6.19 MB - Last synced at: about 1 month ago - Pushed at: over 5 years ago - Stars: 5 - Forks: 0

aymaneallaoui/distributed-file-system

an advanced implementation of a distributed file system in Golang

Language: Go - Size: 21.5 KB - Last synced at: about 2 months ago - Pushed at: 9 months ago - Stars: 4 - Forks: 1

deric/puppet-beegfs

Manage BeeGFS parallel system installations.

Language: HTML - Size: 446 KB - Last synced at: 27 days ago - Pushed at: 11 months ago - Stars: 4 - Forks: 10

prithvianilk/rdfs

An attempt to make a reliable, distributed file system inspired by HDFS

Language: Java - Size: 437 KB - Last synced at: over 1 year ago - Pushed at: almost 3 years ago - Stars: 4 - Forks: 2

DistributedWeb/ddrive

A secure, real-time distributed file system for the dWeb.

Language: JavaScript - Size: 153 KB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 4 - Forks: 2

Jason916/ipapk-server-v2 Fork of zhao0/ipapk-server

Create a HTTPS Server to install iOS ipa and Android apk

Language: C - Size: 651 KB - Last synced at: 14 days ago - Pushed at: over 5 years ago - Stars: 4 - Forks: 1

kkyrenc/mini-gfs

MiniGFS is a Python-based distributed file system inspired by GFS.

Language: Python - Size: 43.9 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 3 - Forks: 0

apkp7/quick-store

Efficient and Stable Peer-To-Peer Distributed Hash Table System

Language: Python - Size: 19 MB - Last synced at: 6 months ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 0

zzhtttsss/catdfs

A lightweight distributed file system implemented with Go

Size: 19.2 MB - Last synced at: almost 2 years ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 0

yyywang/flask-lightdfs

Flask-LightDFS is an extension, distributed storage system, for Flask.

Language: Python - Size: 102 KB - Last synced at: almost 2 years ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 2

Hacky-DH/moosefs-client

Golang API for moosefs client

Language: Go - Size: 146 KB - Last synced at: 12 months ago - Pushed at: about 6 years ago - Stars: 3 - Forks: 0

AlexanderSergeevich1985/DistributedService

Client and service implementation for distributed computation

Language: C++ - Size: 219 KB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 3 - Forks: 1

mohitreddy1996/Distributed-File-System

Distributed File System consisting of a multi threaded name server and storage servers capable of replicating data on storage servers efficiently.

Language: Java - Size: 1.96 MB - Last synced at: almost 2 years ago - Pushed at: about 8 years ago - Stars: 3 - Forks: 0

9231058/ChFTP 📦

Distributed System for File Sharing

Language: Python - Size: 269 KB - Last synced at: 3 months ago - Pushed at: over 9 years ago - Stars: 3 - Forks: 0

geekmug/syncfs

SyncFS, a Synchronous File System

Language: Shell - Size: 465 KB - Last synced at: about 2 years ago - Pushed at: over 11 years ago - Stars: 3 - Forks: 1

ZainAli104/distributed-file-system-go

This project offers a peer-to-peer content-addressable distributed file storage in Go with a peer-to-peer library built on top of TCP from scratch. It also supports data encryption during storage and transmission

Language: Go - Size: 70.3 KB - Last synced at: 2 days ago - Pushed at: 3 months ago - Stars: 2 - Forks: 0

arpan491/Distributed-FileSystem

An open-source, scalable, decentralized, robust, heterogeneous file storage solution which is fault tolerant, replicated, distributed and lets you upload, download, and see the catalog of other cluster with low latency and LRU cache capabilities.

Language: Python - Size: 1.61 MB - Last synced at: 8 months ago - Pushed at: 9 months ago - Stars: 2 - Forks: 0

ritsource/hiro

Hiro is a very simple network-attached (distributed) file system. It's fast, configurable, and horizontally scalable.

Language: Rust - Size: 122 KB - Last synced at: 12 months ago - Pushed at: 12 months ago - Stars: 2 - Forks: 0

Toubat/IDunno

IDunno is a fault-tolerent Distributed Machine Learning Cluster, with a unidirectional, ring-based, ping-ack style Simple Distributed File System implemented using Golang, gRPC, Protobuf, PyTorch, and React.

Language: Go - Size: 14.1 MB - Last synced at: 9 days ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0

emilstahl97/Data-Intensive-Computing-ID2221 📦

The course complements distributed systems courses, with a focus on processing, storing and analyzing massive data.

Language: Jupyter Notebook - Size: 939 MB - Last synced at: 2 months ago - Pushed at: over 3 years ago - Stars: 2 - Forks: 1

menuka94/java-distributed-file-system

Fault-tolerant Distributed File System

Language: Java - Size: 2.15 MB - Last synced at: about 1 year ago - Pushed at: over 3 years ago - Stars: 2 - Forks: 0

oss-tsukuba/gfarm-spack

Spack repository for Gfarm File System

Language: Python - Size: 15.6 KB - Last synced at: 4 days ago - Pushed at: over 4 years ago - Stars: 2 - Forks: 1

JohnPapad/Distributed-File-Management-Tool

A distributed File Management Tool in C++, using processes, forking, named-pipes (FIFOs) and low-level I/O.

Language: C++ - Size: 301 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 2 - Forks: 0

hasanimran96/Project-Distributed-File-System

Final Project for my distributed systems course

Language: Python - Size: 458 KB - Last synced at: 3 months ago - Pushed at: over 5 years ago - Stars: 2 - Forks: 1

tibold/docker-volume-sharedfs

A docker volume driver for shared filesystems.

Language: Go - Size: 46.9 KB - Last synced at: 3 days ago - Pushed at: about 6 years ago - Stars: 2 - Forks: 1

wuchenkang/DistFS

A simple distributed file system.

Size: 1000 Bytes - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 2 - Forks: 0

Hacky-DH/fs-benchmark

perftest fs

Language: Python - Size: 14.6 KB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 2 - Forks: 0

snake0/lab-cse

yfs labs for SJTU CSE 2018 course. All pass test.

Language: C++ - Size: 30.8 MB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 2 - Forks: 0

mateuszkalinowski/DistributedFileSystemClient

DataNode for a sockets based simple distributed file system

Language: Java - Size: 8.79 KB - Last synced at: over 2 years ago - Pushed at: about 7 years ago - Stars: 2 - Forks: 0

tony-hong/yfs

Language: C++ - Size: 181 KB - Last synced at: over 1 year ago - Pushed at: over 8 years ago - Stars: 2 - Forks: 1

jetmuffin/whalefs

Simple distributed file system written in go

Language: Go - Size: 140 KB - Last synced at: 12 months ago - Pushed at: over 8 years ago - Stars: 2 - Forks: 0

pxnt/distributed-file-storage-go

A robust Go-based distributed file storage system with peer-to-peer architecture, content-addressable storage, and stream-based transfers for reliable networked data management.

Language: Go - Size: 17.6 KB - Last synced at: 22 days ago - Pushed at: 22 days ago - Stars: 1 - Forks: 0

dipghoshraj/tejas-fs

Building a distributed file system (DFS) , manages data storage across multiple nodes (servers), enabling scalability, fault tolerance, and high availability

Language: Go - Size: 289 KB - Last synced at: 22 days ago - Pushed at: 2 months ago - Stars: 1 - Forks: 0

uttom-akash/distributed-consensus-file-system

A Consensus file system comprises miners and clients (end-user).

Language: Go - Size: 432 KB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

Elia-Renzoni/RenzoFS

It is a distributed file system that allows users to fill and query their csv files, using json, and share them. Golang + Microservices + RESTful :sparkles:

Language: Go - Size: 491 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0

comevback/UPC

A computational system using Docker, buildpack to simplify the processing of all kind of tasks.

Language: JavaScript - Size: 134 MB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 1 - Forks: 0

MarcoFontana48/GoogleChubby

Implementation of Google's Chubby distributed file system in Java using Raft consensus algorithm

Language: Java - Size: 921 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

anas2204/Simple_DFS Fork of imabhishekl/Simple_DFS

A Simple Distributed File System Designed for USER to store their file on remote system

Language: Java - Size: 1.02 MB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 1 - Forks: 0

tobias-mayer/minidfs

minidfs is a tiny distributed file system inspired by Google's GFS

Language: Go - Size: 160 KB - Last synced at: 12 months ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 0

gabrielluizbh/SMS-DFS

Script para migração de DFS com Storage Migration Service.

Language: PowerShell - Size: 5.86 KB - Last synced at: 3 months ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0

huaxiaozhong1/on-premises-high-available-kubernetes-distributed-storage-cluster

On-prem-kubernetes-ceph-rook-high-available-distributed-storage-cluster

Language: JavaScript - Size: 13.1 MB - Last synced at: over 1 year ago - Pushed at: almost 3 years ago - Stars: 1 - Forks: 0

yashichawla/YAH

A mini-Hadoop clone capable of performing all DFS functionalities through a CLI

Language: Python - Size: 118 KB - Last synced at: 3 months ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 0

FedericoPonzi/MyDFS

A simple distributed file system.

Language: HTML - Size: 4.22 MB - Last synced at: 2 months ago - Pushed at: almost 4 years ago - Stars: 1 - Forks: 0

kezzyhko/distributed-file-system 📦

Self-implemented distributed file system

Language: C++ - Size: 141 KB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 0

JamieOw119/DistributedObjectStorage

A decentralized distributed object storage file system implemented by Golang

Language: Go - Size: 5.39 MB - Last synced at: 12 months ago - Pushed at: almost 5 years ago - Stars: 1 - Forks: 0

madkne/kdfs

KDFS is a distributed file system containse multiple nodes and a node can be a laptop, old computer and all devices can run a python web server and connect to local network!

Language: Python - Size: 108 KB - Last synced at: about 2 years ago - Pushed at: about 5 years ago - Stars: 1 - Forks: 0

SCUTMSC/SCUTMSC-DFS

A distributed file system powered by and applied on South China University of Technology Microsoft Student Club.

Language: Go - Size: 374 KB - Last synced at: 12 months ago - Pushed at: over 5 years ago - Stars: 1 - Forks: 2

khaledsabry97/MegaDistribute

Distributed File and Database System, to make it more available to the users connected and to allow users to upload/download their files and also replicate it in case if one of the nodes was down to ensure availability

Language: Python - Size: 45.4 MB - Last synced at: over 2 years ago - Pushed at: about 6 years ago - Stars: 1 - Forks: 0