Topic: "deduplicate"
knjcode/imgdupes
Identifying and removing near-duplicate images using perceptual hashing.
Language: Python - Size: 928 KB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 357 - Forks: 23
tc39/proposal-array-unique
ECMAScript proposal for Deduplicating method of Array
Language: TypeScript - Size: 62.5 KB - Last synced at: 20 days ago - Pushed at: about 3 years ago - Stars: 140 - Forks: 7
evrignaud/fim
File Integrity Manager -
Language: Java - Size: 5.07 MB - Last synced at: 19 days ago - Pushed at: about 1 month ago - Stars: 122 - Forks: 15
sysulq/dataloader-go
Go implementation of Facebook's DataLoader with 200+ lines of code.
Language: Go - Size: 216 KB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 51 - Forks: 1
shevchenkoartem/lastfm-smart-deduper
JS script that allows you to remove duplicates from your Last.fm scrobbles library.
Language: JavaScript - Size: 1.74 MB - Last synced at: almost 2 years ago - Pushed at: about 5 years ago - Stars: 48 - Forks: 2
WSH032/sd-webui-fast-dataset-maker
A funny extension that integrates image-browsing , downloader , deduplicate , cluster , can quickly collect, classify and process your images. | 一个有趣的扩展，整合了 图库，下载，去重，聚类 ，可以快速搜集、分类、处理你的图片。
Language: Python - Size: 124 KB - Last synced at: 19 days ago - Pushed at: over 1 year ago - Stars: 23 - Forks: 1
routineLife1/MultiPassDedup
Efficient Deduplicate for Anime Video Frame Interpolation
Language: Python - Size: 6.57 MB - Last synced at: 18 days ago - Pushed at: 2 months ago - Stars: 19 - Forks: 0
rsalmei/refine
Refine your file collections using Rust!
Language: Rust - Size: 780 KB - Last synced at: 7 days ago - Pushed at: 8 days ago - Stars: 13 - Forks: 0
thijse/DSPhotoSorter
A command line tool for sorting photo's from the Synology DSPhoto auto-upload tool
Language: C# - Size: 50.8 KB - Last synced at: 20 days ago - Pushed at: over 7 years ago - Stars: 9 - Forks: 1
svandriel/cachify-promise
Smart caching for promises. Like memoization, but better.
Language: TypeScript - Size: 558 KB - Last synced at: 3 days ago - Pushed at: 4 months ago - Stars: 8 - Forks: 0
routineLife1/AVFDU 📦
动漫一拍N自动识别算法
Language: Python - Size: 65.4 KB - Last synced at: 8 months ago - Pushed at: over 1 year ago - Stars: 8 - Forks: 1
mahendraHegde/node-idempotency
makes any request idempotent across nodejs frameworks like nestjs, express, fastify
Language: TypeScript - Size: 639 KB - Last synced at: 4 days ago - Pushed at: 9 months ago - Stars: 7 - Forks: 3
yaroslaff/hashget
Deduplication/backup tool with extremely high 'compression' rate
Language: Python - Size: 194 KB - Last synced at: 14 days ago - Pushed at: almost 2 years ago - Stars: 7 - Forks: 2
infogulch/uniq
Package uniq provides primitives for getting the first unique elements of (aka deduplicate) your existing sorted sort.Interface.
Language: Go - Size: 223 KB - Last synced at: 24 days ago - Pushed at: almost 11 years ago - Stars: 7 - Forks: 1
gblach/reflicate
Deduplicate data by creating reflinks between identical files.
Language: Rust - Size: 127 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 5 - Forks: 0
mterron/swuniq
A command-line tool for deduplicating entries in a file or stream with constant memory usage
Language: C - Size: 124 KB - Last synced at: 5 months ago - Pushed at: about 3 years ago - Stars: 5 - Forks: 1
bmiller1009/deduper
General deduping engine for JDBC sources with output to JDBC/csv targets
Language: Kotlin - Size: 1.23 MB - Last synced at: 5 months ago - Pushed at: over 4 years ago - Stars: 5 - Forks: 0
ojarva/maildir-deduplicate
Deduplicates maildir contents using hard links.
Language: Python - Size: 8.79 KB - Last synced at: 18 days ago - Pushed at: over 4 years ago - Stars: 4 - Forks: 1
helloall1900/vhash
A C++ reimplementation of Near Duplicate Video Detection - Get a 64-bit comparable hash-value for any video (Video Hash).
Language: C++ - Size: 3.23 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 3 - Forks: 2
mimose/deduplicate
😺一款用于防止重复攻击的组件，基于SPI机制实现核心功能 >>> 防重放组件 (A component used to prevent duplicated attacks, such as repeat request, replay attack)
Language: Java - Size: 103 KB - Last synced at: over 1 year ago - Pushed at: over 3 years ago - Stars: 3 - Forks: 1
thomaswyrick/febrl
A Python package designed to allow health, biomedical and other researchers to clean (standardise) and deduplicate or link data sets of all sizes faster, with less effort and with improved quality.
Language: Python - Size: 431 KB - Last synced at: over 1 year ago - Pushed at: about 4 years ago - Stars: 3 - Forks: 0
stdlib-js/iter-unique-by-hash
Create an iterator which returns unique values according to a hash function.
Language: JavaScript - Size: 993 KB - Last synced at: 15 days ago - Pushed at: 20 days ago - Stars: 2 - Forks: 0
stdlib-js/iter-unique
Create an iterator which returns unique values.
Language: JavaScript - Size: 1.12 MB - Last synced at: 14 days ago - Pushed at: about 1 month ago - Stars: 2 - Forks: 0
stdlib-js/iter-union
Create an iterator which returns the union of two or more iterators.
Language: JavaScript - Size: 903 KB - Last synced at: 14 days ago - Pushed at: about 1 month ago - Stars: 2 - Forks: 0
stdlib-js/iter-unique-by
Create an iterator which returns unique values according to a predicate function.
Language: JavaScript - Size: 960 KB - Last synced at: 15 days ago - Pushed at: 5 months ago - Stars: 2 - Forks: 0
jshwi/borgini 📦
ini config for borg backup
Language: Python - Size: 210 KB - Last synced at: 8 days ago - Pushed at: 12 months ago - Stars: 2 - Forks: 0
jaymoulin/google-musicmanager-dedup-api
Deduplication API for Google MusicManager
Language: Python - Size: 59.6 KB - Last synced at: about 2 months ago - Pushed at: 12 months ago - Stars: 2 - Forks: 1
rongrimes/zipfile-dedup
Project to take two similar zipfiles, and to dedupe files that have the same tiemstamp in the older file.
Language: Python - Size: 20.5 KB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 2 - Forks: 1
stdlib-js/iter-dedupe
Create an iterator which removes consecutive duplicated values.
Language: JavaScript - Size: 1010 KB - Last synced at: 14 days ago - Pushed at: 20 days ago - Stars: 1 - Forks: 0
stdlib-js/iter-dedupe-by
Create an iterator which removes consecutive values that resolve to the same value according to a provided function.
Language: JavaScript - Size: 1.02 MB - Last synced at: 7 days ago - Pushed at: 27 days ago - Stars: 1 - Forks: 0
stdlib-js/array-base-to-deduped
Copy elements to a new generic array after removing consecutive duplicated values.
Language: JavaScript - Size: 181 KB - Last synced at: 14 days ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 0
stdlib-js/iter-intersection
Create an iterator which returns the intersection of two or more iterators.
Language: JavaScript - Size: 943 KB - Last synced at: 14 days ago - Pushed at: 3 months ago - Stars: 1 - Forks: 0
stdlib-js/array-base-dedupe
Remove consecutive duplicated values.
Language: JavaScript - Size: 227 KB - Last synced at: 14 days ago - Pushed at: 5 months ago - Stars: 1 - Forks: 0
stdlib-js/iter-intersection-by-hash
Create an iterator which returns the intersection of two or more iterators according to a hash function.
Language: JavaScript - Size: 843 KB - Last synced at: 15 days ago - Pushed at: 5 months ago - Stars: 1 - Forks: 0
hekmon/deduper
Analyse 2 paths to found identical files and hard link them to save space
Language: Go - Size: 151 KB - Last synced at: about 2 months ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0
Nek-12/HostsTools
A simple app for managing hosts files. Available for Win and Linux.
Language: C++ - Size: 5.11 MB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 0
gkjohnson/webpack-script-guard
Webpack loader for guarding against duplicate scripts in separate bundles
Language: HTML - Size: 259 KB - Last synced at: 30 days ago - Pushed at: over 7 years ago - Stars: 1 - Forks: 0
oleg-putseiko/function-performer
Performer providing API for debounce, throttle and deduplication functions
Language: TypeScript - Size: 65.4 KB - Last synced at: 4 days ago - Pushed at: 13 days ago - Stars: 0 - Forks: 0
alxelerator/dedupUMI
Tool extracts the best representative exact duplicate sequences from PE fastq files using UMI approach (reference alignment free).
Language: Perl - Size: 13.7 MB - Last synced at: 17 days ago - Pushed at: 17 days ago - Stars: 0 - Forks: 0
jellyterra/fs-dedup
File system deduplication utility.
Language: Go - Size: 4.88 KB - Last synced at: 2 months ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0
Exisi/UText
多行与行内文本去重工具
Language: JavaScript - Size: 279 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
yugn/yadupe
Yet another tool to find and remove duplicate files.
Language: Python - Size: 735 KB - Last synced at: 8 days ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 1
grjan7/deduplicate-array
Returns duplicates-removed array -- has options for case-sensitivity and strict-typing.
Language: JavaScript - Size: 19.5 KB - Last synced at: 11 days ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
localghost/dedup
Find and remove duplicated files.
Language: Rust - Size: 9.77 KB - Last synced at: 25 days ago - Pushed at: almost 3 years ago - Stars: 0 - Forks: 0
anna-ringwood/cleaning-deduplicating-donor-data
This repository holds the code files used in an undergraduate data wrangling project from March - August 2021.
Size: 40 KB - Last synced at: about 1 year ago - Pushed at: about 3 years ago - Stars: 0 - Forks: 0
EvanCarroll/ytdl-clean
Dedupe from different versions of YouTube download
Language: Python - Size: 4.88 KB - Last synced at: 7 days ago - Pushed at: about 3 years ago - Stars: 0 - Forks: 0
