GitHub topics: duplicates
StephaneCouturier/Katalog
Katalog is an application to manage catalogs of disks and files to search and get statistics.
Language: C++ - Size: 12.2 MB - Last synced at: about 15 hours ago - Pushed at: about 15 hours ago - Stars: 81 - Forks: 7
qarmin/czkawka
Multi functional app to find duplicates, empty folders, similar images etc.
Language: Rust - Size: 4.61 MB - Last synced at: 2 days ago - Pushed at: 4 days ago - Stars: 23,521 - Forks: 739
sahib/rmlint
Extremely fast tool to remove duplicates and other lint from your filesystem
Language: C - Size: 12.4 MB - Last synced at: about 18 hours ago - Pushed at: 10 days ago - Stars: 2,078 - Forks: 137
jaffster595/Duplinator
A tool to find duplicate images in a folder
Language: Python - Size: 88.9 MB - Last synced at: 2 days ago - Pushed at: 3 days ago - Stars: 1 - Forks: 0
kucherenko/jscpd
Copy/paste detector for programming source code.
Language: TypeScript - Size: 9.13 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 4,866 - Forks: 211
veltzer/pyunique
Pyunique helps you get rid of duplicate files
Language: Python - Size: 905 KB - Last synced at: 9 days ago - Pushed at: 9 days ago - Stars: 0 - Forks: 0
cloud-py-api/mediadc
Nextcloud Media Duplicate Collector application
Language: PHP - Size: 91.1 MB - Last synced at: 8 days ago - Pushed at: 8 days ago - Stars: 98 - Forks: 8
scinos/yarn-deduplicate
Deduplication tool for yarn.lock files
Language: TypeScript - Size: 7.27 MB - Last synced at: 4 days ago - Pushed at: 8 days ago - Stars: 1,389 - Forks: 57
mackgorski/ai-duplicate-detector
AI-Powered GitHub Issue Duplicates & Relations Detector
Language: Python - Size: 52.7 KB - Last synced at: 11 days ago - Pushed at: 11 days ago - Stars: 0 - Forks: 0
PJDude/dude
Duplicates Detector is a cross-platform GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for efficient and convenient operation.
Language: Python - Size: 4.46 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 141 - Forks: 11
scrubbbbs/cbird
Command-line program for Content-Based Image Retrieval of images and videos. Includes tools for general search and de-duplication.
Language: C++ - Size: 12.7 MB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 119 - Forks: 5
7room/aya
Disk Usage Analyzer & Duplicate File Finder
Size: 71.3 KB - Last synced at: 2 days ago - Pushed at: over 2 years ago - Stars: 12 - Forks: 0
microsoft/near-duplicate-code-detector
A simple tool for detecting near-duplicate source code
Language: C# - Size: 38.1 KB - Last synced at: 7 days ago - Pushed at: 7 months ago - Stars: 100 - Forks: 31
twpayne/find-duplicates
Find duplicate files quickly.
Language: Go - Size: 82 KB - Last synced at: about 19 hours ago - Pushed at: 3 months ago - Stars: 57 - Forks: 1
raspi/duplikaatti
Remove duplicate files.
Language: Go - Size: 33.2 KB - Last synced at: 19 days ago - Pushed at: over 3 years ago - Stars: 18 - Forks: 1
Canop/backdown
A deduplicator
Language: Rust - Size: 541 KB - Last synced at: 5 days ago - Pushed at: 8 months ago - Stars: 132 - Forks: 7
src-d/gemini
Advanced similarity and duplicate source code at scale.
Language: Scala - Size: 7 MB - Last synced at: 10 days ago - Pushed at: almost 6 years ago - Stars: 55 - Forks: 16
kristiankoskimaki/vidupe
Vidupe is a program that can find duplicate and similar video files. V1.211 released on 2019-09-18, Windows exe here:
Language: C++ - Size: 266 KB - Last synced at: about 1 month ago - Pushed at: over 5 years ago - Stars: 176 - Forks: 18
eyalroz/removedupes
Remove Duplicate Messages
Language: JavaScript - Size: 8.56 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 93 - Forks: 7
nbari/backup
Command line tool for creating encrypted backups avoiding duplicates
Language: Rust - Size: 75.2 KB - Last synced at: 7 days ago - Pushed at: about 1 month ago - Stars: 2 - Forks: 1
DeaDSouL/dugu 📦
Find, remove and avoid duplicates with dugu: The Duplicates Guru
Language: Python - Size: 85.9 KB - Last synced at: about 1 month ago - Pushed at: over 1 year ago - Stars: 14 - Forks: 4
gacarrillor/AppendFeaturesToLayer
QGIS Processing plugin to add an algorithm for upserting features from a source vector layer to an existing target vector layer.
Language: Python - Size: 162 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 25 - Forks: 5
rsalmei/refine
Refine your file collections using Rust!
Language: Rust - Size: 783 KB - Last synced at: 7 days ago - Pushed at: 14 days ago - Stars: 13 - Forks: 0
F483/dejavu
Quickly detect already witnessed data.
Language: Go - Size: 305 KB - Last synced at: 6 days ago - Pushed at: 10 months ago - Stars: 157 - Forks: 5
arikw/outlook-duplicated-items-remover
A VBA script that finds and moves duplicated items in selected outlook folders
Language: VBA - Size: 151 KB - Last synced at: about 2 months ago - Pushed at: over 4 years ago - Stars: 22 - Forks: 2
milkmansson/plex-seeDuplicates
Find duplicates in your Plex library.
Language: Python - Size: 23.4 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0
mkearney/funique
⌚️ A faster unique() function
Language: R - Size: 7.15 MB - Last synced at: about 1 month ago - Pushed at: over 6 years ago - Stars: 19 - Forks: 0
shafirahmad/pydeduper
Duplicate file finder - with % duplication of folders
Language: Python - Size: 30.3 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0
rix4uni/unew
A tool combined of 2 commands features in 1 sort and tee for adding new lines to files, skipping duplicates
Language: Go - Size: 49.8 KB - Last synced at: 30 days ago - Pushed at: 4 months ago - Stars: 8 - Forks: 1
vuolter/deplicate 📦
Advanced Duplicate File Finder for Python
Language: Python - Size: 137 KB - Last synced at: 3 days ago - Pushed at: over 4 years ago - Stars: 77 - Forks: 17
mesqueeb/compare-anything
Compares objects and tells you which props are duplicate, and props are only present once.
Language: TypeScript - Size: 666 KB - Last synced at: 1 day ago - Pushed at: 3 months ago - Stars: 10 - Forks: 0
Robb-Fr/fast-dupes-finder
This repository proposes clean, fast and shell based scripts for identifying finding duplicate files in a folder.
Language: Shell - Size: 17.6 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0
thomas694/finddupe
Enhanced version of finddupe, a duplicate file detector for Windows
Language: C - Size: 204 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 9 - Forks: 0
prowide/prowide-integrator-examples
Source code examples for "Prowide Integrator"
Language: Java - Size: 8.55 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 10 - Forks: 17
kevinpollet/pocket-deduper
Remove duplicates from your Pocket list.
Language: Go - Size: 6.55 MB - Last synced at: about 1 month ago - Pushed at: over 3 years ago - Stars: 17 - Forks: 0
rasheedsulayman/DuplicateContactsRemover
📒 A simple app to optimize your address book and remove duplicate contacts.
Language: Kotlin - Size: 30.9 MB - Last synced at: 22 days ago - Pushed at: about 2 years ago - Stars: 22 - Forks: 10
kouhin/redux-dataloader
Loads async data for Redux apps focusing on preventing duplicated requests and dealing with async dependencies.
Language: JavaScript - Size: 108 KB - Last synced at: 1 day ago - Pushed at: over 7 years ago - Stars: 139 - Forks: 3
raspi/samanlainen
Delete duplicate files
Language: Rust - Size: 54.7 KB - Last synced at: 29 days ago - Pushed at: about 2 years ago - Stars: 8 - Forks: 1
NicolasBizzozzero/dupe_eraser
A command-line tool which automate the deletion of duplicate files based on their hash or perceptual-hash.
Language: Python - Size: 54.7 KB - Last synced at: about 1 month ago - Pushed at: 4 months ago - Stars: 13 - Forks: 0
Navid2zp/dups
A CLI tool to find/remove duplicate files supporting multi-core and different algorithms (MD5, SHA256, and XXHash).
Language: Go - Size: 62.5 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 25 - Forks: 3
tasleson/duplihere
Copy & Paste finder for structured text files.
Language: Rust - Size: 104 KB - Last synced at: about 1 month ago - Pushed at: 7 months ago - Stars: 5 - Forks: 1
vuolter/deplicate-cli
Command Line Interface for deplicate
Language: Python - Size: 37.1 KB - Last synced at: about 1 month ago - Pushed at: over 7 years ago - Stars: 3 - Forks: 1
clement-berard/go-imap-backup
A collection of Go tools for managing IMAP emails, featuring backup capabilities and duplicate detection/cleanup.
Language: Go - Size: 17.6 KB - Last synced at: about 1 month ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0
FreddyFunk/ddk
DeDuplicationKit: Advanced File Storage Deduplication
Language: C++ - Size: 192 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0
dgudim/qt_disk-deduper
A desktop app that will help you find and deal with file duplicates on you drive
Language: C++ - Size: 135 KB - Last synced at: 1 day ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 1
eliasfloreteng/bitwarden_find_duplicates
Find duplicate logins based on domain, from Bitwarden export. Open source for your safety.
Language: HTML - Size: 34.2 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 34 - Forks: 8
softonus-io/prettier-plugin-duplicate-remover
A Prettier plugin that removes duplicate class names in class and className attributes, ensuring cleaner, more efficient code in frontend projects like React, Vue.js, and Angular.
Language: JavaScript - Size: 11.7 KB - Last synced at: 12 days ago - Pushed at: 6 months ago - Stars: 2 - Forks: 1
bkb3/duplicate-bib-fix
Small python script to check and replace duplicated bib entries in your .tex files
Language: TeX - Size: 358 KB - Last synced at: 2 months ago - Pushed at: about 4 years ago - Stars: 2 - Forks: 0
webis-de/sigir20-sampling-bias-due-to-near-duplicates-in-learning-to-rank
Sampling Bias Due to Near-Duplicates in Learning to Rank
Language: Kotlin - Size: 51.3 MB - Last synced at: about 1 month ago - Pushed at: 7 months ago - Stars: 2 - Forks: 2
SuperJMN/DeDup
Tool to detect duplicates and copy them to a curated directory (without duplicates)
Language: C# - Size: 2.81 MB - Last synced at: 3 days ago - Pushed at: 7 months ago - Stars: 0 - Forks: 0
Intera/redmine_subject_autocomplete
makes the new issue subject field show an autocomplete that lists existing issues to prevent duplicate tickets
Language: Ruby - Size: 57.6 KB - Last synced at: about 1 month ago - Pushed at: 8 months ago - Stars: 0 - Forks: 1
VMC10/Simple-Duplicate-Cleaner
A simple app written in Python to delete duplicate files
Language: Python - Size: 2.93 KB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0
guicamest/GDuplicate-Finder
GDuplicate Finder - A Groovy way to find duplicates among your computer and network shares!
Language: Groovy - Size: 49.7 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 30 - Forks: 7
LyonSyonII/akin
Rust crate for writing repetitive code easier and faster.
Language: Rust - Size: 51.8 KB - Last synced at: 29 days ago - Pushed at: over 2 years ago - Stars: 43 - Forks: 2
ruester/midnightdup
MidnightDup - Duplicate File Finder
Language: Perl - Size: 45.2 MB - Last synced at: 4 days ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0
Firefox-1998/PhotoVideoOrganizer
How many of you have thousands of photos scattered everywhere (cloud, folders, external hard drives, USB sticks, etc. etc.)?
Language: C# - Size: 460 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0
arasgungore/job-posting-duplicate-detection
A project aiming to leverage text embeddings and Milvus, a high-performance vector search engine, to detect duplicate job postings.
Language: Python - Size: 289 KB - Last synced at: about 2 months ago - Pushed at: over 1 year ago - Stars: 5 - Forks: 0
Prajjwol09/Data-Cleaning-Project
This project is dedicated to cleaning, standardizing a dataset, dealing with null values from a CSV file named "layoffs" using MySQL, with MySQL Workbench as the workspace environment. The goal is to prepare the data for analysis.
Size: 62.5 KB - Last synced at: 2 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0
innovatrics/dedubcheck
dedubcheck - De-Duplicate Dependency Checker for Node.js monorepos
Language: JavaScript - Size: 29.3 KB - Last synced at: about 1 month ago - Pushed at: almost 6 years ago - Stars: 6 - Forks: 1
sean-public/python-hashes
Interesting (non-cryptographic) hashes implemented in pure Python.
Language: Python - Size: 29.3 KB - Last synced at: 9 months ago - Pushed at: over 3 years ago - Stars: 240 - Forks: 43
cemahseri/Duplica
A very fast duplicate file finder.
Language: C# - Size: 25.4 KB - Last synced at: 10 months ago - Pushed at: 10 months ago - Stars: 14 - Forks: 4
qwertz19281/dupion
Duplicate file/folder finder, can also scan in archives, HDD optimized
Language: Rust - Size: 365 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 3 - Forks: 1
DragonOfMath/dupe-images
Node.js package for finding and removing duplicate image files with extreme precision
Language: JavaScript - Size: 17.6 KB - Last synced at: 11 months ago - Pushed at: almost 7 years ago - Stars: 11 - Forks: 3
cwkingjr/find_duplicate_files
Find duplicate files on your system using inclusion and exclusion folder lists.
Language: Go - Size: 4.88 KB - Last synced at: almost 2 years ago - Pushed at: over 4 years ago - Stars: 0 - Forks: 0
bob3000/dupcrawler
finds duplicate files
Language: Go - Size: 14.6 KB - Last synced at: about 1 year ago - Pushed at: almost 6 years ago - Stars: 0 - Forks: 0
Sina1218/Text
text duplicate edit
Size: 2.93 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0
TheArkive/ConstScanner
C/C++ Constant Scanner - includes lists of constants from groups of headers. Check the docs for the repo that lists several Win10 APIs.
Language: AutoHotkey - Size: 22.5 MB - Last synced at: 10 months ago - Pushed at: over 3 years ago - Stars: 2 - Forks: 1
rustomax/ndf
Duplicate file finder written in Nim
Language: Nim - Size: 22.5 KB - Last synced at: 2 days ago - Pushed at: over 4 years ago - Stars: 20 - Forks: 0
LeonSteinbach/BitwardenTools
This repository is a collection of tools for the usage of Bitwarden
Language: Python - Size: 8.79 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
davidmalasek/awfulsorter
Awful Sorter is a tool that makes it easier to sort files based on their file types and extensions.
Language: Python - Size: 8.79 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
jvirkki/dupd
CLI utility to find duplicate files
Language: C - Size: 1.9 MB - Last synced at: about 1 year ago - Pushed at: almost 2 years ago - Stars: 109 - Forks: 16
deric/es-dedupe
Tool for removing duplicate documents from Elasticsearch
Language: Python - Size: 130 KB - Last synced at: 22 days ago - Pushed at: over 1 year ago - Stars: 54 - Forks: 22
QuietWindUponTheMoor/Quiets-Duplicate-Manager
Quiet's Duplicate Manager is an Electron.js-based desktop application that is currently in the works. It will offer a range of basic features like including any/all files that are duplicates, giving the option to choose whether to delete or archive duplicates, etc. As more features roll out, more will be added here.
Language: JavaScript - Size: 93.8 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0
matteodelabre/mongoose-beautiful-unique-validation
Plugin for Mongoose that turns duplicate errors into regular Mongoose validation errors
Language: JavaScript - Size: 194 KB - Last synced at: 7 days ago - Pushed at: almost 3 years ago - Stars: 117 - Forks: 38
pouyakary/dup
a tiny and fast command line utility to find the duplicate files within a directory
Language: Go - Size: 16.6 KB - Last synced at: about 17 hours ago - Pushed at: over 1 year ago - Stars: 2 - Forks: 0
Dmitriy-Vas/go-file-copies
A Go program to get duplicates from specified paths.
Language: Go - Size: 10.7 KB - Last synced at: about 1 year ago - Pushed at: over 5 years ago - Stars: 7 - Forks: 0
lachhabw/Duplicate-Images-Remover
Python tool for finding and removing duplicate images
Language: Python - Size: 8.79 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
ajmalshahabudeen/Bitwarden-Duplicate-remover
When Importing multiple CSV files Bitwarden creates Duplicate Entries. So this Python script will remove duplicate entries and keep ONE.
Language: Python - Size: 2.93 KB - Last synced at: about 1 year ago - Pushed at: almost 2 years ago - Stars: 3 - Forks: 2
fiddyschmitt/udp_dedupe
Deduplicate UDP datagrams
Language: C# - Size: 36.1 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
hansalemaos/dropduplicatesplanb
Drops duplicates in DataFrames with tedious dtypes
Language: Python - Size: 23.4 KB - Last synced at: about 1 month ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
quattroformaggi/Mindmap-mini-programs
A mindmap & programs so small they don't require their own repository.
Language: Go - Size: 21.5 KB - Last synced at: over 1 year ago - Pushed at: almost 3 years ago - Stars: 0 - Forks: 0
ArtTorres/FileMatch
Find duplicate files in directories.
Language: C# - Size: 20.5 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
carlbeech/fast-duplicate-finder
A python program to locate duplicate files - and do it fast
Language: Python - Size: 74.4 MB - Last synced at: about 1 month ago - Pushed at: almost 4 years ago - Stars: 9 - Forks: 3
LibreTranslate/RemoveDup
Remove duplicates from parallel corpora
Language: Python - Size: 835 KB - Last synced at: 19 days ago - Pushed at: over 1 year ago - Stars: 5 - Forks: 0
ikeratzakis/duplicate-detection
Algorithms for duplicate document and question detection/classification, implemented as part of a project
Language: Python - Size: 21.5 KB - Last synced at: over 1 year ago - Pushed at: about 3 years ago - Stars: 0 - Forks: 0
strong-roots-capital/remove-duplicates-from-sorted 📦
Remove duplicates from a sorted list
Language: TypeScript - Size: 8.79 KB - Last synced at: 24 days ago - Pushed at: almost 6 years ago - Stars: 0 - Forks: 0
GreenComfyTea/duplicate-emote-check-tool
Check if you have duplicate emotes across FrankerFaceZ, BetterTTV and 7TV for Twitch.
Language: JavaScript - Size: 414 KB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 3 - Forks: 0
hansalemaos/arrayhascher
Fast hash in 2D Arrays (Numpy/Pandas/lists/tuples)
Language: C - Size: 101 KB - Last synced at: about 1 month ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0
pcraciunoiu/AndroidSMSBackupRestoreCleaner Fork of NumbGnat/AndroidSMSBackupRestoreCleaner
This cleans up duplicate SMS entries in a backup created by SMS Backup & Restore Android app.
Language: Python - Size: 4.78 MB - Last synced at: about 1 year ago - Pushed at: about 2 years ago - Stars: 5 - Forks: 2
davidefiocco/dockerized-elasticsearch-duplicate-finder
Attempt to use MinHash to find duplicates in an Elasticsearch index
Language: Python - Size: 11.7 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 2 - Forks: 0
codecliff/FdupesAnalyzer
A script to analyze output of fdupes linux utility to find level of overlap between directories. Written in R
Language: R - Size: 237 KB - Last synced at: 3 months ago - Pushed at: over 5 years ago - Stars: 3 - Forks: 1
TheProv1/Java-Codes
Java Codes
Language: Java - Size: 64.5 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
davdiv/hashfolder
Simple command line tool that can create/update an sqlite database that contains the hash (by default SHA256) of all files inside a specified root folder.
Language: TypeScript - Size: 80.1 KB - Last synced at: 28 days ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0
AlexZasorin/delete-duplicates.py
Python script to find, filter, and delete duplicate files. Work in progress.
Language: Python - Size: 3.91 KB - Last synced at: over 1 year ago - Pushed at: almost 3 years ago - Stars: 0 - Forks: 0
lmammino/indexed-string-variation
Experimental JavaScript module to generate all possible variations of strings over an alphabet using an n-ary virtual tree
Language: JavaScript - Size: 52.7 KB - Last synced at: 9 days ago - Pushed at: over 7 years ago - Stars: 18 - Forks: 4
tutts/react-single-image 📦
Centralise duplicate images in your React app, while maintaining a modular file system 🖼
Language: JavaScript - Size: 233 KB - Last synced at: 21 days ago - Pushed at: almost 7 years ago - Stars: 1 - Forks: 0
jonas054/dupfind
Duplication finder for source code and other text files
Language: C++ - Size: 202 KB - Last synced at: about 2 months ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 2
hansalemaos/duplicateindexer
Find duplicates in multiple lists and return their indices and values.
Language: Python - Size: 3.91 KB - Last synced at: 16 days ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
hansalemaos/stridesduplicatefinder
Calculate overlapping values between two arrays and return the results as a DataFrame
Language: Python - Size: 24.4 KB - Last synced at: 10 days ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
AndreyKlychnikov/deduplicate-elasticsearch
Remove duplicate documents from Elasticsearch
Language: Python - Size: 2.93 KB - Last synced at: about 1 year ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 0
