Projects

ZFS

ZFS is a new kind of file system that provides simple administration, transactional semantics, end-to-end data integrity, and immense scalability. ZFS is not an incremental improvement to existing technology; it is a fundamentally new approach to data management. We've blown away 20 years of ... [More] obsolete assumptions, eliminated complexity at the source, and created a storage system that's actually a pleasure to use. [Less]

FSlint

FSlint is a toolkit to find various forms of lint on a filesystem. At the moment it reports duplicate files, bad symbolic links, troublesome file names, empty directories, non stripped executables, temporary files, duplicate/conflicting (binary) names, and unused ext2 directory blocks. The package contains both a GTK GUI and a CLI interface.

memventi

Memventi is a Venti daemon. It speaks the same Venti protocol as the real Venti in Plan 9 from Bell Labs. It is a storage server that stores data blocks up to 56KB using its SHA-1 hash (called its score) to address it. It keeps a mapping of score to disk location in memory (in a memory-efficient ... [More] manner). Blocks written cannot be removed, and blocks are only written once. Memventi writes new blocks to an append-only file, thereby making file corruption due to bugs practically impossible. [Less]

UltraKSM

An improvement upon Linux's memory merging support enabling transparent full system scan at ultra speed.

cyphertite

Cyphertite is a tar-like secure remote archiver. It deduplicates, compresses, and encrypts data prior to transmission, providing total privacy while reducing unnecessary wire traffic. It seamlessly supports IPv6 and IPv4 on a variety of platforms.

ventisrv

Ventisrv is a venti daemon for inferno, written in limbo. It has an in-memory index. Vcache is a (in-memory) venti cache. This package also contains simple tools to write to/read from a venti server. This code is the partial result of a (successful) google summer of code 2007 project, for the ... [More] project "plan 9/inferno", subproject "ventivac". [Less]

Duke (Dupe Killer)

Duke is a fast record linkage and deduplication engine written in Java. It provides both an API and a command-line interface, and supports incremental processing. Duke is based on Lucene.

hadu

HADU is an acronnym for "I Hate Duplicates". It is intended to find and delete duplicate files. This will free space in your hard disk. It is made by inserting the "files" in a B+ tree and searching for duplicates following some criteria.

cocoagaps

GAPS is an image sorting and viewing application written to deal with enormous folders full of images. It can easily handle hundreds of thousands of images per folder and continue to operate quickly and stably. GAPS' DuperFinder feature can scan all of your image folders for duplicates, shows ... [More] you thumbnails of the dupes as confirmation, and easily removes the extra copies to save disk space. With GAPS' image sorting features, you use your keyboard to quickly sort images from an "Incoming" folder into multiple folders of your choosing. You can sort images one at a time or in batches by using your mouse to select multiple images (shift-click). As you sort each image, the file is moved and the next image in your incoming folder is displayed automatically. If you make a mistake, GAPS' Undo feature allows you to go back and change your mind about deleting or categorizing an image. GAPS works best when you use the keys on the numeric keypad to sort, but you can assign any and all keys on your keyboard to perform any sorting action you like. When you don't feel like sorting images, GAPS' slide show (including full screen) displays all the images in your collection either randomly or in order. GAPS has an auto-update feature built in, so once you download and install it from this page, you should be able to get updates from within the program. [Less]

arrow-backup

Arrow is a backup system that combines hashing, error-correction, and rsync-like searching to provide a versioned, deduplicating, verifiable, and safe data backup system. The core of Arrow was done as a part of Casey Marshall's Master's project in computer science at the University of ... [More] California, Santa Cruz. Development of this version will be released under the GNU General Public License, v3. You can get a MIT licensed version (the original one submitted as the final project) at http://git.metastatic.org/, or clone the git repository from http://git.metastatic.org/readonly/arrow.git/. [Less]