github.com/jRimbault/yadf ↗

About jRimbault/yadf

YADF is a command-line duplicate file finder written in Rust, designed to identify and report duplicate files within directory trees. It outputs results in a format compatible with the standard fdupes tool and provides multiple output formatting options. The tool automatically searches through all subdirectories and includes empty files in its analysis.

The project uses a two-step algorithm rather than the more common three-step approach. Instead of grouping by file size first, it uses hashing to group files by their initial bytes and then verifies duplicates by hashing entire file contents. The implementation leverages the standard library's BTreeMap for cache-aware operations and uses the ignore crate's parallel walker alongside rayon for concurrent processing of both grouping steps. The author explicitly notes that this approach performs better on SSDs than the traditional three-step method.

According to benchmarks on the developer's home directory containing over 700,000 files and 39 GB of data, YADF significantly outperforms other duplicate finders like fclones and jdupes when the filesystem cache is warm, completing scans in around 3 seconds compared to 7-18 seconds for competitors. However, the project acknowledges its poor performance on traditional hard drives and recommends fclones as a more feature-rich alternative with better hardware heuristics for general use.