Brute force file deduplication with python
This short guide shows how to deduplicate files by brute force with python. Two strategies are used: file hashing (md5) and file name (Levenshtein algorithm).
WARNING: Please, backup your data before deleting any single bit of YOUR DATA! WARNING
Continuing the saga to workout all my digital library, this time I'm showing how I keep my digital library clean from duplicated files.
I know the easy way would be to have clean file metadata, but most of my PDFs come from the web and because of that, I cannot count on structured, clean metadata.
So, in order to workout this situation, from times to times, I run 2 python scripts to cleanup my storage:
Those 2 scripts work by brute force, that is, it runs as a combinatorial explosion doing an 'all against all' comparison.
and I choose those strategies (even knowing the implications) because I noticed many files were duplicated by hash (the same file with different names) and many files were duplicated by name (same or SIMILAR names, usually duplicated articles in different formats, e.g.: article.doc, article.pdf).
It usually takes 1 ~ 2 hours to run both scripts on all my digital library (~236 GB now, ~489 GB before).
Said all that, let's go to the magic. :)
This is the procedure to check the hashed files
1. Execute a recursive find calling md5 (or any other hashing alg.):
2. Call deduplicate-by-hash.py passing dupl.txt to it:
This is a sample output:
3. Delete one or more files as you wish :)
This is the procedure to check the files with equal/similar names:
1. Execute a recursive find calling sort:
2. Add or remove entries at the method cleanupString():
(hope to make a proper configuration file in future)
e.g.:
3. Call deduplicate-by-name.py passing dupl.txt
This is a sample output:
The number '1' in the end means that we have just one character of difference.
4. Delete one or more files as you wish :)
I know the easy way would be to have clean file metadata, but most of my PDFs come from the web and because of that, I cannot count on structured, clean metadata.
So, in order to workout this situation, from times to times, I run 2 python scripts to cleanup my storage:
Those 2 scripts work by brute force, that is, it runs as a combinatorial explosion doing an 'all against all' comparison.
and I choose those strategies (even knowing the implications) because I noticed many files were duplicated by hash (the same file with different names) and many files were duplicated by name (same or SIMILAR names, usually duplicated articles in different formats, e.g.: article.doc, article.pdf).
It usually takes 1 ~ 2 hours to run both scripts on all my digital library (~236 GB now, ~489 GB before).
Said all that, let's go to the magic. :)
This is the procedure to check the hashed files
1. Execute a recursive find calling md5 (or any other hashing alg.):
netto@morpheus ~/ $ find ./ -type f -exec md5sum '{}' \; | sort > dupl.txt
2. Call deduplicate-by-hash.py passing dupl.txt to it:
netto@morpheus ~/ $ deduplicate-by-hash.py dupl.txt
This is a sample output:
...
d5b6d5 documentacao/pdf/sm.pdf
d5b6d5 documentacao/pdf/sm.pdf.1
d5b6d5 documentacao/pdf/s/sm.pdf
equal files: 51568 / 182253
3. Delete one or more files as you wish :)
This is the procedure to check the files with equal/similar names:
1. Execute a recursive find calling sort:
netto@morpheus ~/ $ find ./ -type f -exec md5sum '{}' \; | sort > dupl.txt
2. Add or remove entries at the method cleanupString():
(hope to make a proper configuration file in future)
e.g.:
return entry \
... \
.replace("monography", "") \
.replace("some string here", "")
3. Call deduplicate-by-name.py passing dupl.txt
netto@morpheus ~/ $ deduplicate-by-name.py dupl.txt
This is a sample output:
...
umentacaosm;umentacaosm1;1
umentacaosm1;umentacaosm;1
The number '1' in the end means that we have just one character of difference.
4. Delete one or more files as you wish :)