I have a got a few folders which may have same files under different
names.
Is there any way I can find which these files are using ruby ?
The files with text (*.doc, *.txt ...) should be pretty easy to check
but what about pdf files, exe files etc ?
I wondering if there is some sort of diff module that can do this.
--
Posted via http://www.ruby-forum.com/.
There's File.compare.
Depending on how many comparisons you're going to do, it might be a
good idea to precompute checksums and compare the checksums.
On Mon, Mar 7, 2011 at 11:48 AM, New C. <coding25@yahoo.com> wrote:
I have a got a few folders which may have same files under different
names.
Is there any way I can find which these files are using ruby ?The files with text (*.doc, *.txt ...) should be pretty easy to check
but what about pdf files, exe files etc ?I wondering if there is some sort of diff module that can do this.
New C. wrote in post #985925:
I have a got a few folders which may have same files under different
names.
Is there any way I can find which these files are using ruby ?
Here is a little ruby script I use for finding and/or deleting duplicate
image and video files downloaded from my camera - it will work for any
sort of file.
#!/usr/bin/ruby -w
require 'digest/sha1'
if ARGV[0] == "-d"
do_delete = true
ARGV.shift
end
seen = {}
dirs = ARGV.empty? ? ["#{ENV["HOME"]}/Pictures"] : ARGV
dirs.each do |dir|
Dir["#{dir}/**/*"].sort.each do |fn|
next if File.directory?(fn)
hash = Digest::SHA1.file(fn).hexdigest
if seen[hash]
puts "#{fn} is dupe of #{seen[hash]}"
if do_delete
File.delete(fn)
puts "DELETED"
end
else
seen[hash] = fn
end
end
end
--
Posted via http://www.ruby-forum.com/\.
It looks pretty obfuscated to my eyes, but each to his own.
dirs.each do |dir|
Dir["#{dir}/**/*"].sort.each do |fn|
next if File.directory?(fn)
hash = Digest::SHA1.file(fn).hexdigest
if seen.fetch(hash) { seen[hash]=fn; false }
puts "#{fn} is dupe of #{seen[hash]}"
if do_delete
File.delete(fn)
puts "DELETED"
end
end
end
end
--
Posted via http://www.ruby-forum.com/.
+1
On 03/07/2011 06:09 AM, Xavier Noria wrote:
Depending on how many comparisons you're going to do, it might be a
good idea to precompute checksums and compare the checksums.
I'm interested in any (Ruby) solutions (actual or ideas) for this, as
I have needed to do it in the past, and want to do something similar
in the very near future.
For comparing directories where the file names might have changed what
I've done in the past is to first match on file name, then for the
unmatching files in each directory see if there are any matches on
file size, and for those matches either make a direct File.compare (if
only two files match on a size) or compute checksums and use those to
exclude definitely unmatching files, and then use File.compare on what
(if anything) remains matching for that file size and checksum.
I assume something similar would work for finding duplicates in
general, not just comparing directories? (If there are likely to be
many matches on file size, then presumably one might as well compute
checksums for all files?)
On Mon, Mar 7, 2011 at 11:09 AM, Xavier Noria <fxn@hashref.com> wrote:
On Mon, Mar 7, 2011 at 11:48 AM, New C. <coding25@yahoo.com> wrote:
I have a got a few folders which may have same files under different
names. Is there any way I can find which these files are using ruby ?
...
I wondering if there is some sort of diff module that can do this.There's File.compare.
Depending on how many comparisons you're going to do, it might be a
good idea to precompute checksums and compare the checksums.
For this idiom Hash#fetch can be used nicely:
irb(main):008:0> h={};
10.times {|i|
puts i
h.fetch(i % 3) {|x| printf "first %p\n", i; h=true; nil} and
printf "duplicate %p\n", i}
0
first 0
1
first 1
2
first 2
3
duplicate 3
4
duplicate 4
5
duplicate 5
6
duplicate 6
7
duplicate 7
8
duplicate 8
9
duplicate 9
=> 10
irb(main):009:0>
... for arbitrary values of "nice". 
Kind regards
robert
On Tue, Mar 8, 2011 at 9:48 AM, Brian Candler <b.candler@pobox.com> wrote:
New C. wrote in post #985925:
I have a got a few folders which may have same files under different
names.
Is there any way I can find which these files are using ruby ?Here is a little ruby script I use for finding and/or deleting duplicate
image and video files downloaded from my camera - it will work for any
sort of file.#!/usr/bin/ruby -w
require 'digest/sha1'
if ARGV[0] == "-d"
do_delete = true
ARGV.shift
endseen = {}
dirs = ARGV.empty? ? ["#{ENV["HOME"]}/Pictures"] : ARGVdirs.each do |dir|
Dir["#{dir}/**/*"].sort.each do |fn|
next if File.directory?(fn)
hash = Digest::SHA1.file(fn).hexdigest
if seen[hash]
puts "#{fn} is dupe of #{seen[hash]}"
if do_delete
File.delete(fn)
puts "DELETED"
end
else
seen[hash] = fn
end
end
end
--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
I have played with this as an exercise. The idea is to filter
candidates iteratively applying different criteria, from cheap to
expensive, until you arrive at the solution.
It is just a proof of concept in pseudocode, I wrote it off the top of
my head, it does not even run:
The code above assumes a generic scenario with m-n possible
duplicates, if a particular situation has details that can speed up
the process they should be taken into account of course.
On Mon, Mar 7, 2011 at 6:38 PM, Colin Bartlett <colinb2r@googlemail.com> wrote:
I'm interested in any (Ruby) solutions (actual or ideas) for this, as
I have needed to do it in the past, and want to do something similar
in the very near future.For comparing directories where the file names might have changed what
I've done in the past is to first match on file name, then for the
unmatching files in each directory see if there are any matches on
file size, and for those matches either make a direct File.compare (if
only two files match on a size) or compute checksums and use those to
exclude definitely unmatching files, and then use File.compare on what
(if anything) remains matching for that file size and checksum.