I have encountered a problem with zlib at work. I have to parse through
huge gzipped text files (500000+ lines). However, it seems that zlib
only reads the first 66000 or so lines.
Initially I used the code:
Zlib::GZipReader.open(file) do |f|
# process text
puts f.readlines.size
end
66196
However:
IO.open("zcat #{file}",'r') do |io|
puts io.readlines.size
end
513222
I went through about 2 hours of confusion and struggling not believing
that I couldn't find what I expected using the code, but grepping
returned results. Then I somehow stumbled upon this unexpected cause of
my problems.
Is this a known issue?
-Szymon