Rubyzip - `dup': can't dup NilClass (TypeError)

ruby-talk
1

Hello. I've trying to figure out rubyzip. Here's the code I had:

require 'rubygems'
require 'zip/zip'

zf = Zip::ZipFile.open('616910.zip')

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup': can't dup NilClass (TypeError)
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`map'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1399:in
`initialize'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`new'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open'
        from zip.rb:5

I'm pretty sure the zip file is there and is not empty. What might be
causing it? I googled the problem, but couldn't find a definitive
answer.

Thank you,
Luka

···

--
Posted via http://www.ruby-forum.com/.

2

Luka Stolyarov wrote:

require 'rubygems'
require 'zip/zip'

zf = Zip::ZipFile.open('616910.zip')

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup': can't dup NilClass (TypeError)

Works fine for me with rubyzip-0.9.1 + "ruby 1.8.6 (2007-09-24
patchlevel 111) [i486-linux]", and I just upgraded to rubyzip-0.9.4 with
the same results.

What platform are you on?

The offending code is here:

    # deep clone
    def dup
      newZipEntrySet = ZipEntrySet.new(@entrySet.values.map { |e| e.dup
})
    end

which suggests to me that the zipfile is corrupt or an unsupported
format, since @entrySet must contain a {value=>nil} pair.

Try modifying this code (line 657):

    def ZipEntry.read_c_dir_entry(io) #:nodoc:all
      entry = new(io.path)
      entry.read_c_dir_entry(io)
      return entry
    rescue ZipError
      return nil
    end

For example, comment out the rescue ZipError // return nil pair.

It seems that this error handling is bad. Either an exception should be
raised here, or the bad entry should be skipped (not saved as a nil
value in @entrySet which causes the dup error you saw)

Regards,

Brian.

···

--
Posted via http://www.ruby-forum.com/\.

3

I'll give it a try, thank you!

···

--
Posted via http://www.ruby-forum.com/.

4

Luka Stolyarov wrote:

I'll give it a try, thank you!

Just to be clear: I'd expect the program to crash still, but this time
at an earlier point which will give a much more useful error about what
went wrong when parsing the zip directory entry.

···

--
Posted via http://www.ruby-forum.com/\.

5

The 1.57 zipfile on this page fails for me:
http://www.vim.org/scripts/script.php?script_id=2441

The Unix unzip command unzips it just fine.

Here's the direct link to the failing zip:
http://www.vim.org/scripts/download_script.php?src_id=11978

To test it in irb:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src_id=11978',
'rb') { |f| Zip::ZipFile.open(f) }

this dies with:

TypeError: can't dup NilClass
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `dup'
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `block in dup'
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `map'
        ...etc

the zipfiles before 1.57 are fine. For instance:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src_id=11524',
'rb') { |f| Zip::ZipFile.open(f) }

works just fine.

The nil entry is actually being added by read_central_directory_entries
around line 1250 (I added the raise):

    @entrySet = ZipEntrySet.new
    @size.times {
        @entrySet << ZipEntry.read_c_dir_entry(io) || raise("nil
entry!")
    }

For some reason, read_c_dir_entry is returning nil. I haven't tried to
figure out why since I'm not familiar with the internals of a zipfile.

···

--
Posted via http://www.ruby-forum.com/.

6

Scott Bronson wrote:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src_id=11978&#39;,
'rb') { |f| Zip::ZipFile.open(f) }

For me (with ruby 1.8.7 and rubyzip-0.9.4) that dies with "cannot
convert Tempfile to String", but if I download the zip locally and then
do Zip::ZipFile.open("ert.zip") then I get the same error as you.

For some reason, read_c_dir_entry is returning nil. I haven't tried to
figure out why since I'm not familiar with the internals of a zipfile.

I suggest you apply the following patch:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:34:19.673351372 +0100
@@ -658,8 +658,8 @@
       entry = new(io.path)
       entry.read_c_dir_entry(io)
       return entry
- rescue ZipError
- return nil
+ #rescue ZipError
+ # return nil
     end

     def file_stat(path) # :nodoc:

Then you get a more helpful error:

/var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:645:in
`read_c_dir_entry': unknown file type 00 (Zip::ZipInternalError)
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:659:in
`read_c_dir_entry'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1255:in
`read_central_directory_entries'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
`times'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
`read_central_directory_entries'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1261:in
`read_from_stream'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`initialize'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`open'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`initialize'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in `new'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open'

To be honest, I have no idea why rubyzip is catching these errors and
returning nil, instead of letting them propagate upwards. All it does is
make a more obscure error later on ("cannot dup nil")

With another patch:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:38:07.475854345 +0100
@@ -642,7 +642,7 @@
         when 012
           @ftype = :symlink
         else
- raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)}"
+ raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)} for entry #{@name.inspect}"
         end
       else
         if name_is_directory?

you can see that the affected entry is .DS_Store. That unpacks as a
regular file from unix unzip.

I don't know why this entry happens to have a type of 0, but you can
allow it like this:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
         case (@externalFileAttributes >> 28)
         when 04
           @ftype = :directory
- when 010
+ when 010, 00
           @ftype = :file
         when 012
           @ftype = :symlink

Anyway, since you have a good test case for this, I suggest you post it
to the rubyzip mailing list or tracker, if there is one.

Regards,

Brian.

···

--
Posted via http://www.ruby-forum.com/\.

7

Brian Candler wrote:

I don't know why this entry happens to have a type of 0, but you can
allow it like this:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
         case (@externalFileAttributes >> 28)
         when 04
           @ftype = :directory
- when 010
+ when 010, 00
           @ftype = :file
         when 012
           @ftype = :symlink

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running "zipinfo -v" on the zip-file
and notice that it prints a '?' as the file type in the external file
attributes.

rubyzip should not throw if the value is unknown - it should simply
translate it to :unknown.

My familiarity with both ruby and rubyzip are rusting, but perhaps
something like this:

diff -u -r1.47 zip.rb
--- lib/zip/zip.rb 14 May 2010 20:19:08 -0000 1.47
+++ lib/zip/zip.rb 10 Sep 2010 19:51:32 -0000
@@ -643,7 +643,7 @@
         when 012
           @ftype = :symlink
         else
- raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)}"
+ @ftype = :unknown
         end
       else
         if name_is_directory?
@@ -709,11 +709,11 @@
         when :symlink
           ft = 012
           @unix_perms ||= 0755
- else
- raise ZipInternalError, "unknown file type #{self.inspect}"
         end

- @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) <<
16
+ if (!ft.nil?)
+ @externalFileAttributes = (ft << 12 | (@unix_perms & 07777))
<< 16
+ end
       end

       io <<

The problem should also be reported to the author of the tool generating
the zip archives.

Regards,

Thomas

···

--
Posted via http://www.ruby-forum.com/\.

8

Thomas Sondergaard wrote:

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running "zipinfo -v" on the zip-file
and notice that it prints a '?' as the file type in the external file
attributes.

Very strange. My zipinfo -v doesn't show anything amiss. I'm using:

   ZipInfo 3.00 of 20 April 2009, by Greg Roelofs and the Info-ZIP
group.

The problem should also be reported to the author of the tool generating
the zip archives.

Whatever tool is generating them, it's already in widespread use. :frowning:
For instance, most zipfiles in the following packages are valid, but
rubyzip can't handle them.

http://www.vim.org/scripts/script.php?script_id=3114
http://www.vim.org/scripts/script.php?script_id=3123
http://www.vim.org/scripts/script.php?script_id=3148
http://www.vim.org/scripts/script.php?script_id=3150
http://www.vim.org/scripts/script.php?script_id=3169

Thanks Thomas. For now I've worked around it by shelling out to the
unzip command. If I have time I'll try your patch next week.

    - Scott

···

--
Posted via http://www.ruby-forum.com/\.

9

Scott Bronson wrote:

Whatever tool is generating them, it's already in widespread use. :frowning:

That might not actually be true... Most failing zipfiles were upped by
the same author. I'll ask Peter Odding what he used to zip them.

    - Scott

···

--
Posted via http://www.ruby-forum.com/\.

10

Hi Scott,

Whatever tool is generating them, it's already in widespread use. :frowning:

That might not actually be true... Most failing zipfiles were upped by
the same author. I'll ask Peter Odding what he used to zip them.

I publish my Vim plug-ins with a Python script that uses the zipfile [1]
module to generate ZIP files. All of the archives uploaded in the last
few months were generated on Ubuntu 9.10 (Karmic) with Python 2.6,
although I just yesterday upgraded my laptop to Ubuntu 10.4 (Lucid) so I
can't check the exact version. The ZIP file generation comes down to the
following code:

  from zipfile import ZipFile, ZIP_DEFLATED
  archive = ZipFile('easytags.zip', 'w', ZIP_DEFLATED)
  for filename in ['plugin/easytags.vim', 'autoload/easytags.vim', ...]:
    # This is where "git cat-file" is used, see below.
    with open(filename) as handle:
      archive.writestr(filename, handle.read())
  archive.close()

One peculiarity about the Python code is that it uses ZipFile.writestr()
[2] instead of ZipFile.write() [3] so that it will only package
committed versions of my scripts (using "git cat-file -p
HEAD:some/file").

Let me know if you need any more information. If it helps I might be
able to resurrect my old environment in a virtual machine, but I haven't
tried yet.

- Peter Odding

[1] zipfile — Work with ZIP archives — Python 3.12.1 documentation
[2] zipfile — Work with ZIP archives — Python 3.12.1 documentation
[3] zipfile — Work with ZIP archives — Python 3.12.1 documentation

···

--
Posted via http://www.ruby-forum.com/\.

11

I've filed the following issue against the python ZipFile module:

http://bugs.python.org/issue9835

···

--
Posted via http://www.ruby-forum.com/.