File size revisit

ruby-talk
1

Hi, folks

the subject has been reported before, basically, ruby File.size() won't
work properly on windows when file size exceed certain limit (2GB
maybe). One workaround is to require win32/file. Yes, this will get you
correct file, however, this will break the code (in my case, the find
module) ... the previous recursive scan of file system stop
functioning. The question is, is there a more "generic" or "elegant"
way of getting correct file size on both *nix and windows?

thanks

-Oliver

2

python152@gmail.com wrote:

Hi, folks

the subject has been reported before, basically, ruby File.size() won't
work properly on windows when file size exceed certain limit (2GB
maybe). One workaround is to require win32/file. Yes, this will get you
correct file, however, this will break the code (in my case, the find
module) ... the previous recursive scan of file system stop
functioning.

Whoa, what? Can you please elaborate on how win32-file causes problems
with your filesystem scan?

Thanks,

Dan

3

Hi,

From: python152@gmail.com
Reply-To: ruby-talk@ruby-lang.org
To: ruby-talk@ruby-lang.org (ruby-talk ML)
Subject: file size revisit
Date: Thu, 2 Nov 2006 10:40:06 +0900

Hi, folks

the subject has been reported before, basically, ruby File.size() won't
work properly on windows when file size exceed certain limit (2GB
maybe). One workaround is to require win32/file. Yes, this will get you
correct file, however, this will break the code (in my case, the find
module) ... the previous recursive scan of file system stop
functioning. The question is, is there a more "generic" or "elegant"
way of getting correct file size on both *nix and windows?

thanks

-Oliver

As the another workaround in windows, try this:

require 'Win32API'
def File.size(name)
  buf = "\0"*36
  Win32API.new('kernel32','GetFileAttributesEx','PLP','L').call(name,0,buf)
  (buf[28,4].unpack("L").first << 32) + buf[32,4].unpack("L").first
end

Regards,

Park Heesob

···

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

4

please see the code segment, once I put in "require win32/file", the
scanning process is working properly anymore.

require 'find'
require 'win32/file'
...

# the following code recursively go into each sub directory and read
file information
# options save certain options such as directory to scan and if run
verbosely.

Find.find(options[:dir]) do |path|
    if File.file?(path)
        if options[:verbose]
            puts "Scaning #{path}"
        end
        size = File.size?(path)
        # ... do some work on the file
        end
    end
end

···

On Nov 1, 9:53 pm, "Daniel Berger" <djber...@gmail.com> wrote:

python...@gmail.com wrote:
> Hi, folks

> the subject has been reported before, basically, ruby File.size() won't
> work properly on windows when file size exceed certain limit (2GB
> maybe). One workaround is to require win32/file. Yes, this will get you
> correct file, however, this will break the code (in my case, the find
> module) ... the previous recursive scan of file system stop
> functioning.Whoa, what? Can you please elaborate on how win32-file causes problems
with your filesystem scan?

Thanks,

Dan

5

Hi Park -

thanks for the suggestion. With this redefined function, it works fine
on windows now.
why it is not the case with win32/file, and how can I make it work
cross the platform?

Regards,

-Oliver

···

On Nov 2, 1:11 am, "Park Heesob" <phasi...@hotmail.com> wrote:

Hi,

>From: python...@gmail.com
>Reply-To: ruby-t...@ruby-lang.org
>To: ruby-t...@ruby-lang.org (ruby-talk ML)
>Subject: file size revisit
>Date: Thu, 2 Nov 2006 10:40:06 +0900

>Hi, folks

>the subject has been reported before, basically, ruby File.size() won't
>work properly on windows when file size exceed certain limit (2GB
>maybe). One workaround is to require win32/file. Yes, this will get you
>correct file, however, this will break the code (in my case, the find
>module) ... the previous recursive scan of file system stop
>functioning. The question is, is there a more "generic" or "elegant"
>way of getting correct file size on both *nix and windows?

>thanks

>-OliverAs the another workaround in windows, try this:

require 'Win32API'
def File.size(name)
  buf = "\0"*36
  Win32API.new('kernel32','GetFileAttributesEx','PLP','L').call(name,0,buf)
  (buf[28,4].unpack("L").first << 32) + buf[32,4].unpack("L").first
end

Regards,

Park Heesob

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

6

Park Heesob wrote:

<snip>

As the another workaround in windows, try this:

require 'Win32API'
def File.size(name)
  buf = "\0"*36
  Win32API.new('kernel32','GetFileAttributesEx','PLP','L').call(name,0,buf)
  (buf[28,4].unpack("L").first << 32) + buf[32,4].unpack("L").first
end

I don't think the problem is File.size. I suspect it's the "if
File.file?(path)" line failing somewhere it shouldn't, but the OP isn't
clear.

Regards,

Dan

7

correction: should be "not working anymore" ...
Also, I am not sure why these two modules (or other reasons) are not
working together. Maybe you or someone can enlighten me.

thanks

Oliver

···

On Nov 1, 11:09 pm, python...@gmail.com wrote:

please see the code segment, once I put in "require win32/file", the
scanning process is working properly anymore.

require 'find'
require 'win32/file'
...

# the following code recursively go into each sub directory and read
file information
# options save certain options such as directory to scan and if run
verbosely.

Find.find(options[:dir]) do |path|
    if File.file?(path)
        if options[:verbose]
            puts "Scaning #{path}"
        end
        size = File.size?(path)
        # ... do some work on the file
        end
    end
end

On Nov 1, 9:53 pm, "Daniel Berger" <djber...@gmail.com> wrote:

> python...@gmail.com wrote:
> > Hi, folks

> > the subject has been reported before, basically, ruby File.size() won't
> > work properly on windows when file size exceed certain limit (2GB
> > maybe). One workaround is to require win32/file. Yes, this will get you
> > correct file, however, this will break the code (in my case, the find
> > module) ... the previous recursive scan of file system stop
> > functioning.Whoa, what? Can you please elaborate on how win32-file causes problems
> with your filesystem scan?

> Thanks,

> Dan

8

python152@gmail.com wrote:

please see the code segment, once I put in "require win32/file", the
scanning process is working properly anymore.

require 'find'
require 'win32/file'
...

# the following code recursively go into each sub directory and read
file information
# options save certain options such as directory to scan and if run
verbosely.

Find.find(options[:dir]) do |path|
    if File.file?(path)
        if options[:verbose]
            puts "Scaning #{path}"
        end
        size = File.size?(path)
        # ... do some work on the file
        end
    end
end

Please define "not working any more". What isn't working exactly? Is
there an error message? Or is File.file?(path) not working? If so,
what does "path" look like?

I'm afraid I need more information in order to help you.

Regards,

Dan

9

ok, usually, I ran the program and supply directory name "." as
current, that doesn't generate any warning message. However, if I
supply a full path, say "C;\temp", these are the error messages on the
console. This happen as soon as I put in "require win32/file"

c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:35:
warning: method rede
ned; discarding old initialize
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:113:
warning: method red
ined; discarding old <=>
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:119:
warning: method red
ined; discarding old blockdev?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:125:
warning: method red
ined; discarding old chardev?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:132:
warning: method red
ined; discarding old executable?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:136:
warning: discarding
ld executable_real?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:141:
warning: method red
ined; discarding old file?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:148:
warning: method red
ined; discarding old ftype
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:168:
warning: method red
ined; discarding old grpowned?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:173:
warning: method red
ined; discarding old owned?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:179:
warning: method red
ined; discarding old pipe?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:183:
warning: discarding
ld socket?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:187:
warning: method red
ined; discarding old readable?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:193:
warning: method red
ined; discarding old readable_real?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:199:
warning: method red
ined; discarding old setgid?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:205:
warning: method red
ined; discarding old setuid?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:212:
warning: method red
ined; discarding old size?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:218:
warning: method red
ined; discarding old sticky?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:224:
warning: method red
ined; discarding old symlink?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:230:
warning: method red
ined; discarding old writable?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:236:
warning: method red
ined; discarding old writable_real?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:242:
warning: method red
ined; discarding old zero?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:262:
warning: method red
ined; discarding old directory?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:335:
warning: method red
ined; discarding old atime
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:341:
warning: method red
ined; discarding old blksize
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:352:
warning: method red
ined; discarding old blocks
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:359:
warning: method red
ined; discarding old ctime
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:366:
warning: method red
ined; discarding old dev
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:376:
warning: method red
ined; discarding old gid
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:382:
warning: method red
ined; discarding old ino
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:392:
warning: method red
ined; discarding old mode
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:398:
warning: method red
ined; discarding old mtime
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:404:
warning: method red
ined; discarding old rdev
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:410:
warning: method red
ined; discarding old nlink
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:416:
warning: method red
ined; discarding old size
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:422:
warning: method red
ined; discarding old uid
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:428:
warning: method red
ined; discarding old inspect
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:451:
warning: method red
ined; discarding old pretty_print
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-0.5.2-mswin32/lib/win32/file.rb:392:
warning: redefine basename
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-0.5.2-mswin32/lib/win32/file.rb:446:
warning: redefine dirname
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-0.5.2-mswin32/lib/win32/file.rb:503:
warning: redefine split
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-0.5.2-mswin32/lib/win32/file.rb:518:
warning: redefine stat
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-0.5.2-mswin32/lib/win32/file.rb:530:
warning: redefine blockdev?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-0.5.2-mswin32/lib/win32/file.rb:537:
warning: redefine chardev?
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-0.5.2-mswin32/lib/win32/file.rb:546:
warning: redefine size
C:\temp
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:263:
warning: instance v
iable @directory not initialized

···

On Nov 1, 11:29 pm, "Daniel Berger" <djber...@gmail.com> wrote:

python...@gmail.com wrote:
> please see the code segment, once I put in "require win32/file", the
> scanning process is working properly anymore.

> require 'find'
> require 'win32/file'
> ...

> # the following code recursively go into each sub directory and read
> file information
> # options save certain options such as directory to scan and if run
> verbosely.

> Find.find(options[:dir]) do |path|
> if File.file?(path)
> if options[:verbose]
> puts "Scaning #{path}"
> end
> size = File.size?(path)
> # ... do some work on the file
> end
> end
> endPlease define "not working any more". What isn't working exactly? Is
there an error message? Or is File.file?(path) not working? If so,
what does "path" look like?

I'm afraid I need more information in order to help you.

Regards,

Dan

10

Oliver wrote:

ok, usually, I ran the program and supply directory name "." as
current, that doesn't generate any warning message. However, if I
supply a full path, say "C;\temp", these are the error messages on the
console. This happen as soon as I put in "require win32/file"

<snip>

These are warnins not errors. They are harmless - I redefine a bunch
of methods in the File class.

C:\temp
c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:263:
warning: instance v
iable @directory not initialized

This one is strange, since @directory *is* initialized. But, I doubt
it's the source of your problem

None of this helps me, however. You still haven't told me exactly what
line specifically is failing. With or without win32/file, you are
going to get nil for File.size?("C:\\temp") or File.size?("."), since
they are directories. It must be something else.

- Dan

11

Hi Dan -

Find.find(some path) do |path|
puts path
...
end

suppose to loop all the files and sub directories, this is not the case
once import win32/file, the only print out I can get is the very top
level directory I pass in: say, if I run ruby test.rb -d "c:\temp",
then the only print out is "c:\temp", all files in that directory are
ignored. It should be pretty easy to verify. It almost seems like find
module has a dependency on File module, and win32/file is in odd with
it.

This is probably as much as I can find at this point ...

Oliver

···

On Nov 2, 10:50 am, "Daniel Berger" <djber...@gmail.com> wrote:

Oliver wrote:
> ok, usually, I ran the program and supply directory name "." as
> current, that doesn't generate any warning message. However, if I
> supply a full path, say "C;\temp", these are the error messages on the
> console. This happen as soon as I put in "require win32/file"<snip>

These are warnins not errors. They are harmless - I redefine a bunch
of methods in the File class.

> C:\temp
> c:/ruby/lib/ruby/gems/1.8/gems/win32-file-stat-1.2.2-mswin32/lib/win32/file/stat.rb:263:
> warning: instance v
> iable @directory not initializedThis one is strange, since @directory *is* initialized. But, I doubt
it's the source of your problem

None of this helps me, however. You still haven't told me exactly what
line specifically is failing. With or without win32/file, you are
going to get nil for File.size?("C:\\temp") or File.size?("."), since
they are directories. It must be something else.

- Dan

12

Oliver wrote:

Hi Dan -

Find.find(some path) do |path|
puts path
...
end

suppose to loop all the files and sub directories, this is not the case
once import win32/file, the only print out I can get is the very top
level directory I pass in: say, if I run ruby test.rb -d "c:\temp",
then the only print out is "c:\temp", all files in that directory are
ignored. It should be pretty easy to verify. It almost seems like find
module has a dependency on File module, and win32/file is in odd with
it.

Aha! The culprit appears to be lstat. In the find module you'll see
this line:

if File.lstat(file).directory? then

That returns nil when you include win32-file:

irb(main):001:0> dir = "C:\\TMP"
=> "C:\\TMP"
irb(main):002:0> File.lstat(dir)
=> #<File::Stat dev=0x2, ino=0, mode=040755, nlink=1, uid=0, gid=0,
rdev=0x2, size=0, blksize=nil, blocks=nil, atime=Thu
Nov 02 09:56:43 -0700 2006, mtime=Fri Oct 21 09:47:09 -0600 2005,
ctime=Fri Oct 21 09:47:09 -0600 2005>
irb(main):003:0> File.lstat(dir).directory?
=> true
irb(main):004:0> require 'win32/file'
=> true
irb(main):005:0> File.lstat(dir).directory?
=> nil

I didn't reallize lstat was implemented on Windows. I'm guessing it's
an alias, so I'll need to check the source. But, that's definitely a
bug in win32-file (or rather, win32-file-stat).

I'll get this fixed and put out a release asap.

Thanks,

Dan

13

Daniel Berger wrote:

<snip>

I didn't reallize lstat was implemented on Windows. I'm guessing it's
an alias, so I'll need to check the source. But, that's definitely a
bug in win32-file (or rather, win32-file-stat).

I'll get this fixed and put out a release asap.

Ok, fixed in CVS. I'll have a release out tonight. If you want the
fix now, just open up win32/file.rb and add this:

def self.lstat(file)
   File::Stat.new(file)
end

Regards,

Dan

14

Dan -

First of all, thanks for fixing this.

A general comment on this issue is: why in Ruby, the basic operation on
files needs special treatment, and short of being cross-platform? If I
am invoking a exotic feature that is windows only, import a win32
module would make more sense to me.

In Python, the operation is uniform cross the platform
import os.path
os.path.getsize("filename")

maybe many folks probably won't agree this, but I tend to regard the
core File.size() won't work with larger file on windows a bug, though a
minor one.

Best,
Oliver

···

On Nov 2, 12:54 pm, "Daniel Berger" <djber...@gmail.com> wrote:

Daniel Berger wrote:<snip>

> I didn't reallize lstat was implemented on Windows. I'm guessing it's
> an alias, so I'll need to check the source. But, that's definitely a
> bug in win32-file (or rather, win32-file-stat).

> I'll get this fixed and put out a release asap.Ok, fixed in CVS. I'll have a release out tonight. If you want the
fix now, just open up win32/file.rb and add this:

def self.lstat(file)
   File::Stat.new(file)
end

Regards,

Dan

15

Oliver wrote:

Dan -

First of all, thanks for fixing this.

A general comment on this issue is: why in Ruby, the basic operation on
files needs special treatment, and short of being cross-platform? If I
am invoking a exotic feature that is windows only, import a win32
module would make more sense to me.

In Python, the operation is uniform cross the platform
import os.path
os.path.getsize("filename")

maybe many folks probably won't agree this, but I tend to regard the
core File.size() won't work with larger file on windows a bug, though a
minor one.

Actually, we all agree. It's a bug in Ruby. It has been brought up,
more than once. The problem is the use of stat instead of stat64 in the
win32.c file.

If I'm feeling up to it this weekend, I'll submit a patch that fixes
File.size and a few of the other methods.

Regards,

Dan

16

Hi,

From: "Daniel Berger" <djberg96@gmail.com>
Reply-To: ruby-talk@ruby-lang.org
To: ruby-talk@ruby-lang.org (ruby-talk ML)
Subject: Re: file size revisit
Date: Fri, 3 Nov 2006 04:30:14 +0900

Actually, we all agree. It's a bug in Ruby. It has been brought up,
more than once. The problem is the use of stat instead of stat64 in the
win32.c file.

If I'm feeling up to it this weekend, I'll submit a patch that fixes
File.size and a few of the other methods.

Regards,

Dan

The File.size of win32-file still has a bug.
It cannot get correct size when file size is greater than 4Gb(4294967295 byte).

Regards,

Park Heesob

···

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

17

Park Heesob wrote:

Hi,

>From: "Daniel Berger" <djberg96@gmail.com>
>Reply-To: ruby-talk@ruby-lang.org
>To: ruby-talk@ruby-lang.org (ruby-talk ML)
>Subject: Re: file size revisit
>Date: Fri, 3 Nov 2006 04:30:14 +0900
>
>
>
>Actually, we all agree. It's a bug in Ruby. It has been brought up,
>more than once. The problem is the use of stat instead of stat64 in the
>win32.c file.
>
>If I'm feeling up to it this weekend, I'll submit a patch that fixes
>File.size and a few of the other methods.
>
>Regards,
>
>Dan
>
>
The File.size of win32-file still has a bug.
It cannot get correct size when file size is greater than 4Gb(4294967295
byte).

Looks like I used "L" instead of "Q" in the original buffer. Oops.
I'll fix that this weekend.

Thanks,

Dan