File.stat(‘file_name’).size returns erroneous file sizes when the
files are over 2 Gigs in size (at which point the number wraps around
and becomes negative). Is this an expected limitation or a bug?
I am using the Pragmatic Programmers 1.8.1-11 build on a windows 2000
machine.
Is there a work around?
Thanks,
Walt
···
Walter Szewelanczyk
IS Director
M.W. Sewall & CO. email : walter@mwsewall.com
259 Front St. Phone : (207) 442-7994 x 128
Bath, ME 04530 Fax : (207) 443-6284
File.stat(‘file_name’).size returns erroneous file sizes when the
files are over 2 Gigs in size (at which point the number wraps around
and becomes negative). Is this an expected limitation or a bug?
I’d like to think it’s a bug. Not sure.
Is there a work around?
What’s your real limit? If it’s 4 gigs, you might try adding 4 gigs
to the (negative) size. E.g., -20,000,000 => roughly 2,020,000,000
bytes.
Of course, 4 gigs really is 2**32 or some such.
If the real limit is more than 4 gigs, I don’t know if there’s any
reasonable workaround.
File.stat(‘file_name’).size returns erroneous file sizes when the
files are over 2 Gigs in size (at which point the number wraps around
and becomes negative). Is this an expected limitation or a bug?
I am using the Pragmatic Programmers 1.8.1-11 build on a windows 2000
machine.
It looks like a bug (that is, it doesn’t happen that way on my box).
I’m guessing that it’s a signed unsigned integer problem. When it
wraps around, does it start out at negative one and go upward fro
increasing file sizes?
I’m not sure about windows, but on unix[1] the file size usually is an
off_t, which is a signed value[2], and traditionally was 32bit (i.e. +/- 2Gig).
Following, some unices (namely solaris comes to mind) offered ‘large file
versions’ of many of their calls, others (e.g. BSDs) just switched to a
signed 64 bit value for off_t. I think the latter is called for in the windows
compat layer (as I assume there is one, implementing stat) to really solve your
problem.
[1] – in stat(2)'s struct stat that is
[2] – to seek(2) backwards
File.stat(‘file_name’).size returns erroneous file sizes when the
files are over 2 Gigs in size (at which point the number wraps around
and becomes negative). Is this an expected limitation or a bug?
I am using the Pragmatic Programmers 1.8.1-11 build on a windows 2000
machine.
> What's your real limit? If it's 4 gigs, you might try adding 4 gigs to
> the (negative) size. E.g., -20,000,000 => roughly 2,020,000,000
> bytes.
>
> Of course, 4 gigs really is 2**32 or some such.
>
> If the real limit is more than 4 gigs, I don't know if there's any
> reasonable workaround.
>
> Hal
>
>
unfortunately, I am looking at some files over 14 Gigs.
Its part of an automated backup system, that tries to keep as many
days worth of data as possible. It looks at the amount of free space
left and then tries to find out the size of the last backup (all in
one directory tree) adds a growth factor and deletes the oldest
directory, repeat until we have the space needed. My problem is
getting the directory tree size, and I tracked it down to
File.stat.size always returning 2 gigs or less.
I may have to revert to ding a dir and parsing that out.
I was hoping for another Ruby Way of find the file size.
Thanks though,
Walt
···
Walter Szewelanczyk
IS Director
M.W. Sewall & CO. email : walter@mwsewall.com
259 Front St. Phone : (207) 442-7994 x 128
Bath, ME 04530 Fax : (207) 443-6284
It looks like a bug (that is, it doesn’t happen that way on my
box).
Is it a windows or *nix box? and can you read files over 4 gigs in
size?
I’m guessing that it’s a signed unsigned integer problem. When it
wraps around, does it start out at negative one and go upward fro
increasing file sizes?
Thats cool and does work for files less than 4 gigs but (obviously)
fails for larger files.
This packs the number as a signed integer, and unpacks it as an
unsigned one. If it works, you can make a quick function:
class Integer
def unsign() [self].pack(“i”).unpack(“I”) end
end
and…
File.stat(“somefile”).size.unsign
–Mark
Thanks,
Walt
···
Walter Szewelanczyk
IS Director
M.W. Sewall & CO. email : walter@mwsewall.com
259 Front St. Phone : (207) 442-7994 x 128
Bath, ME 04530 Fax : (207) 443-6284
> I'm not sure about windows, but on unix[1] the file size usually is
an
> off_t, which is a signed value[2], and traditionally was 32bit (i.e.
> +/- 2Gig).
>
> Following, some unices (namely solaris comes to mind) offered 'large
> file versions' of many of their calls, others (e.g. BSDs) just
> switched to a signed 64 bit value for off_t. I think the latter is
> called for in the windows compat layer (as I assume there is one,
> implementing stat) to really solve your problem.
>
> [1] -- in stat(2)'s struct stat that is
> [2] -- to seek(2) backwards
>
> Regards,
>
> -Martin
>
hmmm…I think your right. What is the correct way to report this?
ruby-core? Or is this list ok?
Walt
···
Walter Szewelanczyk
IS Director
M.W. Sewall & CO. email : walter@mwsewall.com
259 Front St. Phone : (207) 442-7994 x 128
Bath, ME 04530 Fax : (207) 443-6284
File.stat(‘file_name’).size returns erroneous file sizes when the
files are over 2 Gigs in size (at which point the number wraps around
and becomes negative). Is this an expected limitation or a bug?
I am using the Pragmatic Programmers 1.8.1-11 build on a windows 2000
machine.
I’m not sure about windows, but on unix[1] the file size usually is an
off_t, which is a signed value[2], and traditionally was 32bit (i.e. +/- 2Gig).
Following, some unices (namely solaris comes to mind) offered ‘large file
versions’ of many of their calls, others (e.g. BSDs) just switched to a
signed 64 bit value for off_t. I think the latter is called for in the windows
compat layer (as I assume there is one, implementing stat) to really solve your
problem.
[1] – in stat(2)'s struct stat that is
[2] – to seek(2) backwards
Regards,
-Martin
If you’re on a 64 bit platform you can build Ruby as a 64 bit app, and
that will also solve this problem.
My problem on Solaris is that, when I use a 32 bit Ruby 1.8.x, I must
disable large file support in order to get any extensions that use
procfs.h to compile. I’ve posted on this before with no solution.
It looks like a bug (that is, it doesn’t happen that way on my
box).
Is it a windows or *nix box? and can you read files over 4 gigs in
size?
It’s a mac, so it’s a *nix box.
I’m guessing that it’s a signed unsigned integer problem. When it
wraps around, does it start out at negative one and go upward fro
increasing file sizes?
It’s not a ruby issue. You need to compile it (and your system as
well) with large file support.
In my case, I just did # emerge ruby (I Love My Gentoo) e.g.
File.stat(“/distribs/debian/sid/dvd/debian-20040129-i386-binary-1.iso”)
=> #<File::Stat dev=0x303, ino=5177355, mode=0100644, nlink=1, uid=500,
gid=100, rdev=0x0, size=4674289664, blksize=4096, blocks=9138416,
atime=Sat Jan 31 20:52:50 CET 2004, mtime=Sat Jan 31 22:33:24 CET 2004,
ctime=Sat Jan 31 22:33:24 CET 2004>
Hth,
Xavier
···
On Thu, 25 Mar 2004 07:38:34 +0900, walter wrote:
> I'm not sure about windows, but on unix[1] the file size usually is
an
> off_t, which is a signed value[2], and traditionally was 32bit (i.e.
> +/- 2Gig).
>
> Following, some unices (namely solaris comes to mind) offered 'large
> file versions' of many of their calls, others (e.g. BSDs) just
> switched to a signed 64 bit value for off_t. I think the latter is
> called for in the windows compat layer (as I assume there is one,
> implementing stat) to really solve your problem.
>
> [1] -- in stat(2)'s struct stat that is
> [2] -- to seek(2) backwards
> >>> Is there a work around?
> >>
> >> try:
> >> [File.stat("filename").size].pack("i").unpack("I")
> >>
> >
> > Thats cool and does work for files less than 4 gigs but (obviously)
> > fails for larger files.
>
> whoops :/ I didn't think about that until I read your followup posting
> where you said 14 gigs...
>
> Okay, new workaround! :) Open in append mode, read the position.
>
> >> File.open("/Users/mark/bigfile","a"){|f|f.pos}
> => 12006195200
>
> Does that work?
>
> --Mark
>
>
I tried that but just got zero, so I tried
File.open(“file_name”,“a”) do |f|
f.seek(0, IO::SEEK_END)
puts “size = #{f.pos}”}
end
and this works for files less than 2 gigs but for files over 2 gigs I
get the following error :
`seek’: Invalid argument - s:\2004-03-
22\mssql\backup\ct_prod_back.DAT (Errno::EINVAL)
So it seems that the same issue would be at play here as well.
Thanks,
Walt
···
Walter Szewelanczyk
IS Director
M.W. Sewall & CO. email : walter@mwsewall.com
259 Front St. Phone : (207) 442-7994 x 128
Bath, ME 04530 Fax : (207) 443-6284
The legacy OS'' you're talking about happens to be the platform the OP runs on, so spare the derision for the appropriate lists please. (Others could be laughing you explicitely have to compile for large
file support’’ – where are we, in the 80s? Just spare it.)
…
So what does windows itself use to offer access onto those large numbers ?
Seems the stat compat (ms-w wasn’t posix last time I looked, so I’ll boldly
assume it has no stat itself) needs to use another library call. After all
it’s possible to write apps in C, compiled with msvc, running on windows,
which are able to properly report the filesize.
-Martin
···
On Thu, Mar 25, 2004 at 10:34:30PM +0900, Xavier wrote:
On Thu, 25 Mar 2004 21:10:43 +0900, nobu.nokada wrote:
Hi,
At Thu, 25 Mar 2004 20:59:27 +0900,
Xavier wrote in [ruby-talk:95826]:
It’s not a ruby issue. You need to compile it (and your system as
well) with large file support.
The main problem is that MSVCRT doesn’t provide it.
I have no idea what MSVCRT is, but since it starts with MS, it must be
part of a legacy OS that I haven’t touched in a light year.
At Thu, 25 Mar 2004 20:59:27 +0900,
Xavier wrote in [ruby-talk:95826]:
It’s not a ruby issue. You need to compile it (and your system as
well) with large file support.
The main problem is that MSVCRT doesn’t provide it.
I have no idea what MSVCRT is, but since it starts with MS, it must be
part of a legacy OS that I haven’t touched in a light year.
It smells like microsoft visual c runtime.
I didn’t know, thanks.
The ``legacy OS’’ you’re talking about happens to be the platform the
OP runs on, so spare the derision for the appropriate lists please.
No offense meant, sorry if I offended you.
(Others could be laughing you explicitely have to ``compile for large
file support’’ – where are we, in the 80s? Just spare it.)
When did I say you had to explicitely compile for large file support
yourself? It has to be done, though.
You had not mentioned what your platform was so I could only assume that
either your environment or your particular installation of ruby did not
support large files.
As mentioned, the only thing I did was “emerge ruby”. It’s not very
complex, is it?
BTW, why the 80s? Linux is younger, Unix and myself are quite older.
That said, I leave this discussion as I obviously can’t help you, sorry.
···
On Thu, 25 Mar 2004 22:44:20 +0900, Martin Weber wrote:
On Thu, Mar 25, 2004 at 10:34:30PM +0900, Xavier wrote:
On Thu, 25 Mar 2004 21:10:43 +0900, nobu.nokada wrote:
At Thu, 25 Mar 2004 22:44:20 +0900,
Martin Weber wrote in [ruby-talk:95833]:
So what does windows itself use to offer access onto those large numbers ?
Seems the stat compat (ms-w wasn’t posix last time I looked, so I’ll boldly
assume it has no stat itself) needs to use another library call. After all
it’s possible to write apps in C, compiled with msvc, running on windows,
which are able to properly report the filesize.
Unfortunately, it is not all of large file support. Of course,
Windows provides methods to access large files in their way,
but not quite portable.
I didn’t think about that until I read your followup posting
Open in append mode, read the position.