Last week I posted a question about incorrect file sizes coming back
incorrectly from File.size? for really large files. I changed my script
to use the win32 version, which returns the correct sizes, and now the
script is unacceptably slow. The bottleneck is the File.size call. My
scripts startup went from about a minute to, well I don't know because
it's never finished. I'm walking a file path that is somewhat indirect,
but the other file operations perform well. I'm reading files, using
UNC paths, through samba on linux boxes that mount a giant file system
through fiber.
So anyway, does anyone have any alternatives?
···
________________________________
Carey Nation <mailto:carey.nation@cnn.com>
Lead Software Engineer
CNN BEST
Broadcast Production Systems
Video Solutions Group
(404)827-2935 (wk)
(404)824-0033 (cell)
________________________________
That's interesting. The File.size method is really a wrapper for
File::Stat#size, which uses stat64() to stat the file in question. I
wonder if it's choking on the Samba/UNC path. Wouldn't be the first
app to do so.
What happens if you replace File.size in lib/win32/file.rb with this
version:
# Untested
class << self
def size(file)
handle = CreateFile(
file,
GENERIC_READ,
FILE_SHARE_READ,
nil,
OPEN_EXISTING,
FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING,
nil
)
if handle == INVALID_HANDLE_VALUE
raise Error, get_last_error
end
file_size = [0].pack('Q')
GetFileSizeEx(handle, file_size)
file_size = file_size.unpack('Q')[0]
unless CloseHandle(handle)
raise Error, get_last_error
end
file_size
end
end
···
On Mar 18, 12:56 pm, "Nation, Carey" <Carey.Nat...@turner.com> wrote:
Last week I posted a question about incorrect file sizes coming back
incorrectly from File.size? for really large files. I changed my script
to use the win32 version, which returns the correct sizes, and now the
script is unacceptably slow. The bottleneck is the File.size call. My
scripts startup went from about a minute to, well I don't know because
it's never finished. I'm walking a file path that is somewhat indirect,
but the other file operations perform well. I'm reading files, using
UNC paths, through samba on linux boxes that mount a giant file system
through fiber.
I'll try this. I don't think that it's the path because it _does_ work. It's just not in any hurry. I'll let you know in a bit about the call below...
···
-----Original Message-----
From: Daniel Berger [mailto:djberg96@gmail.com]
Sent: Tuesday, March 18, 2008 6:45 PM
To: ruby-talk ML
Subject: Re: win32 File.size slow?
On Mar 18, 12:56 pm, "Nation, Carey" <Carey.Nat...@turner.com> wrote:
Last week I posted a question about incorrect file sizes coming back
incorrectly from File.size? for really large files. I changed my script
to use the win32 version, which returns the correct sizes, and now the
script is unacceptably slow. The bottleneck is the File.size call. My
scripts startup went from about a minute to, well I don't know because
it's never finished. I'm walking a file path that is somewhat indirect,
but the other file operations perform well. I'm reading files, using
UNC paths, through samba on linux boxes that mount a giant file system
through fiber.
That's interesting. The File.size method is really a wrapper for
File::Stat#size, which uses stat64() to stat the file in question. I
wonder if it's choking on the Samba/UNC path. Wouldn't be the first
app to do so.
What happens if you replace File.size in lib/win32/file.rb with this
version:
# Untested
class << self
def size(file)
handle = CreateFile(
file,
GENERIC_READ,
FILE_SHARE_READ,
nil,
OPEN_EXISTING,
FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING,
nil
)
if handle == INVALID_HANDLE_VALUE
raise Error, get_last_error
end
file_size = [0].pack('Q')
GetFileSizeEx(handle, file_size)
file_size = file_size.unpack('Q')[0]
unless CloseHandle(handle)
raise Error, get_last_error
end
file_size
end
end
That works and the performance is good. Two caveats. First, in the CreateFile, it wants 0 rather than nil. Second, CloseHandle isn't implemented in windows/file.rb, so I added it to my local copy. This is actually a really big oversight since leaving handles of any kind open in win32 is a big leak problem.
Thanks!
···
-----Original Message-----
From: Daniel Berger [mailto:djberg96@gmail.com]
Sent: Tuesday, March 18, 2008 6:45 PM
To: ruby-talk ML
Subject: Re: win32 File.size slow?
On Mar 18, 12:56 pm, "Nation, Carey" <Carey.Nat...@turner.com> wrote:
Last week I posted a question about incorrect file sizes coming back
incorrectly from File.size? for really large files. I changed my script
to use the win32 version, which returns the correct sizes, and now the
script is unacceptably slow. The bottleneck is the File.size call. My
scripts startup went from about a minute to, well I don't know because
it's never finished. I'm walking a file path that is somewhat indirect,
but the other file operations perform well. I'm reading files, using
UNC paths, through samba on linux boxes that mount a giant file system
through fiber.
That's interesting. The File.size method is really a wrapper for
File::Stat#size, which uses stat64() to stat the file in question. I
wonder if it's choking on the Samba/UNC path. Wouldn't be the first
app to do so.
What happens if you replace File.size in lib/win32/file.rb with this
version:
# Untested
class << self
def size(file)
handle = CreateFile(
file,
GENERIC_READ,
FILE_SHARE_READ,
nil,
OPEN_EXISTING,
FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING,
nil
)
if handle == INVALID_HANDLE_VALUE
raise Error, get_last_error
end
file_size = [0].pack('Q')
GetFileSizeEx(handle, file_size)
file_size = file_size.unpack('Q')[0]
unless CloseHandle(handle)
raise Error, get_last_error
end
file_size
end
end
That works and the performance is good. Two caveats. First, in the CreateFile, it wants 0 rather than nil.
It should work. I'll look into that.
Second, CloseHandle isn't implemented in windows/file.rb, so I added it to my local copy. This is actually a really big oversight since leaving handles of any kind open in win32 is a big leak problem.
Ah, ok. Not to panic, since win32-file uses stat64(), so there are no
open handles to begin with.
Regards,
Dan
···
On Mar 19, 12:23 pm, "Nation, Carey" <Carey.Nat...@turner.com> wrote: