Reading a file in chunks, to a byte array

ruby-talk
1

Hi everyone,

I'm pretty new to Ruby, I'm taking courses and reading as much info as I can, but I'm stuck on this one problem that may/may not have a simple answer.

A quick summary of what I'm trying to accomplish:

I've written a simple SOAP client that interfaces with a web app used for file transfer... It works great, however, if a file is over a certain size it just hangs - and I know the reason why...
The webapp receives the file as a chunk, in the form of a byte array, the great thing about Ruby is that strings are basically an arrangement of arbitrary bytes, so I can just pass a string with the file object into the web service and upload small files.

But I'd like to split a file into "chunks" .. I know that I can do this:

···

---

def get_chunk(bytearray)
bytearray.read(61440) #read a 60kb chunk of the byte array
end

localFile = File.open('test.dat')

get_chunk(localFile)

---

But I need to keep track of more information .. The web app is looking for an offset integer , which basically works like this:

(with a 60k chunksize)

the first request will be:
offset = 0

second request:
offset = 61440

third request:
offset = 122880

Does anyone know how I could iterate over a bytearray, keeping track of the offset, and read 60k chunks all at the same time?
in pseudo code:

get_chunk(bytearray)

bytearray.read(61440)
offset = 0
puts bytearray.pos
=> 61440

bytearray.read(61440)
offset += 61440
puts offset
=> 61440
puts bytearray.pos
=> 122880
....

After I have the proper offset, and chunks of a bytearray - I need to keep calling the web service method with that data until the file = eof.

I'm horrible at code blocks and iteration, if anyone has any suggestions, it would be greatly appreciated.

Thanks,

Brian

2

It's not fully clear to me which one is the driving application here. If it is your client application, it's easy

File.open "foo", "rb" do |io|
   pos = io.tell
   while buffer = io.read(CHUNK_SIZE)
     send_to_web_app(pos, buffer)
     pos = io.tell
   end
end

Kind regards

  robert

···

On 29.01.2009 22:23, draygen@comcast.net wrote:

[Note: parts of this message were removed to make it a legal post.]

Hi everyone,

I'm pretty new to Ruby, I'm taking courses and reading as much info as I can, but I'm stuck on this one problem that may/may not have a simple answer.

A quick summary of what I'm trying to accomplish:

I've written a simple SOAP client that interfaces with a web app used for file transfer... It works great, however, if a file is over a certain size it just hangs - and I know the reason why... The webapp receives the file as a chunk, in the form of a byte array, the great thing about Ruby is that strings are basically an arrangement of arbitrary bytes, so I can just pass a string with the file object into the web service and upload small files.

But I'd like to split a file into "chunks" .. I know that I can do this:

---

def get_chunk(bytearray) bytearray.read(61440) #read a 60kb chunk of the byte array end

localFile = File.open('test.dat')

get_chunk(localFile)

---

But I need to keep track of more information .. The web app is looking for an offset integer , which basically works like this:

(with a 60k chunksize)

the first request will be: offset = 0

second request: offset = 61440

third request: offset = 122880

Does anyone know how I could iterate over a bytearray, keeping track of the offset, and read 60k chunks all at the same time? in pseudo code:

get_chunk(bytearray)

bytearray.read(61440) offset = 0 puts bytearray.pos => 61440

bytearray.read(61440) offset += 61440 puts offset => 61440 puts bytearray.pos => 122880 ...

After I have the proper offset, and chunks of a bytearray - I need to keep calling the web service method with that data until the file = eof.

I'm horrible at code blocks and iteration, if anyone has any suggestions, it would be greatly appreciated.

--
remember.guy do |as, often| as.you_can - without end

3

Robert,

That did it! Thank you so much, very simple solution - I just need to understand iteration/blocks a bit better for me to figure that out on my own.

thanks,

Brian

···

----- Original Message -----
From: "Robert Klemme" <shortcutter@googlemail.com>
To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
Sent: Thursday, January 29, 2009 4:57:37 PM GMT -05:00 US/Canada Eastern
Subject: Re: Reading a file in chunks, to a byte array

On 29.01.2009 22:23, draygen@comcast.net wrote:

[Note: parts of this message were removed to make it a legal post.]

Hi everyone,

I'm pretty new to Ruby, I'm taking courses and reading as much info as I can, but I'm stuck on this one problem that may/may not have a simple answer.

A quick summary of what I'm trying to accomplish:

I've written a simple SOAP client that interfaces with a web app used for file transfer... It works great, however, if a file is over a certain size it just hangs - and I know the reason why...
The webapp receives the file as a chunk, in the form of a byte array, the great thing about Ruby is that strings are basically an arrangement of arbitrary bytes, so I can just pass a string with the file object into the web service and upload small files.

But I'd like to split a file into "chunks" .. I know that I can do this:

---

def get_chunk(bytearray)
bytearray.read(61440) #read a 60kb chunk of the byte array
end

localFile = File.open('test.dat')

get_chunk(localFile)

---

But I need to keep track of more information .. The web app is looking for an offset integer , which basically works like this:

(with a 60k chunksize)

the first request will be:
offset = 0

second request:
offset = 61440

third request:
offset = 122880

Does anyone know how I could iterate over a bytearray, keeping track of the offset, and read 60k chunks all at the same time?
in pseudo code:

get_chunk(bytearray)

bytearray.read(61440)
offset = 0
puts bytearray.pos
=> 61440

bytearray.read(61440)
offset += 61440
puts offset
=> 61440
puts bytearray.pos
=> 122880
...

After I have the proper offset, and chunks of a bytearray - I need to keep calling the web service method with that data until the file = eof.

I'm horrible at code blocks and iteration, if anyone has any suggestions, it would be greatly appreciated.

It's not fully clear to me which one is the driving application here.
If it is your client application, it's easy

File.open "foo", "rb" do |io|
pos = io.tell
while buffer = io.read(CHUNK_SIZE)
send_to_web_app(pos, buffer)
pos = io.tell
end
end

Kind regards

    robert

--
remember.guy do |as, often| as.you_can - without end