Newbie: unpacking binary data

ruby-talk
1

Hi,
Thanks for the efforts put into making Ruby available!
While new to Ruby it seems worth investing some time in. Eventually I'd
like to unpack some data from a binary file and insert it into a MySQL
table.
At the moment I'm having trouble looping over the source data file, and
just viewing the data to make sure things are proceeding correctly.

FYI: eventually I need this to run quickly (many GB's to be processed),
but for the moment I'll settle for getting anything to work :slight_smile:

The ruby details are:
$ ruby -v
ruby 1.8.5 (2006-08-25) [i386-cygwin]

···

-----------
Attempt 1 ( an error )
-----------
File.foreach("./INPUT/T200510A.BIN", "rb") do |f|
   print f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
end
NoMethodError: [FATAL] failed to allocate memory

-----------
Attempt 2 ( an error )
-----------
File.open("./INPUT/T200510A.BIN", "rb").readlines do |f|
  data = f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
  print data
end
/usr/lib/ruby/1.8/irb.rb:298:in `inspect': failed to allocate memory
(NoMemoryEr
ror)
        from /usr/lib/ruby/1.8/irb.rb:298:in `output_value'
        from /usr/lib/ruby/1.8/irb.rb:151:in `eval_input'
        from /usr/lib/ruby/1.8/irb.rb:259:in `signal_status'
        from /usr/lib/ruby/1.8/irb.rb:147:in `eval_input'
        from /usr/lib/ruby/1.8/irb/ruby-lex.rb:244:in
`each_top_level_statement'

        from /usr/lib/ruby/1.8/irb/ruby-lex.rb:230:in `loop'
        from /usr/lib/ruby/1.8/irb/ruby-lex.rb:230:in
`each_top_level_statement'

        from /usr/lib/ruby/1.8/irb/ruby-lex.rb:229:in `catch'
        from /usr/lib/ruby/1.8/irb/ruby-lex.rb:229:in
`each_top_level_statement'

        from /usr/lib/ruby/1.8/irb.rb:146:in `eval_input'
        from /usr/lib/ruby/1.8/irb.rb:70:in `start'
        from /usr/lib/ruby/1.8/irb.rb:69:in `catch'
        from /usr/lib/ruby/1.8/irb.rb:69:in `start'
        from /usr/bin/irb:13

--
Posted via http://www.ruby-forum.com/.

2

Hi,
Thanks for the efforts put into making Ruby available!
While new to Ruby it seems worth investing some time in. Eventually I'd
like to unpack some data from a binary file and insert it into a MySQL
table.
At the moment I'm having trouble looping over the source data file, and
just viewing the data to make sure things are proceeding correctly.

FYI: eventually I need this to run quickly (many GB's to be processed),
but for the moment I'll settle for getting anything to work :slight_smile:

The ruby details are:
$ ruby -v
ruby 1.8.5 (2006-08-25) [i386-cygwin]

-----------
Attempt 1 ( an error )
-----------
File.foreach("./INPUT/T200510A.BIN", "rb") do |f|
   print f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
end
NoMethodError: [FATAL] failed to allocate memory

This does not work because the second argument to foreach is the /line separator/ - not the file mode.

-----------
Attempt 2 ( an error )
-----------
File.open("./INPUT/T200510A.BIN", "rb").readlines do |f|
  data = f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
  print data
end
/usr/lib/ruby/1.8/irb.rb:298:in `inspect': failed to allocate memory
(NoMemoryError)

This fails because you use readlines - this will try to read the whole file into memory as /lines/ - and since there are no lines in a binary file this chokes while attempting to store the whole file or a large portion of it in a single line (i.e. String).

You should do this

File.open("./INPUT/T200510A.BIN", "rb") do |io|
   while ( data = io.read(19) )
     p data.unpack('I4I4I4I2I2a2a1')
   end
end

Regards

  robert

···

On 14.12.2006 12:14, Mark Van de vyver wrote:

3

Robert Klemme wrote:

···

On 14.12.2006 12:14, Mark Van de vyver wrote:

You should do this

File.open("./INPUT/T200510A.BIN", "rb") do |io|
   while ( data = io.read(19) )
     p data.unpack('I4I4I4I2I2a2a1')
   end
end

Regards

  robert

Thanks Robert. Much appreciated.