Reading and writing a binary file

ruby-talk
1

Hi,

I am having trouble reading values from an existing (or maybe not yet
exisiting) binary file:
My target is to open a binary file for reading and writing (if it does not
exisit it should be created).
If the file was not yes existing I'd like to fill it with binary data to a
certain predefined size.
Later on I'd like to position my file position, and read and write whatever
I like.

The following piece does not work as intended:

  require 'pp'
  UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
  filename='foo.bin'
  size=1024
  dont_write=FileTest.exists?(filename) && File.stat(filename).size==size
  begin
       fd=File.open(filename,"ab+")
       fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
  rescue => e
       pp e
       fd=File.open(filename,"wb+")
       puts "opened #{filename} with 'wib+'"
  end

  fd.sync=true
  fd.seek(0)
  unless dont_write
      bin_data = [0].pack('c')*(size)
      fd.write(bin_data)
      # OK now I really have a file of size 1024 with a binay 0 contents
  end

   ###.... then sometime later using the same file descriptor (fd) :
    requestes_stream_position = 900
    fd.pos=(requestes_stream_position)
    p "tell=#{fd.tell}"
    # Now please read 30 binary 0s into a binary string
    bin_value=fd.read(30)

    #### I ASSUMED THAT bin_value now is a "binary string of size 30", but
unfortunately it is nil !!!

Thanks in adcance,
C.

2

"Chris Guenther" <chris_guenther@freenet.de> schrieb im Newsbeitrag
news:d1mhu3$bi6$1@news.mch.sbs.de...

Hi,

I am having trouble reading values from an existing (or maybe not yet
exisiting) binary file:
My target is to open a binary file for reading and writing (if it does

not

exisit it should be created).
If the file was not yes existing I'd like to fill it with binary data to

a

certain predefined size.
Later on I'd like to position my file position, and read and write

whatever

I like.

The following piece does not work as intended:

  require 'pp'
  UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
  filename='foo.bin'
  size=1024
  dont_write=FileTest.exists?(filename) &&

File.stat(filename).size==size

  begin
       fd=File.open(filename,"ab+")
       fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
  rescue => e
       pp e
       fd=File.open(filename,"wb+")
       puts "opened #{filename} with 'wib+'"
  end

  fd.sync=true
  fd.seek(0)
  unless dont_write
      bin_data = [0].pack('c')*(size)
      fd.write(bin_data)
      # OK now I really have a file of size 1024 with a binay 0 contents
  end

   ###.... then sometime later using the same file descriptor (fd) :
    requestes_stream_position = 900
    fd.pos=(requestes_stream_position)
    p "tell=#{fd.tell}"
    # Now please read 30 binary 0s into a binary string
    bin_value=fd.read(30)

    #### I ASSUMED THAT bin_value now is a "binary string of size 30",

but

unfortunately it is nil !!!

Thanks in adcance,
C.

I don't know what you're doing wrong. Your script works fine for me.
Btw, you don't close the fd.

Here's how I'd do it:

filename='foo.bin'
size=1024

File.open(filename, "ab+") do |fd|
  fd.seek(0, IO::SEEK_END)

  if fd.tell < size
    fd.seek 0
    fd.write( "\000" * size );
  elsif fd.tell > size
    # if you need the file to be exact this size
    fd.truncate size
  end

  requestes_stream_position = 900
  fd.seek requestes_stream_position

  bin_value = fd.read(30)
  p bin_value
end

Notes:

- don't change binmode depending on platform, always use "b" for binary
files.

- use File.open with block for proper close / cleanup

Kind regards

    robert

3

Chris Guenther wrote:

Hi,

I am having trouble reading values from an existing (or maybe not yet
exisiting) binary file:
My target is to open a binary file for reading and writing (if it does not
exisit it should be created).
If the file was not yes existing I'd like to fill it with binary data to a
certain predefined size.
Later on I'd like to position my file position, and read and write whatever
I like.

The following piece does not work as intended:

  require 'pp'
  UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
  filename='foo.bin'
  size=1024
  dont_write=FileTest.exists?(filename) && File.stat(filename).size==size
  begin
       fd=File.open(filename,"ab+")
       fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
  rescue => e
       pp e
       fd=File.open(filename,"wb+")
       puts "opened #{filename} with 'wib+'"
  end

  fd.sync=true
  fd.seek(0)
  unless dont_write
      bin_data = [0].pack('c')*(size)
      fd.write(bin_data)
      # OK now I really have a file of size 1024 with a binay 0 contents
  end

   ###.... then sometime later using the same file descriptor (fd) :
    requestes_stream_position = 900
    fd.pos=(requestes_stream_position)
    p "tell=#{fd.tell}"
    # Now please read 30 binary 0s into a binary string
    bin_value=fd.read(30)

    #### I ASSUMED THAT bin_value now is a "binary string of size 30", but
unfortunately it is nil !!!

Thanks in adcance,
C.

Thanks Robert,

you are absolutely right with your remarks using a block. The code I posted was just a small copied fraction of the real code, in which I cannor use a block because the same fd will be stored in a hash for later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are right, it was not in the code I posted) , but now I have another related problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading it back from the patched binary file.
Any ideas??

Thanks again in advance,
Chris

#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
     fd.pos=0
     bin_data = [0].pack('c')*FSIZE
     fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
     fd.pos=0
     data=fd.read(FSIZE)
     data[OFFSET,VALUE.length]=VALUE
     fd.pos=0
     fd.write(data)
     p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

File.truncate(FNAME,FSIZE)
File.open(FNAME,"ab+") {|fd|
     fd.pos=OFFSET
     patched=fd.read(VALUE.length)
     p "read modified piece of data = '#{patched}'"
     puts "Oooops !!!" unless VALUE==patched
}

4

Thanks Robert,

you are absolutely right with your remarks using a block. The code I posted was just a small copied fraction of the real code, in which I cannor use a block because the same fd will be stored in a hash for later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are right, it was not in the code I posted) , but now I have another related problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading it back from the patched binary file.
Any ideas??

Thanks again in advance,
Chris

#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
     fd.pos=0
     bin_data = [0].pack('c')*FSIZE
     fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
     fd.pos=0
     data=fd.read(FSIZE)
     data[OFFSET,VALUE.length]=VALUE
     fd.pos=0
     fd.write(data)
     p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

File.truncate(FNAME,FSIZE)
File.open(FNAME,"ab+") {|fd|
     fd.pos=OFFSET
     patched=fd.read(VALUE.length)
     p "read modified piece of data = '#{patched}'"
     puts "Oooops !!!" unless VALUE==patched
}

5

"Chris Günther" <cg_@gmx.de> schrieb im Newsbeitrag
news:42400070.2050609@gmx.de...

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??

Thanks again in advance,
Chris

#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
     fd.pos=0
     bin_data = [0].pack('c')*FSIZE
     fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
     fd.pos=0
     data=fd.read(FSIZE)
     data[OFFSET,VALUE.length]=VALUE
     fd.pos=0
     fd.write(data)
     p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

You truncate too often:

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
     fd.pos=OFFSET
     patched=fd.read(VALUE.length)
     p "read modified piece of data = '#{patched}'"
     puts "Oooops !!!" unless VALUE==patched
}

Careful when you're copying and pasting.

Regards

    robert

6

Chris Günther schrieb:

... I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading it back from the patched binary file.
Any ideas??
...
File.open(FNAME,"ab+") {|fd|
    fd.pos=0
...

Hi Chris,

it seems that the mode "ab+" is your problem. According to [1], "a+" means open the file for reading and writing, starting at the end of the file. This also means (at least on windows), that the end of file is position 0. So what you are doing above is writing the patched data after the end of the original file.

For patching a binary file I'd change your code to:

   FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
   [?c,?a,?f,?e].pack('cccc')

   # "wb+" for initial data: truncate existing content
   File.open(FNAME,"wb+") {|fd|
       bin_data = [0].pack('c')*FSIZE
       fd.write(bin_data)
   }

   # "rb+" for the patch: open at the beginning
   File.open(FNAME,"rb+") {|fd|
       fd.pos=OFFSET
       fd.write(VALUE)
       p "written modified piece of data = '#{VALUE}'"
   }

   # "rb" for reading the data
   File.open(FNAME,"rb") {|fd|
       fd.pos=OFFSET
       patched=fd.read(VALUE.length)
       p "read modified piece of data = '#{patched}'"
       puts "Oooops !!!" unless VALUE==patched
   }

Regards,
Pit

[1] class IO - RDoc Documentation

7

[Chris Günther <cg_@gmx.de>, 2005-03-22 12.29 CET]

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??

All your problems come from the fact that your are opening the file for
appending ("ab+") instead of for reading ("r+b"). With "ab+", you can set fd.pos=0,
if you want, but it will always write at the end of file.

8

Robert Klemme wrote:

"Chris Günther" <cg_@gmx.de> schrieb im Newsbeitrag
news:42400070.2050609@gmx.de...

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??

Thanks again in advance,
Chris

#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
    fd.pos=0
    bin_data = [0].pack('c')*FSIZE
    fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
    fd.pos=0
    data=fd.read(FSIZE)
    data[OFFSET,VALUE.length]=VALUE
    fd.pos=0
    fd.write(data)
    p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

You truncate too often:

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
    fd.pos=OFFSET
    patched=fd.read(VALUE.length)
    p "read modified piece of data = '#{patched}'"
    puts "Oooops !!!" unless VALUE==patched
}

Careful when you're copying and pasting.

Regards

    robert

I dont see your point. I have to truncate after every write to ensure the file size does not change. Where is the connection to the problem that the read value does not match the written value ?

regards,
C.

9

Robert Klemme wrote:

"Chris Günther" <cg_@gmx.de> schrieb im Newsbeitrag
news:42400070.2050609@gmx.de...

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??

Thanks again in advance,
Chris

#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
    fd.pos=0
    bin_data = [0].pack('c')*FSIZE
    fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
    fd.pos=0
    data=fd.read(FSIZE)
    data[OFFSET,VALUE.length]=VALUE
    fd.pos=0
    fd.write(data)
    p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

You truncate too often:

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
    fd.pos=OFFSET
    patched=fd.read(VALUE.length)
    p "read modified piece of data = '#{patched}'"
    puts "Oooops !!!" unless VALUE==patched
}

Careful when you're copying and pasting.

Regards

    robert

I dont see your point. I have to truncate after every write to ensure the file size does not change. Where is the connection to the problem that the read value does not match the written value ?

regards,
C.

10

Pit Capitain wrote:

Chris Günther schrieb:

... I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading it back from the patched binary file.
Any ideas??
...
File.open(FNAME,"ab+") {|fd|
    fd.pos=0
...

Hi Chris,

it seems that the mode "ab+" is your problem. According to [1], "a+" means open the file for reading and writing, starting at the end of the file. This also means (at least on windows), that the end of file is position 0. So what you are doing above is writing the patched data after the end of the original file.

For patching a binary file I'd change your code to:

  FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
  [?c,?a,?f,?e].pack('cccc')

  # "wb+" for initial data: truncate existing content
  File.open(FNAME,"wb+") {|fd|
      bin_data = [0].pack('c')*FSIZE
      fd.write(bin_data)
  }

  # "rb+" for the patch: open at the beginning
  File.open(FNAME,"rb+") {|fd|
      fd.pos=OFFSET
      fd.write(VALUE)
      p "written modified piece of data = '#{VALUE}'"
  }

  # "rb" for reading the data
  File.open(FNAME,"rb") {|fd|
      fd.pos=OFFSET
      patched=fd.read(VALUE.length)
      p "read modified piece of data = '#{patched}'"
      puts "Oooops !!!" unless VALUE==patched
  }

Regards,
Pit

[1] class IO - RDoc Documentation

Hi Pit,
thanks for your contribution.
That's exactly the solution I already have. What I dont like about this solution is that you have to open and close file descriptors 3 times just for patching one little file (For sure a working solution but quite some overhead). Just for academical reason it seeems I have to fall back reading the C-sources :frowning: to find out what is going wrong under the hood.

Chris

11

Carlos wrote:

[Chris Günther <cg_@gmx.de>, 2005-03-22 12.29 CET]

Thanks Robert,

you are absolutely right with your remarks using a block. The code I posted was just a small copied fraction of the real code, in which I cannor use a block because the same fd will be stored in a hash for later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are right, it was not in the code I posted) , but now I have another related problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading it back from the patched binary file.
Any ideas??

All your problems come from the fact that your are opening the file for
appending ("ab+") instead of for reading ("r+b"). With "ab+", you can set fd.pos=0,
if you want, but it will always write at the end of file.

Ok, I got it - (maybe only for me) this kind of semanctics is really confusing (and not well described either).

Anyway thanks a lot,
Chris

12

"Chris Günther" <cg_@gmx.de> schrieb im Newsbeitrag
news:42405416.5050603@gmx.de...

<snip/>

I dont see your point. I have to truncate after every write to ensure
the file size does not change. Where is the connection to the problem
that the read value does not match the written value ?

I don't see the point either. :-} Sorry for the confusion, I overlooked
that you used the other form of truncate with a length.

Regards

    robert

13

Chris Günther schrieb:

That's exactly the solution I already have. What I dont like about this solution is that you have to open and close file descriptors 3 times just for patching one little file (For sure a working solution but quite some overhead).

In order to patch an existing file, all you have to do is

  File.open(FNAME,"rb+") {|fd|
      fd.pos=OFFSET
      fd.write(VALUE)
      p "written modified piece of data = '#{VALUE}'"
  }

The other File.open calls are for creating the test file and for testing the contents of the patched file. I copied them from your initial script. Of course you could do all three steps (create, patch, test) in a single File.open block.

Regards,
Pit