How to edit a file without loading it all into memory

ruby-talk
1

I need to change the texture paths in maya files. The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

Thanks.

···

--
Posted via http://www.ruby-forum.com/.

2

newer versions of sed can replace text in place in files.

there's also perl's

perl -pi -e 's/to change this/to change that/' filename

···

On Sat, 2007-11-10 at 05:51 +0900, Jonathan Dobbie wrote:

I need to change the texture paths in maya files. The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

Thanks.

3

As you said: do it line by line. This is the usual approach to this problem. You might want to have a look at Ruby's command line options. Basically you can do something like this for an inplace edit:

ruby -ni.bak -e 'your replacement code'

Kind regards

  robert

···

On 09.11.2007 21:51, Jonathan Dobbie wrote:

I need to change the texture paths in maya files. The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

4

Jonathan Dobbie wrote:

The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

File.foreach("data.txt") do |line|
  #look for something in line
end

···

--
Posted via http://www.ruby-forum.com/\.

5

7stud -- wrote:

Jonathan Dobbie wrote:

The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

File.foreach("data.txt") do |line|
  #look for something in line
end

Here is what I originally wrote, and what I just tried. Am I missing
something obvious? The first one duplicates the lines, and the second
one does nothing. (doesn't save the changes)

  def change_texture_paths
    File.open(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |file|
      file.grep( /setAttr ".fileTextureName" -type "string"/ ) do |line|
        texture_file_path = line[52..-2]
        texture_file_name = texture_file_path.split("\/").last
        line = " setAttr \".fileTextureName\" -type \"string\"
\"" + ENV['HOME'] + "/include/" + texture_file_name + "\n"
        file.print line
      end
    end

    File.foreach(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |line|
      if (line =~ /setAttr ".fileTextureName" -type "string"/ )
        texture_file_path = line[52..-2]
        texture_file_name = texture_file_path.split("\/").last
        line = " setAttr \".fileTextureName\" -type \"string\"
\"" + ENV['HOME'] + "/include/" + texture_file_name + "\n"
        file.print line
      end
    end
end

···

--
Posted via http://www.ruby-forum.com/\.

6

Err.

Well only one of these textures is open at once right?
If this box is actually capable of running maya well, it's sure to
have more than a few hundred megs of free ram (when maya isn't running
of course ;).

My guess is that you'll get a hefty performance hit for parsing the
files line by line. The box has memory to spare, so why not just use
it?

Of course if there was a way to memap the file under ruby that would
probably be the best...but I dunno how to do that.

--Kyle

7

Jonathan Dobbie wrote:

7stud -- wrote:

Jonathan Dobbie wrote:

The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

File.foreach("data.txt") do |line|
  #look for something in line
end

Here is what I originally wrote, and what I just tried. Am I missing
something obvious? The first one duplicates the lines, and the second
one does nothing. (doesn't save the changes)

1) Are you under the impression that every method in ruby has the same
parameters?

2) When you see an error message, do you just ignore it?

···

--
Posted via http://www.ruby-forum.com/\.

8

mem map. not memap..but you get the idea.