# How to edit a file without loading it all into memory **URL:** \ **Category:** ruby-talk\ **Created:** [9 November 2007 20:51 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058 "2007-11-09T20:51:29Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Jonathan\_Dobbie](https://avatars.discourse-cdn.com/v4/letter/j/258eb7/32.png) [@Jonathan\_Dobbie](https://rubytalk.org/u/Jonathan_Dobbie)\ **Post date:** [9 November 2007 20:51 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058/1 "2007-11-09T20:51:29Z")
I need to change the texture paths in maya files. The lines are easy to find with grep, and I'm able to create the new line easily enough, but how can I do this without holding the entire file into memory? The files can be hundreds of megabytes. Thanks. > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Reid\_Thompson1](https://avatars.discourse-cdn.com/v4/letter/r/6de8d8/32.png) [@Reid\_Thompson1](https://rubytalk.org/u/Reid_Thompson1)\ **Post date:** [9 November 2007 21:03 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058/2 "2007-11-09T21:03:43Z")
newer versions of sed can replace text in place in files. there's also perl's perl -pi -e 's/to change this/to change that/' filename > **···** > > On Sat, 2007-11-10 at 05:51 +0900, Jonathan Dobbie wrote: > > > I need to change the texture paths in maya files. The lines are easy to > > find with grep, and I'm able to create the new line easily enough, but > > how can I do this without holding the entire file into memory? The > > files can be hundreds of megabytes. > > > > Thanks. ---
**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\ **Post date:** [9 November 2007 21:10 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058/3 "2007-11-09T21:10:04Z")
As you said: do it line by line. This is the usual approach to this problem. You might want to have a look at Ruby's command line options. Basically you can do something like this for an inplace edit: ruby -ni.bak -e 'your replacement code' Kind regards   robert > **···** > > On 09.11.2007 21:51, Jonathan Dobbie wrote: > > > I need to change the texture paths in maya files. The lines are easy to > > find with grep, and I'm able to create the new line easily enough, but > > how can I do this without holding the entire file into memory? The > > files can be hundreds of megabytes. ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [9 November 2007 21:10 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058/4 "2007-11-09T21:10:47Z")
Jonathan Dobbie wrote: > The lines are easy to > find with grep, and I'm able to create the new line easily enough, but > how can I do this without holding the entire file into memory? The > files can be hundreds of megabytes. File.foreach("data.txt") do |line|   #look for something in line end > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Jonathan\_Dobbie](https://avatars.discourse-cdn.com/v4/letter/j/258eb7/32.png) [@Jonathan\_Dobbie](https://rubytalk.org/u/Jonathan_Dobbie)\ **Post date:** [9 November 2007 21:21 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058/5 "2007-11-09T21:21:30Z")
7stud -- wrote: > Jonathan Dobbie wrote: > > > The lines are easy to > > find with grep, and I'm able to create the new line easily enough, but > > how can I do this without holding the entire file into memory? The > > files can be hundreds of megabytes. > > File.foreach("data.txt") do |line| >   #look for something in line > end Here is what I originally wrote, and what I just tried. Am I missing something obvious? The first one duplicates the lines, and the second one does nothing. (doesn't save the changes)   def change\_texture\_paths     File.open(Dir.glob(ENV['HOME'] + "/\*.ma").first, "r+") do |file|       file.grep( /setAttr ".fileTextureName" -type "string"/ ) do |line|         texture\_file\_path = line[52..-2]         texture\_file\_name = texture\_file\_path.split("\/").last         line = " setAttr \".fileTextureName\" -type \"string\" \"" + ENV['HOME'] + "/include/" + texture\_file\_name + "\n"         file.print line       end     end     File.foreach(Dir.glob(ENV['HOME'] + "/\*.ma").first, "r+") do |line|       if (line =~ /setAttr ".fileTextureName" -type "string"/ )         texture\_file\_path = line[52..-2]         texture\_file\_name = texture\_file\_path.split("\/").last         line = " setAttr \".fileTextureName\" -type \"string\" \"" + ENV['HOME'] + "/include/" + texture\_file\_name + "\n"         file.print line       end     end end > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Kyle\_Schmitt](https://avatars.discourse-cdn.com/v4/letter/k/9e8a1a/32.png) [@Kyle\_Schmitt](https://rubytalk.org/u/Kyle_Schmitt)\ **Post date:** [9 November 2007 21:39 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058/6 "2007-11-09T21:39:51Z")
Err. Well only one of these textures is open at once right? If this box is actually capable of running maya well, it's sure to have more than a few hundred megs of free ram (when maya isn't running of course ;). My guess is that you'll get a hefty performance hit for parsing the files line by line. The box has memory to spare, so why not just use it? Of course if there was a way to memap the file under ruby that would probably be the best...but I dunno how to do that. --Kyle ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [10 November 2007 00:54 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058/7 "2007-11-10T00:54:18Z")
Jonathan Dobbie wrote: > 7stud -- wrote: > > > Jonathan Dobbie wrote: > > > > > The lines are easy to > > > find with grep, and I'm able to create the new line easily enough, but > > > how can I do this without holding the entire file into memory? The > > > files can be hundreds of megabytes. > > > > File.foreach("data.txt") do |line| > >   #look for something in line > > end > > Here is what I originally wrote, and what I just tried. Am I missing > something obvious? The first one duplicates the lines, and the second > one does nothing. (doesn't save the changes) 1) Are you under the impression that every method in ruby has the same parameters? 2) When you see an error message, do you just ignore it? > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Kyle\_Schmitt](https://avatars.discourse-cdn.com/v4/letter/k/9e8a1a/32.png) [@Kyle\_Schmitt](https://rubytalk.org/u/Kyle_Schmitt)\ **Post date:** [9 November 2007 21:40 UTC](https://rubytalk.org/t/how-to-edit-a-file-without-loading-it-all-into-memory/42058/8 "2007-11-09T21:40:10Z")
mem map. not memap..but you get the idea.