# Opening a large file many times / optimisation **URL:** \ **Category:** ruby-talk\ **Created:** [11 March 2007 16:30 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063 "2007-03-11T16:30:08Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Paul\_Nulty](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@Paul\_Nulty](https://rubytalk.org/u/Paul_Nulty)\ **Post date:** [11 March 2007 16:30 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/1 "2007-03-11T16:30:08Z")
hello, I have a method that basically searches through a largish (5mb) text file for a word. I need to call this method about 1400 times, and i care about speed. If i open the file at the start of the script and then pass the file object as a parameter to my method each time its called, the code runs quite a bit faster than if i open the file inside the method each time; but this seems ugly to me. Is there a standard way to do this in ruby? How much overhead is involved in opening a large text file? thanks. ---
**Author:** ![James\_Edward\_Gray\_II](https://avatars.discourse-cdn.com/v4/letter/j/ea5d25/32.png) [@James\_Edward\_Gray\_II](https://rubytalk.org/u/James_Edward_Gray_II)\ **Post date:** [11 March 2007 16:42 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/2 "2007-03-11T16:42:52Z")
Well, if you have enough RAM to support pulling it into memory, that's certainly going to be faster. However, there are some techniques you could use to speed up and index and query operation. See this old Ruby Quiz for some ideas: [http://www.rubyquiz.com/quiz54.html](http://www.rubyquiz.com/quiz54.html) James Edward Gray II > **···** > > On Mar 11, 2007, at 11:30 AM, Paul Nulty wrote: > > > I have a method that basically searches through a largish (5mb) text > > file for a word. I need to call this method about 1400 times, and i > > care about speed. > > > > If i open the file at the start of the script and then pass the file > > object as a parameter to my method each time its called, the code runs > > quite a bit faster than if i open the file inside the method each > > time; but this seems ugly to me. > > > > Is there a standard way to do this in ruby? How much overhead is > > involved in opening a large text file? ---
**Author:** ![M\_Edward\_Ed\_Borasky1](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@M\_Edward\_Ed\_Borasky1](https://rubytalk.org/u/M_Edward_Ed_Borasky1)\ **Post date:** [11 March 2007 16:46 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/3 "2007-03-11T16:46:15Z")
Paul Nulty wrote: > hello, > > I have a method that basically searches through a largish (5mb) text > file for a word. I need to call this method about 1400 times, and i > care about speed. > > If i open the file at the start of the script and then pass the file > object as a parameter to my method each time its called, the code runs > quite a bit faster than if i open the file inside the method each > time; but this seems ugly to me. > > Is there a standard way to do this in ruby? How much overhead is > involved in opening a large text file? > > thanks. >    1. You need to define the problem better. Are you searching for a different word each time, does the file change each time, etc. Why do you have to call it 1400 times? 2. Searching and indexing are extremely well documented areas of computer science. Once you've correctly defined your problem, I'm sure you'll come up with something far more efficient than a brute force "open a five megabyte file, read the whole enchilada into RAM, and do a text search for the word, then close the file and wait for the next request". 3. Do you care about scalability, or is the file \*never\* going to get bigger than 5 MBytes? Is the method \*always\* going to be called "only" 1400 times, or will someone see your success and say, "Great -- here's 20 million words!"? > **···** > > -- > M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P) > [http://borasky-research.blogspot.com/](http://borasky-research.blogspot.com/) > > If God had meant for carrots to be eaten cooked, He would have given rabbits fire. ---
**Author:** ![Paul\_Nulty](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@Paul\_Nulty](https://rubytalk.org/u/Paul_Nulty)\ **Post date:** [11 March 2007 17:25 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/4 "2007-03-11T17:25:08Z")
> 1. You need to define the problem better. Are you searching for a > different word each time, does the file change each time, etc. Why do > you have to call it 1400 times? ok here's a few lines from the file i'm searching (its a wordnet file that holds different senses of words) concavity%1:07:00:: 05070032 2 0 concavity%1:25:00:: 13864965 1 0 concavo-concave%5:00:00:concave:00 00536008 1 0 concavo-convex%5:00:00:concave:00 00536416 1 0 conceal%2:39:00:: 02146790 2 1 conceal%2:39:01:: 02144835 1 8 concealed%3:00:00:: 02088404 2 1 concealed%5:00:00:invisible:00 02517817 1 2 concealing%1:04:00:: 01048912 1 0 concealing%3:00:00:: 02091020 1 0 i need to search for the first part (e.g. conceal%2:39:00:🙂 and return the second last number (eg. 2). (getting the sense from the sense key, if you know wordnet) i have 1400 words, the wordnet file will never change. i'm unlikely to need to scale up much past 1400. here's my code: (senseKey is eg "conceal%2:39:00::") lines=File.readlines("/usr/local/WordNet-3.0/dict/index.sense") #gets a sysnet number from a sense key def getSense(senseKey,lines)   for line in lines     if line.index(senseKey)==0       words=line.split(" ")       return words[-2]     end   end end thanks again! ---
**Author:** ![M\_Edward\_Ed\_Borasky1](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@M\_Edward\_Ed\_Borasky1](https://rubytalk.org/u/M_Edward_Ed_Borasky1)\ **Post date:** [11 March 2007 17:35 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/5 "2007-03-11T17:35:19Z")
Paul Nulty wrote: > > 1. You need to define the problem better. Are you searching for a > > different word each time, does the file change each time, etc. Why do > > you have to call it 1400 times? > >      > > ok here's a few lines from the file i'm searching (its a wordnet file > > that holds different senses of words) > > concavity%1:07:00:: 05070032 2 0 > concavity%1:25:00:: 13864965 1 0 > concavo-concave%5:00:00:concave:00 00536008 1 0 > concavo-convex%5:00:00:concave:00 00536416 1 0 > conceal%2:39:00:: 02146790 2 1 > conceal%2:39:01:: 02144835 1 8 > concealed%3:00:00:: 02088404 2 1 > concealed%5:00:00:invisible:00 02517817 1 2 > concealing%1:04:00:: 01048912 1 0 > concealing%3:00:00:: 02091020 1 0 > > i need to search for the first part (e.g. conceal%2:39:00:🙂 and > return the second last number (eg. 2). (getting the sense from the > sense key, if you know wordnet) > > i have 1400 words, the wordnet file will never change. i'm unlikely to > need to scale up much past 1400. > > here's my code: (senseKey is eg "conceal%2:39:00::") > > lines=File.readlines("/usr/local/WordNet-3.0/dict/index.sense") > > #gets a sysnet number from a sense key > def getSense(senseKey,lines) >   for line in lines >     if line.index(senseKey)==0 >       words=line.split(" ") >       return words[-2] >     end >   end > end > > thanks again! >    Isn't there a Ruby/Wordnet interface? Doctor Google recommended [dev(E)iate](http://www.deveiate.org/projects/Ruby-WordNet/) > **···** > > -- > M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P) > [http://borasky-research.blogspot.com/](http://borasky-research.blogspot.com/) > > If God had meant for carrots to be eaten cooked, He would have given rabbits fire. ---
**Author:** ![Brian\_Candler](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@Brian\_Candler](https://rubytalk.org/u/Brian_Candler)\ **Post date:** [11 March 2007 20:42 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/6 "2007-03-11T20:42:12Z")
> \> > \> 1. You need to define the problem better. Are you searching for a > \> different word each time, does the file change each time, etc. Why do > \> you have to call it 1400 times? > > ok here's a few lines from the file i'm searching (its a wordnet file > that holds different senses of words) > > concavity%1:07:00:: 05070032 2 0 > concavity%1:25:00:: 13864965 1 0 > concavo-concave%5:00:00:concave:00 00536008 1 0 > concavo-convex%5:00:00:concave:00 00536416 1 0 > conceal%2:39:00:: 02146790 2 1 > conceal%2:39:01:: 02144835 1 8 > concealed%3:00:00:: 02088404 2 1 > concealed%5:00:00:invisible:00 02517817 1 2 > concealing%1:04:00:: 01048912 1 0 > concealing%3:00:00:: 02091020 1 0 > > i need to search for the first part (e.g. conceal%2:39:00:🙂 and > return the second last number (eg. 2). (getting the sense from the > sense key, if you know wordnet) > > i have 1400 words, the wordnet file will never change. i'm unlikely to > need to scale up much past 1400. If you're searching a 5MB file 1400 times, it's almost certainly worth reading it in once and building a hash as you go. Remember that on average, you are reading half the lines in the file on every search. So you should speed up by a factor of nearly 700 just by doing this. If the wordnet file is too big to fit into RAM, then there are ways of indexing the file on disk to make it quicker to search (external searching) > here's my code: (senseKey is eg "conceal%2:39:00::") > > lines=File.readlines("/usr/local/WordNet-3.0/dict/index.sense") > > #gets a sysnet number from a sense key > def getSense(senseKey,lines) >   for line in lines >     if line.index(senseKey)==0 >       words=line.split(" ") >       return words[-2] >     end >   end > end Try something like: class Wordnet   def initialize(filename)     @words = {}     File.open(filename) do |f|       f.each\_line do |line|         fields = line.chomp.split(/ /)         key = fields.shift         @words[key] = fields       end     end   end   def sysnet(senseKey)     @words[senseKey][1]   end end wn = Wordnet.new("/usr/local/WordNet-3.0/dict/index.sense") # Now do this 1400 times for different keys puts wn.sysnet("conceal%2:39:00::") > **···** > > On Mon, Mar 12, 2007 at 02:25:08AM +0900, Paul Nulty wrote: ---
**Author:** ![Paul\_Nulty](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@Paul\_Nulty](https://rubytalk.org/u/Paul_Nulty)\ **Post date:** [11 March 2007 18:00 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/7 "2007-03-11T18:00:05Z")
> Isn't there a Ruby/Wordnet interface? Doctor Google recommended > [dev(E)iate](http://www.deveiate.org/projects/Ruby-WordNet/) yep i'm using that; it's great but as far as i can tell it doesn't use sense keys, it uses sense numbers. I only have the sense keys, so i need to get the sense number from the sense key manually. ---
**Author:** ![Paul\_Nulty](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@Paul\_Nulty](https://rubytalk.org/u/Paul_Nulty)\ **Post date:** [12 March 2007 13:30 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/8 "2007-03-12T13:30:05Z")
Thanks! before: 142.800000 0.100000 142.900000 (156.818797) after (with hash) 9.900000 0.100000 10.000000 ( 11.259273) thanks again. ---
**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\ **Post date:** [11 March 2007 20:45 UTC](https://rubytalk.org/t/opening-a-large-file-many-times-optimisation/36063/9 "2007-03-11T20:45:05Z")
Try reading the file and storing all combinations in a Hash with sense key as key and number as value.   robert > **···** > > On 11.03.2007 18:55, Paul Nulty wrote: > > > > Isn't there a Ruby/Wordnet interface? Doctor Google recommended > > > [dev(E)iate](http://www.deveiate.org/projects/Ruby-WordNet/) > > > > yep i'm using that; it's great but as far as i can tell it doesn't use > > sense keys, it uses sense numbers. I only have the sense keys, so i > > need to get the sense number from the sense key manually.