Too many files open?

ruby-talk
1

I wrote a pretty simple web scraping tool, and i'm having it organize
the results into folders based upon search queries. After six hundred
or so of these folders are created, i get a bizarre, "too many files
open" error(Errno::EMFILE).

This is the line of code it crashes on:
`mkdir #{foldername}`

And the only time I ever open and close files in my program is in
these few lines, but the program isn't crashing on these lines:

fileTempFile = File.new(filePath + "full_" + fileName +
".html","w")
fileTempFile.puts(response)
fileLinkText = File.new(filePath + "relevant_" + fileName +
".txt","w")
fileLinkText.puts(link + "\n" +
match.to_s.gsub(/<.*?>/,"").gsub(/&[a-zA-Z0-9#]*?;/,''))
fileLinkText.close
fileTempFile.close

As you can see, the files are being opened and closed almost
immediately.

Is there a different way I should be creating directories rather than
through a shell command?

2

I wrote a pretty simple web scraping tool, and i'm having it organize
the results into folders based upon search queries. After six hundred
or so of these folders are created, i get a bizarre, "too many files
open" error(Errno::EMFILE).

This is the line of code it crashes on:
`mkdir #{foldername}`

This opens a file to recieve the shell's response, instead use Dir.mkdir or FileUtils.

And the only time I ever open and close files in my program is in
these few lines, but the program isn't crashing on these lines:

fileTempFile = File.new(filePath + "full_" + fileName +
".html","w")
fileTempFile.puts(response)
fileLinkText = File.new(filePath + "relevant_" + fileName +
".txt","w")
fileLinkText.puts(link + "\n" +
match.to_s.gsub(/<.*?>/,"").gsub(/&[a-zA-Z0-9#]*?;/,''))
fileLinkText.close
fileTempFile.close

You should be using blocks here, to ensure the file is really closed.

match = match.to_s
match = match.gsub(/<.*?>/, "")
match = match.gsub(/&[a-zA-Z0-9#]*?;/, "")

link_text = "#{link}\n#{match}"

File.open "#{file_path}full_#{file_name}.html", "w" do |temp_file|
   temp_file.puts response
end

File.open "#{file_path}relevant_#{file_name}.txt", "w" do |link_file|
   link_file.puts link_text
end

As you can see, the files are being opened and closed almost
immediately.

If an exception is raised, your File may not be closed if you don't use the block form.

Is there a different way I should be creating directories rather than
through a shell command?

Dir.mkdir, and FileUtils has (I believe) mkpath as well.

ยทยทยท

On 16 Dec 2004, at 10:37, The Weeg wrote: