Error handling with exceptions in multi-thread download

ruby-talk
1

there are three program fragments,all of them have same problems,
    all of them can run ,but when i downloaded 30% data,an error
ocurr
    undefined local variable or method `web' for Data:Class (NameError)
    cthreaddown is an array which contains many web.
    would you kind to tell me how to fix it?
    p1
     [code]
     for page_to_fetch in cthreaddown
     threads<<Thread.new(page_to_fetch) do |web|
         datafile="/home/pt/stock/"
         open(datafile,'a+'){|refile|
         open(web){|webfile|
              refile.puts webfile.read
              puts "#{web} is over"
         }}
        end
      end
     threads.each {|thr|
        begin
         thr.join
         rescue Timeout::Error => e
           puts "#{web} failed"
         rescue NameError => e
           puts "#{web} failed"
         rescue => e
           puts "#{web} failed"
        end
       }
      [/code]
     p2
     [code]
     for page_to_fetch in cthreaddown
     threads<<Thread.new(page_to_fetch) do |web|
         datafile="/home/pt/stock/"
         open(datafile,'a+'){|refile|
         begin
        open(web){|webfile|
              refile.puts webfile.read
              puts "#{web} is over"
         }}
        rescue NameError => e
           puts "#{web} failed"
        rescue => e
           puts "#{web} wrong"
         end
       end
      end
      threads.each {|thr| thr.join }
     [/code]

     p3
     [code]
     for page_to_fetch in cthreaddown
     threads<<Thread.new(page_to_fetch) do |web|
         datafile="/home/pt/stock/"
         open(datafile,'a+'){|refile|
         begin
        open(web){|webfile|
              refile.puts webfile.read
              puts "#{web} is over"
         }}
        rescue NameError => e
           puts "#{web} failed"
        rescue => e
           puts "#{web} wrong"
         end
       end
      end
   threads.each {|thr|
        begin
         thr.join
         rescue Timeout::Error => e
           puts "#{web} failed"
         rescue NameError => e
           puts "#{web} failed"
         rescue => e
           puts "#{web} failed"
        end
       }

···

--
Posted via http://www.ruby-forum.com/.

2

You probably have a TimeOut error. And the errors actually comes from:

        rescue Timeout::Error => e
          puts "#{web} failed"

because web is not defined there.

B

···

On Fri, Aug 27, 2010 at 11:58 AM, Pen Ttt <myocean135@yahoo.cn> wrote:

   there are three program fragments,all of them have same problems,
   all of them can run ,but when i downloaded 30% data,an error
ocurr
   undefined local variable or method `web' for Data:Class (NameError)
   cthreaddown is an array which contains many web.
   would you kind to tell me how to fix it?
   p1
    [code]
    for page_to_fetch in cthreaddown
    threads<<Thread.new(page_to_fetch) do |web|
        datafile="/home/pt/stock/"
        open(datafile,'a+'){|refile|
        open(web){|webfile|
             refile.puts webfile.read
             puts "#{web} is over"
        }}
       end
     end
    threads.each {|thr|
       begin
        thr.join
        rescue Timeout::Error => e
          puts "#{web} failed"
        rescue NameError => e
          puts "#{web} failed"
        rescue => e
          puts "#{web} failed"
       end
      }
     [/code]
    p2
    [code]
    for page_to_fetch in cthreaddown
    threads<<Thread.new(page_to_fetch) do |web|
        datafile="/home/pt/stock/"
        open(datafile,'a+'){|refile|
        begin
       open(web){|webfile|
             refile.puts webfile.read
             puts "#{web} is over"
        }}
       rescue NameError => e
          puts "#{web} failed"
       rescue => e
          puts "#{web} wrong"
        end
      end
     end
     threads.each {|thr| thr.join }
    [/code]

    p3
    [code]
    for page_to_fetch in cthreaddown
    threads<<Thread.new(page_to_fetch) do |web|
        datafile="/home/pt/stock/"
        open(datafile,'a+'){|refile|
        begin
       open(web){|webfile|
             refile.puts webfile.read
             puts "#{web} is over"
        }}
       rescue NameError => e
          puts "#{web} failed"
       rescue => e
          puts "#{web} wrong"
        end
      end
     end
  threads.each {|thr|
       begin
        thr.join
        rescue Timeout::Error => e
          puts "#{web} failed"
        rescue NameError => e
          puts "#{web} failed"
        rescue => e
          puts "#{web} failed"
       end
      }
--
Posted via http://www.ruby-forum.com/\.

3

there are three program fragments,all of them have same problems,
all of them can run ,but when i downloaded 30% data,an error
ocurr
undefined local variable or method `web' for Data:Class (NameError)

You need to look into class methods of class Data. Somewhere there
you will find usage of "web" which is undefined. Unfortunately you
did not post that class so we cannot tell you where your error is.

cthreaddown is an array which contains many web.
would you kind to tell me how to fix it?
p1
[code]
for page_to_fetch in cthreaddown
threads<<Thread.new(page_to_fetch) do |web|
datafile="/home/pt/stock/"
open(datafile,'a+'){|refile|
open(web){|webfile|
refile.puts webfile.read

You should use #write with #read and not #puts, because #puts does not
write out arguments unmodified. Also note that if webfile is large
this approach will break. You should better use blocked reading.

buffer = ""
while webfile.read(1024, buffer)
  refile.write buffer
end

         puts  &quot;\#\{web\}  is over&quot;
    \}\}
   end
 end
threads\.each \{|thr|
   begin
    thr\.join
    rescue Timeout::Error =&gt; e
      puts    &quot;\#\{web\}  failed&quot;
    rescue NameError =&gt; e
      puts    &quot;\#\{web\}  failed&quot;
    rescue =&gt; e
      puts    &quot;\#\{web\}  failed&quot;
   end
  \}
 \[/code\]

Cheers

robert

···

2010/8/27 Pen Ttt <myocean135@yahoo.cn>:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

4

i have omitted many lines of code,no TimeOut error,
   for page_to_fetch in cthreaddown
     threads<<Thread.new(page_to_fetch) do |web|
i have got an array--- cthreaddown before
it can run , can get 30% data ,and then stop,

undefined local variable or method `web' for Data:Class (NameError)

···

--
Posted via http://www.ruby-forum.com/.

5

require 'rubygems'
  require 'open-uri'
  require 'nokogiri'
  require 'mysql'
  require 'date'

class Data
  def self.getdaystock()
  time0=Time.now
  i=0
  cdown=[]
  threads=[]
  cthreaddown=[]
  open("/home/pt/usa/stocklist",“r”){|item|
      while line=item.gets
        cdown<<line
      end}
  a=(fromdate.split("-")[1].to_i-1).to_s
  b=fromdate.split("-")[2].to_s
  c=fromdate.split("-")[0].to_s
  d=Time.now.mon.to_s
  e=Time.now.day.to_s
  f=Time.now.year.to_s
  cdown=cdown.map {|item|
  item=‘http://ichart.finance.yahoo.com/table.csv?s=’+item.to_s+’&a=’+a+’&b=’+b+’&c=’+c+
         ‘&amp;d=’+d+’&amp;e=’+e+’&amp;f=’+f+’&amp;g=d&amp;ignore=.csv’}
  cdown.each.with_index {|item,idx|
    cthreaddown<< item
    i=i+1
    if i==100 or idx==cdown.size-1 then
      cthreaddown=cthreaddown.uniq
      for page_to_fetch in cthreaddown
      threads<<Thread.new(page_to_fetch) do |web|
         datafile="/home/pt/usa/stock/"+web.split("=")[1].to_s.gsub("&amp;a","")
         begin
           open(datafile,‘a+’){|refile|
           open(web){|webfile|
              refile.puts webfile.read
              puts “#{web} over” }}
         rescue Timeout::Error => e
           puts “#{web} failed”
         rescue NameError => e
           puts “#{web} failed”
         rescue => e
           puts “#{web} wrong”
         end
       end
      end
      threads.each {|thr|
        begin
          thr.join
        rescue Timeout::Error => e
          puts “#{web} failed”
        rescue NameError => e
          puts “#{web} failed”
        rescue => e
          puts “#{web} failed”
        end }
       i=0
       cthreaddown=[]
       threads = []
       sleep 0.01
    else
    end
    }
  end
end

i have opened 100 threads. there are 6000 companies in
/home/pt/usa/stocklist .

···

--
Posted via http://www.ruby-forum.com/.

6

sorry for that: def self.getdaystock(fromdate)

···

--
Posted via http://www.ruby-forum.com/.

7

Well, there you have it. The error is in your exception handling code.

Btw, the whole code could use a bit of commenting and also be simplified in areas. For example: you store Time.now initially and then call it over and over again to fetch individual fields. This is not only inefficient but also error prone because you will not get all values from the same point in time.

Also, it's a bad die to hardcode the URL in the middle of the code. It's much better to pass this as parameter or at least store it in some constant for easier reference and documentation purposes.

Cheers

  robert

···

On 27.08.2010 16:22, Pen Ttt wrote:

   require 'rubygems'
   require 'open-uri'
   require 'nokogiri'
   require 'mysql'
   require 'date'

class Data
   def self.getdaystock()
   time0=Time.now
   i=0
   cdown=
   threads=
   cthreaddown=
   open("/home/pt/usa/stocklist","r"){|item|
       while line=item.gets
         cdown<<line
       end}
   a=(fromdate.split("-")[1].to_i-1).to_s
   b=fromdate.split("-")[2].to_s
   c=fromdate.split("-")[0].to_s
   d=Time.now.mon.to_s
   e=Time.now.day.to_s
   f=Time.now.year.to_s
   cdown=cdown.map {|item|
   item='http://ichart.finance.yahoo.com/table.csv?s=‘+item.to_s+’&amp;a=‘+a+’&amp;b=‘+b+’&amp;c='+c+\
          '&amp;d='+d+'&amp;e='+e+'&amp;f='+f+'&amp;g=d&amp;ignore=.csv'}
   cdown.each.with_index {|item,idx|
     cthreaddown<< item
     i=i+1
     if i==100 or idx==cdown.size-1 then
       cthreaddown=cthreaddown.uniq
       for page_to_fetch in cthreaddown
       threads<<Thread.new(page_to_fetch) do |web|
          datafile="/home/pt/usa/stock/"+web.split("=")[1].to_s.gsub("&amp;a","")
          begin
            open(datafile,'a+'){|refile|
            open(web){|webfile|
               refile.puts webfile.read
               puts "#{web} over" }}
          rescue Timeout::Error => e
            puts "#{web} failed"
          rescue NameError => e
            puts "#{web} failed"
          rescue => e
            puts "#{web} wrong"
          end
        end
       end
       threads.each {|thr|
         begin
           thr.join
         rescue Timeout::Error => e
           puts "#{web} failed"
         rescue NameError => e
           puts "#{web} failed"
         rescue => e
           puts "#{web} failed"
         end }
        i=0
        cthreaddown=
        threads =
        sleep 0.01
     else
     end
     }
   end
  end

i have opened 100 threads. there are 6000 companies in
/home/pt/usa/stocklist .

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/