Problem with concurrency

Hi guys,

I'm having an issue with a multi-threaded system I'm developing.

Sorry if my question sounds too generic and unspecific but:
Are there any known pitfalls that one might come across when using
threads?

This is the code that runs the threads:
(I'm new to this so the code is borrowed from another post from this
list.)

···

---------
    def run_mods( page_data, structure )

        mod_queue = Queue.new

        @threads = ( 1..@opts[:threads] ).map {
            >i>
            Thread.new( mod_queue ) {
                >q>
                until( q == ( curr_mod = q.deq ) )

                    if( !run_module?( curr_mod , structure ) )
                        print_verbose( 'Skipping ' + curr_mod.to_s +
                            ', nothing to audit.' )
                        next
                    end

                    print_debug( )
                    print_debug( 'Thread-' + i.to_s + " " +
curr_mod.inspect )
                    print_debug( )

                    print_status( curr_mod.to_s )

                    mod_new = curr_mod.new( page_data, structure )

                    mod_new.prepare if curr_mod.method_defined?(
'prepare' )
                    mod_new.run
                    mod_new.clean_up if curr_mod.method_defined?(
'clean_up' )

                    while( handle_interrupt( ) )
                    end

                end
            }
        }

        # enque the loaded mods
        for mod in ls_loaded_mods
            mod_queue.enq mod
        end

        @threads.size.times { mod_queue.enq mod_queue }
        @threads.each { |t| t.join }

    end
-----------------

The modules call their parent and the parent uses Net::HTTP.
Then they get the HTTP response via the parent and manipulate it in some
way.
My main problem is that from what I've gathered some HTTP calls fail
without an error.

Some modules just don't produce any results when the thread count is set
higher than 1.

I normally wouldn't bother you with something so broad but I've been
debugging this for 24 hours straight so I figured that someone else
might have run to a similar problem and save me 24 more hours of
debugging.

Has anyone else had a similar experience?
--
Posted via http://www.ruby-forum.com/.

Try this at the top of your program:

Thread.abort_on_exception = true

Then you will probably see errors.

Kind regards

  robert

···

On 07/29/2010 07:55 AM, Tasos Laskos wrote:

Hi guys,

I'm having an issue with a multi-threaded system I'm developing.

Sorry if my question sounds too generic and unspecific but:
Are there any known pitfalls that one might come across when using
threads?

This is the code that runs the threads:
(I'm new to this so the code is borrowed from another post from this
list.)
---------
     def run_mods( page_data, structure )

         mod_queue = Queue.new

         @threads = ( 1..@opts[:threads] ).map {
             >i>
             Thread.new( mod_queue ) {
                 >q>
                 until( q == ( curr_mod = q.deq ) )

                     if( !run_module?( curr_mod , structure ) )
                         print_verbose( 'Skipping ' + curr_mod.to_s +
                             ', nothing to audit.' )
                         next
                     end

                     print_debug( )
                     print_debug( 'Thread-' + i.to_s + " " +
curr_mod.inspect )
                     print_debug( )

                     print_status( curr_mod.to_s )

                     mod_new = curr_mod.new( page_data, structure )

                     mod_new.prepare if curr_mod.method_defined?(
'prepare' )
                     mod_new.run
                     mod_new.clean_up if curr_mod.method_defined?(
'clean_up' )

                     while( handle_interrupt( ) )
                     end

                 end
             }
         }

         # enque the loaded mods
         for mod in ls_loaded_mods
             mod_queue.enq mod
         end

         @threads.size.times { mod_queue.enq mod_queue }
         @threads.each { |t| t.join }

     end
-----------------

The modules call their parent and the parent uses Net::HTTP.
Then they get the HTTP response via the parent and manipulate it in some
way.
My main problem is that from what I've gathered some HTTP calls fail
without an error.

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

Robert Klemme wrote:

···

On 07/29/2010 07:55 AM, Tasos Laskos wrote:

list.)

                     while( handle_interrupt( ) )

without an error.

Try this at the top of your program:

Thread.abort_on_exception = true

Then you will probably see errors.

Kind regards

  robert

Hi Robert,

Thanks for the reply, however I didn't see any errors.

This is going to be a bitch to debug...
--
Posted via http://www.ruby-forum.com/\.

Tasos Laskos wrote:

Robert Klemme wrote:

list.)

                     while( handle_interrupt( ) )

without an error.

Try this at the top of your program:

Thread.abort_on_exception = true

Then you will probably see errors.

Kind regards

  robert

Hi Robert,

Thanks for the reply, however I didn't see any errors.

This is going to be a bitch to debug...

Huh...it seems that the problem was actually caused by the localhost
webserver.
My laptop had a crazy uptime and all its memory (virtual and ram) had
been somewhat exhausted.

After a reboot everything started to work fine.

I can't believe all those hours of debugging went to waist.

···

On 07/29/2010 07:55 AM, Tasos Laskos wrote:

--
Posted via http://www.ruby-forum.com/\.

If it can happen once chances are that it will happen again. It's probably worth to dig a bit and try to find out what caused the memory issue in the first place. Maybe you have a leaky application. What OS btw?

Kind regards

  robert

···

On 07/29/2010 10:16 AM, Tasos Laskos wrote:

Tasos Laskos wrote:

Robert Klemme wrote:

On 07/29/2010 07:55 AM, Tasos Laskos wrote:

list.)

                      while( handle_interrupt( ) )

without an error.

Try this at the top of your program:

Thread.abort_on_exception = true

Then you will probably see errors.

Thanks for the reply, however I didn't see any errors.

This is going to be a bitch to debug...

Huh...it seems that the problem was actually caused by the localhost
webserver.
My laptop had a crazy uptime and all its memory (virtual and ram) had
been somewhat exhausted.

After a reboot everything started to work fine.

I can't believe all those hours of debugging went to waist.

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

Robert Klemme wrote:

Thread.abort_on_exception = true

Then you will probably see errors.

I can't believe all those hours of debugging went to waist.

If it can happen once chances are that it will happen again. It's
probably worth to dig a bit and try to find out what caused the memory
issue in the first place. Maybe you have a leaky application. What OS
btw?

Kind regards

  robert

Well, the laptop was on for 2-3 days constantly running fullscreen flash
video, eclipse, apache2 & mysql ( being constantly hammered by Arachni
due to the debugging itself) and around 13 tabs of Firefox spread over
several windows.

So in hindsight it's not much of a surprise.

I'm running the latest Kubuntu with Xfce, so I guess it has now become
Xubuntu...

···

On 07/29/2010 10:16 AM, Tasos Laskos wrote:

--
Posted via http://www.ruby-forum.com/\.