Problem in Unit Testing Methods that start new threads

ruby-talk
1

I have a bit of doubt, in Unit Testing Programs that start new threads.
Please have a look at the code below:

class Foobar

  def hello_world
    p "Hello World"
    @thread_status = false
  end

  def new_thread_start
    Thread.new do
      sleep(100)
      @thread_status = true
    end
  end

end

# here goes the lame test case
require "test/unit"

module Test::Unit::Assertions
  def assert_false(t_object,message=nil)
    boolean = !t_object
    full_message = build_message(message,'<?> Object is not
false',boolean)
    assert_block(full_message) { boolean}
  end
end

class TestFoobar < Test::Unit::TestCase
  def setup
    @foo = Foobar.new
    class << @foo
      def ivar var
        instance_variable_get(:"@#{var}")
      end
    end
  end

  def test_hello_world
    @foo.hello_world
    assert_false @foo.ivar(:thread_status)
  end

  def test_new_thread
    # sorry for a bit of not so DRY thingy
    @foo.hello_world
    assert_false @foo.ivar(:thread_status)
    @foo.new_thread_start

    # next assert is true because method was started in a new thread
    # and control came back immediately, what i would probably want is
    # to wait here so that i can have proper check on the method and
    # state of the program, but i am not sure, if that's exactly
    # approach i should take.

    assert_false @foo.ivar(:thread_status)
  end
end

Now, as someone suggested on IRC, I can do a join and wait for the
thread to finish. But the problem is, I don't exactly have an instance
to the thread, because its managed by a plugin and i am just using the
plugin to do stuff.

Any ideas/suggestions are more than welcome.

gnufied

2

Or is it this a good idea?

Thread.list.each { |t| t.join }

···

On Wed, 2007-01-03 at 22:30 +0900, Hemant Kumar wrote:

I have a bit of doubt, in Unit Testing Programs that start new threads.
Please have a look at the code below:

class Foobar

  def hello_world
    p "Hello World"
    @thread_status = false
  end

  def new_thread_start
    Thread.new do
      sleep(100)
      @thread_status = true
    end
  end

end

# here goes the lame test case
require "test/unit"

module Test::Unit::Assertions
  def assert_false(t_object,message=nil)
    boolean = !t_object
    full_message = build_message(message,'<?> Object is not
false',boolean)
    assert_block(full_message) { boolean}
  end
end

class TestFoobar < Test::Unit::TestCase
  def setup
    @foo = Foobar.new
    class << @foo
      def ivar var
        instance_variable_get(:"@#{var}")
      end
    end
  end

  def test_hello_world
    @foo.hello_world
    assert_false @foo.ivar(:thread_status)
  end

  def test_new_thread
    # sorry for a bit of not so DRY thingy
    @foo.hello_world
    assert_false @foo.ivar(:thread_status)
    @foo.new_thread_start

    # next assert is true because method was started in a new thread
    # and control came back immediately, what i would probably want is
    # to wait here so that i can have proper check on the method and
    # state of the program, but i am not sure, if that's exactly
    # approach i should take.

    assert_false @foo.ivar(:thread_status)
  end
end

Now, as someone suggested on IRC, I can do a join and wait for the
thread to finish. But the problem is, I don't exactly have an instance
to the thread, because its managed by a plugin and i am just using the
plugin to do stuff.

Any ideas/suggestions are more than welcome.

3

it seems that all you've managed to do is write a very long race condition.
i'm not one of those people who think the mere mention of the word
'unit-testing' bestows any sort of robustness on code. for example, the
testing of 'thread_status' in your test means nothing, as it's the source of
the race condition: you need to wrap setting and reading this var with a
semaphore. this shows why:

   harp:~ > cat a.rb
   class C
     attr :thread
     def initialize
       @thread = nil
     end
     def new_thread
       Thread.new{ @thread = Thread.current }
     end
   end

   4242.times{|i| raise "race condition @ loopno #{ i }!" unless((c = C.new) and (Thread === c.new_thread) and c.thread) }

   harp:~ > ruby a.rb
   a.rb:11: race condition @ loopno 942! (RuntimeError)
           from a.rb:11

regarding your specific question though, you need to verify that a thread is
created and that it's status is running. even if you don't have a handle on
the thread you can set things up in your unit test to get one. something like

   harp:~ > cat a.rb
   def tracking_threads &b
     before = Thread.list
     yield
     after = Thread.list
     return after - before
   end

   threads = tracking_threads{ 2.times{ Thread.new{ sleep } } }

   threads.each{|t| p t.status}

   harp:~ > ruby a.rb
   "sleep"

cool. now we've managed to shows that Thread.new works! :wink:

i wouldn't bother with this at all unless i could also test that the created
thread did the right thing.

my 2 cts.

kind regards.

-a

···

On Wed, 3 Jan 2007, Hemant Kumar wrote:

I have a bit of doubt, in Unit Testing Programs that start new threads.
Please have a look at the code below:

class Foobar

def hello_world
   p "Hello World"
   @thread_status = false
end

def new_thread_start
   Thread.new do
     sleep(100)
     @thread_status = true
   end
end

end

# here goes the lame test case
require "test/unit"

module Test::Unit::Assertions
def assert_false(t_object,message=nil)
   boolean = !t_object
   full_message = build_message(message,'<?> Object is not
false',boolean)
   assert_block(full_message) { boolean}
end
end

class TestFoobar < Test::Unit::TestCase
def setup
   @foo = Foobar.new
   class << @foo
     def ivar var
       instance_variable_get(:"@#{var}")
     end
   end
end

def test_hello_world
   @foo.hello_world
   assert_false @foo.ivar(:thread_status)
end

def test_new_thread
   # sorry for a bit of not so DRY thingy
   @foo.hello_world
   assert_false @foo.ivar(:thread_status)
   @foo.new_thread_start

   # next assert is true because method was started in a new thread
   # and control came back immediately, what i would probably want is
   # to wait here so that i can have proper check on the method and
   # state of the program, but i am not sure, if that's exactly
   # approach i should take.

   assert_false @foo.ivar(:thread_status)
end
end

Now, as someone suggested on IRC, I can do a join and wait for the
thread to finish. But the problem is, I don't exactly have an instance
to the thread, because its managed by a plugin and i am just using the
plugin to do stuff.

Any ideas/suggestions are more than welcome.

--
if you find yourself slandering anybody, first imagine that your mouth is
filled with excrement. it will break you of the habit quickly enough. - the
dalai lama

4

Ara's right on this one. It's a race condition. But it looks like you're interested in checking that the thread has indeed started. If I were to write something like this I would probably have the unit test run @foo.new_thread_start and immediately sleep waiting for a signal from the child thread. The child thread would then signal back to the any waiting threads that it had set it's thread status to true. Altough the more I ponder it, the more I feel like I've just moved the race condition to the sleep call. I'll have to review my concurrent programming texts a bit. Seems like you should do the same.

Either way, check out Monitor for some insight:
http://www.ruby-doc.org/stdlib/libdoc/monitor/rdoc/index.html

-Mat

···

On Jan 3, 2007, at 8:30 AM, Hemant Kumar wrote:

I have a bit of doubt, in Unit Testing Programs that start new threads.
Please have a look at the code below:

class Foobar

  def hello_world
    p "Hello World"
    @thread_status = false
  end

  def new_thread_start
    Thread.new do
      sleep(100)
      @thread_status = true
    end
  end

end

# here goes the lame test case
require "test/unit"

module Test::Unit::Assertions
  def assert_false(t_object,message=nil)
    boolean = !t_object
    full_message = build_message(message,'<?> Object is not
false',boolean)
    assert_block(full_message) { boolean}
  end
end

class TestFoobar < Test::Unit::TestCase
  def setup
    @foo = Foobar.new
    class << @foo
      def ivar var
        instance_variable_get(:"@#{var}")
      end
    end
  end

  def test_hello_world
    @foo.hello_world
    assert_false @foo.ivar(:thread_status)
  end

  def test_new_thread
    # sorry for a bit of not so DRY thingy
    @foo.hello_world
    assert_false @foo.ivar(:thread_status)
    @foo.new_thread_start

    # next assert is true because method was started in a new thread
    # and control came back immediately, what i would probably want is
    # to wait here so that i can have proper check on the method and
    # state of the program, but i am not sure, if that's exactly
    # approach i should take.

    assert_false @foo.ivar(:thread_status)
  end
end

Now, as someone suggested on IRC, I can do a join and wait for the
thread to finish. But the problem is, I don't exactly have an instance
to the thread, because its managed by a plugin and i am just using the
plugin to do stuff.

Any ideas/suggestions are more than welcome.

5

Hemant Kumar schrieb:

I have a bit of doubt, in Unit Testing Programs that start new threads.
Please have a look at the code below:
(...)

Hemant, in addition to what the others said, it's not clear to me what you really want to test: whether a new thread is started, whether the instance variable changed after a certain amount of time, whether the instance variable changed at the end of the new thread, ...

I'm a fan of black box unit tests, so for me, the tests should specify what you expect from the *interface* of the object under test. I wouldn't test for values of certain instance variables or for threads created internally, unless those are part of the desired interface of your objects.

Regards,
Pit

6

good idea mat. it's easy in ruby too:

   require 'thread'

   q = Queue.new

   Thread.new{
     q.push :running
     # run
   }

   q.pop # wait for thread to start...

regards.

-a

···

On Thu, 4 Jan 2007, Mat Schaffer wrote:

Ara's right on this one. It's a race condition. But it looks like you're interested in checking that the thread has indeed started. If I were to write something like this I would probably have the unit test run @foo.new_thread_start and immediately sleep waiting for a signal from the child thread. The child thread would then signal back to the any waiting threads that it had set it's thread status to true. Altough the more I ponder it, the more I feel like I've just moved the race condition to the sleep call. I'll have to review my concurrent programming texts a bit. Seems like you should do the same.

Either way, check out Monitor for some insight:
http://www.ruby-doc.org/stdlib/libdoc/monitor/rdoc/index.html

-Mat

--
if you find yourself slandering anybody, first imagine that your mouth is
filled with excrement. it will break you of the habit quickly enough. - the
dalai lama

7

> Now, as someone suggested on IRC, I can do a join and wait for the
> thread to finish. But the problem is, I don't exactly have an instance
> to the thread, because its managed by a plugin and i am just using the
> plugin to do stuff.
>
> Any ideas/suggestions are more than welcome.

it seems that all you've managed to do is write a very long race condition.
i'm not one of those people who think the mere mention of the word
'unit-testing' bestows any sort of robustness on code. for example, the
testing of 'thread_status' in your test means nothing, as it's the source of
the race condition: you need to wrap setting and reading this var with a
semaphore. this shows why:

   harp:~ > cat a.rb
   class C
     attr :thread
     def initialize
       @thread = nil
     end
     def new_thread
       Thread.new{ @thread = Thread.current }
     end
   end

   4242.times{|i| raise "race condition @ loopno #{ i }!" unless((c = C.new) and (Thread === c.new_thread) and c.thread) }

   harp:~ > ruby a.rb
   a.rb:11: race condition @ loopno 942! (RuntimeError)
           from a.rb:11

Thanks for the insight Ara, although my class is singleton so i
shouldn't get the problem you have described above.

However can you show me a code sample, that doesn't cause race condition
in above code. I tried using Mutex on above code,

require "thread"
class C
  attr :thread
  def initialize
    mutex = Mutex.new
    mutex.lock
    @thread = nil
    mutex.unlock
  end

  def new_thread
    Thread.new{
      mutex = Mutex.new
      mutex.lock
      @thread = Thread.current
      mutex.unlock
    }
  end

end

4242.times{|i| raise "race condition @ loopno #{ i }!" unless((c =
C.new) and (Thread === c.new_thread) and c.thread) }

And I am still getting a race condition. Using MonitorMixin to signal
execution seems like an overkill to me in above code.

So can you show me a way of making above code not race.

regarding your specific question though, you need to verify that a thread is
created and that it's status is running. even if you don't have a handle on
the thread you can set things up in your unit test to get one. something like

   harp:~ > cat a.rb
   def tracking_threads &b
     before = Thread.list
     yield
     after = Thread.list
     return after - before
   end

   threads = tracking_threads{ 2.times{ Thread.new{ sleep } } }

   threads.each{|t| p t.status}

   harp:~ > ruby a.rb
   "sleep"
   "sleep"

Cool thanks.

···

On Thu, 2007-01-04 at 01:04 +0900, ara.t.howard@noaa.gov wrote:

8

Agreed Pit, but basically I want to test state of my program through
Unit Tests and whether each method modifies state of program as it was
intended?

Now, unit testing methods that return something on invocation is easy
and i don't need to go around running asserts on instance variables.

But I have some doubt regarding testing methods that don't return
anything explicitly and rather update instance variables.

Earlier, I was writing Unit Tests for a networking application which i
wrote using EventMachine. Now since, EM is completely based on
callbacks, you can't do a check on return values of methods. Hence I had
to rely on doing asserts on instance variables.

I would love to know, how do i go about unit testing in such cases. I
can't yet grasp concept of code that can be unit tested easily i guess.

gnufied

···

On Thu, 2007-01-04 at 06:57 +0900, Pit Capitain wrote:

Hemant Kumar schrieb:
> I have a bit of doubt, in Unit Testing Programs that start new threads.
> Please have a look at the code below:
> (...)

Hemant, in addition to what the others said, it's not clear to me what
you really want to test: whether a new thread is started, whether the
instance variable changed after a certain amount of time, whether the
instance variable changed at the end of the new thread, ...

I'm a fan of black box unit tests, so for me, the tests should specify
what you expect from the *interface* of the object under test. I
wouldn't test for values of certain instance variables or for threads
created internally, unless those are part of the desired interface of
your objects.

Regards,
Pit

9

Hemant Kumar wrote:

> > Now, as someone suggested on IRC, I can do a join and wait for the
> > thread to finish. But the problem is, I don't exactly have an instance
> > to the thread, because its managed by a plugin and i am just using the
> > plugin to do stuff.
> >
> > Any ideas/suggestions are more than welcome.
>
> it seems that all you've managed to do is write a very long race condition.
> i'm not one of those people who think the mere mention of the word
> 'unit-testing' bestows any sort of robustness on code. for example, the
> testing of 'thread_status' in your test means nothing, as it's the source of
> the race condition: you need to wrap setting and reading this var with a
> semaphore. this shows why:
>
> harp:~ > cat a.rb
> class C
> attr :thread
> def initialize
> @thread = nil
> end
> def new_thread
> Thread.new{ @thread = Thread.current }
> end
> end
>
> 4242.times{|i| raise "race condition @ loopno #{ i }!" unless((c = C.new) and (Thread === c.new_thread) and c.thread) }
>
>
> harp:~ > ruby a.rb
> a.rb:11: race condition @ loopno 942! (RuntimeError)
> from a.rb:11
>
>

Thanks for the insight Ara, although my class is singleton so i
shouldn't get the problem you have described above.

However can you show me a code sample, that doesn't cause race condition
in above code. I tried using Mutex on above code,

require "thread"
class C
  attr :thread
  def initialize
    mutex = Mutex.new
    mutex.lock
    @thread = nil
    mutex.unlock
  end

  def new_thread
    Thread.new{
      mutex = Mutex.new
      mutex.lock
      @thread = Thread.current
      mutex.unlock
    }
  end

end

4242.times{|i| raise "race condition @ loopno #{ i }!" unless((c =
C.new) and (Thread === c.new_thread) and c.thread) }

And I am still getting a race condition. Using MonitorMixin to signal
execution seems like an overkill to me in above code.

So can you show me a way of making above code not race.

> regarding your specific question though, you need to verify that a thread is
> created and that it's status is running. even if you don't have a handle on
> the thread you can set things up in your unit test to get one. something like
>
> harp:~ > cat a.rb
> def tracking_threads &b
> before = Thread.list
> yield
> after = Thread.list
> return after - before
> end
>
> threads = tracking_threads{ 2.times{ Thread.new{ sleep } } }
>
> threads.each{|t| p t.status}
>
>
> harp:~ > ruby a.rb
> "sleep"
> "sleep"
>
>

Cool thanks.

Try removing the "mutex = Mutex.new" from "new_thread" That's causing a
thread local mutex to be created that's only being used by that thread
and none of the others, because they have their own mutex. The threads
need to share the SAME mutex in order to enable mutual exclusion
between them, ie, the mutex you create in initialize(). That said, us
better understanding what you actually want to unit test would help us
to help you.

Cheers,

Ken

···

On Thu, 2007-01-04 at 01:04 +0900, ara.t.howard@noaa.gov wrote:

10

often methods should do one of two things:

   - succeed or throw an exception

   - return a value indicating success

in your case the former should be sufficient - you can assert that nothing is
raised and move on. if that's not sufficient consider changing the way your
method works: testing internal state is, at least, going to make maintaining
your tests very very hard since they're so cozy with your impl.

cheers.

-a

···

On Thu, 4 Jan 2007, Hemant Kumar wrote:

Agreed Pit, but basically I want to test state of my program through
Unit Tests and whether each method modifies state of program as it was
intended?

Now, unit testing methods that return something on invocation is easy
and i don't need to go around running asserts on instance variables.

But I have some doubt regarding testing methods that don't return
anything explicitly and rather update instance variables.

Earlier, I was writing Unit Tests for a networking application which i
wrote using EventMachine. Now since, EM is completely based on
callbacks, you can't do a check on return values of methods. Hence I had
to rely on doing asserts on instance variables.

I would love to know, how do i go about unit testing in such cases. I
can't yet grasp concept of code that can be unit tested easily i guess.

--
if you find yourself slandering anybody, first imagine that your mouth is
filled with excrement. it will break you of the habit quickly enough. - the
dalai lama

11

Hemant Kumar schrieb:

(...)
Earlier, I was writing Unit Tests for a networking application which i
wrote using EventMachine. Now since, EM is completely based on
callbacks, you can't do a check on return values of methods. Hence I had
to rely on doing asserts on instance variables.

I would love to know, how do i go about unit testing in such cases. I
can't yet grasp concept of code that can be unit tested easily i guess.

Hemant, in this case I'd try to use a framework like Mocha to create a mock for EventMachine. Then you can invoke the callbacks from the test code and verify that your application reacts as expected. I'm sure the results of invoking the callbacks can be examined from the outside, without having to look at instance variables.

Regards,
Pit

12

Since We are calling new each time, even using instance variables won't
solve the race condition. I have even tried using class variables, and I
am still getting race condition.

Ara, please demonstrate how you would solve the race condition in above
shown code of yours?

···

On Thu, 2007-01-04 at 07:15 +0900, Kenosis wrote:

Hemant Kumar wrote:
> On Thu, 2007-01-04 at 01:04 +0900, ara.t.howard@noaa.gov wrote:
> > > Now, as someone suggested on IRC, I can do a join and wait for the
> > > thread to finish. But the problem is, I don't exactly have an instance
> > > to the thread, because its managed by a plugin and i am just using the
> > > plugin to do stuff.
> > >
> > > Any ideas/suggestions are more than welcome.
> >
> > it seems that all you've managed to do is write a very long race condition.
> > i'm not one of those people who think the mere mention of the word
> > 'unit-testing' bestows any sort of robustness on code. for example, the
> > testing of 'thread_status' in your test means nothing, as it's the source of
> > the race condition: you need to wrap setting and reading this var with a
> > semaphore. this shows why:
> >
> > harp:~ > cat a.rb
> > class C
> > attr :thread
> > def initialize
> > @thread = nil
> > end
> > def new_thread
> > Thread.new{ @thread = Thread.current }
> > end
> > end
> >
> > 4242.times{|i| raise "race condition @ loopno #{ i }!" unless((c = C.new) and (Thread === c.new_thread) and c.thread) }
> >
> >
> > harp:~ > ruby a.rb
> > a.rb:11: race condition @ loopno 942! (RuntimeError)
> > from a.rb:11
> >
> >
>
> Thanks for the insight Ara, although my class is singleton so i
> shouldn't get the problem you have described above.
>
> However can you show me a code sample, that doesn't cause race condition
> in above code. I tried using Mutex on above code,
>
> require "thread"
> class C
> attr :thread
> def initialize
> mutex = Mutex.new
> mutex.lock
> @thread = nil
> mutex.unlock
> end
>
> def new_thread
> Thread.new{
> mutex = Mutex.new
> mutex.lock
> @thread = Thread.current
> mutex.unlock
> }
> end
>
> end
>
> 4242.times{|i| raise "race condition @ loopno #{ i }!" unless((c =
> C.new) and (Thread === c.new_thread) and c.thread) }
>
> And I am still getting a race condition. Using MonitorMixin to signal
> execution seems like an overkill to me in above code.
>
> So can you show me a way of making above code not race.
>
>
> > regarding your specific question though, you need to verify that a thread is
> > created and that it's status is running. even if you don't have a handle on
> > the thread you can set things up in your unit test to get one. something like
> >
> > harp:~ > cat a.rb
> > def tracking_threads &b
> > before = Thread.list
> > yield
> > after = Thread.list
> > return after - before
> > end
> >
> > threads = tracking_threads{ 2.times{ Thread.new{ sleep } } }
> >
> > threads.each{|t| p t.status}
> >
> >
> > harp:~ > ruby a.rb
> > "sleep"
> > "sleep"
> >
> >
>
> Cool thanks.

Try removing the "mutex = Mutex.new" from "new_thread" That's causing a
thread local mutex to be created that's only being used by that thread
and none of the others, because they have their own mutex. The threads
need to share the SAME mutex in order to enable mutual exclusion
between them, ie, the mutex you create in initialize(). That said, us
better understanding what you actually want to unit test would help us
to help you.

Cheers,

Ken

13

hi hemant-

without thinking too hard about it i'd probably do something like this:

   harp:~ > cat a.rb
   require 'sync'
   require 'thread'

   class Module
     def tattr a
       module_eval <<-code
         def #{ a }= arg
           synchronize(:EX){ @#{ a } = arg }
         end
         def #{ a }
           synchronize(:SH){ @#{ a } }
         end
       code
     end
   end

   class C
     include Sync_m

     tattr :thread

     def initialize
       @thread = nil
       sync_initialize
     end

     def new_thread
       q = Queue.new
       Thread.new{ q.push( self.thread = Thread.current ) }
     ensure
       q.pop
     end
   end

   4242.times{|i| raise "race @ loop #{ i } condition!" unless((c = C.new) and (Thread === c.new_thread) and c.thread) }

   p 42

   harp:~ > ruby a.rb
   42

   harp:~ > ruby a.rb
   42

   harp:~ > ruby a.rb
   42

   harp:~ > ruby a.rb
   42

the methodology of waiting for the 'q.push' is useful so your method only
returns after the thread has 'started' running.

kind regards.

-a

···

On Thu, 4 Jan 2007, Hemant Kumar wrote:

Since We are calling new each time, even using instance variables won't
solve the race condition. I have even tried using class variables, and I am
still getting race condition.

Ara, please demonstrate how you would solve the race condition in above
shown code of yours?

--
if you find yourself slandering anybody, first imagine that your mouth is
filled with excrement. it will break you of the habit quickly enough. - the
dalai lama