Rb_yield(), semaphores and 'break'

I've run into a problem that sounds superficially similar to the
one that Noah Easterly posted around a month ago, but I don't think
the suggested solutions work for me.

Here's the sitch... I'm working on an extension to handle real-time
MIDI, and my concept is to have an 'each' method for the MidiIn class
that will just feed events to a block as they arrive. This is implemented
by the 'each' (C++) function making the block available to a callback
function, which is itself called by the Midi handler for each arriving
event, and does a rb_yield(). The original function is meanwhile just
sitting on a semaphore, and will return when that is released.

This all works fine, as the semaphore can be released either by a
particular yield return value or by a timeout, *unless* there is
a 'break' in the invoked block. The base problem is that -- although
I can catch the break with an 'ensure' for instance -- the top-level
'each' is still sitting on the semaphore until that is released,
but as the break has *already* popped the stack, when that function
*does* terminate it just segfaults!

What I seem to need to do is to prevent the break bypassing the
normal return, but I can't see any way to do that. Otherwise I guess
I just have to put a notice in large letters: "DO NOT USE BREAK!" (:-/)

          -- Pete --

···

--

The address in the header is a Spam Bucket -- don't bother replying to it...
(If you do need to email, replace the account name with my true name.)

Hi,

At Wed, 16 May 2007 11:25:05 +0900,
Pete wrote in [ruby-talk:251756]:

Here's the sitch... I'm working on an extension to handle real-time
MIDI, and my concept is to have an 'each' method for the MidiIn class
that will just feed events to a block as they arrive. This is implemented
by the 'each' (C++) function making the block available to a callback
function, which is itself called by the Midi handler for each arriving
event, and does a rb_yield(). The original function is meanwhile just
sitting on a semaphore, and will return when that is released.

What semaphore? ruby's or C++'s?

This all works fine, as the semaphore can be released either by a
particular yield return value or by a timeout, *unless* there is
a 'break' in the invoked block. The base problem is that -- although
I can catch the break with an 'ensure' for instance -- the top-level
'each' is still sitting on the semaphore until that is released,
but as the break has *already* popped the stack, when that function
*does* terminate it just segfaults!

Have you tried rb_ensure()?

···

--
Nobu Nakada

In article <200705160246.l4G2kB3u024158@sharui.nakada.kanuma.tochigi.jp>,

Hi,

At Wed, 16 May 2007 11:25:05 +0900,
Pete wrote in [ruby-talk:251756]:

Here's the sitch... I'm working on an extension to handle real-time
MIDI, and my concept is to have an 'each' method for the MidiIn class
that will just feed events to a block as they arrive. This is implemented
by the 'each' (C++) function making the block available to a callback
function, which is itself called by the Midi handler for each arriving
event, and does a rb_yield(). The original function is meanwhile just
sitting on a semaphore, and will return when that is released.

What semaphore? ruby's or C++'s?

Sorry -- too brief... This is an OS (BeOS! (:-/)) semaphore (C++ level).

Have you tried rb_ensure()?

Yes. That's what I was referring to. I can 'capture' the break
with a function pointed to by an rb_ensure call, but the problem
is that the 'body' part of that call is *still* sitting on the
semaphore, so that if I release it from the ensure section, it
is too late -- the stack is already popped. It tries to return
at that point, and Boom! At least that's what seems to be happening.
Unless someone can correct me...

        -- Pete --

···

Nobuyoshi Nakada <nobu@ruby-lang.org> wrote:

--

The address in the header is a Spam Bucket -- don't bother replying to it...
(If you do need to email, replace the account name with my true name.)

Hi,

At Wed, 16 May 2007 12:05:04 +0900,
Pete wrote in [ruby-talk:251758]:

>Have you tried rb_ensure()?
Yes. That's what I was referring to. I can 'capture' the break
with a function pointed to by an rb_ensure call, but the problem
is that the 'body' part of that call is *still* sitting on the
semaphore, so that if I release it from the ensure section, it
is too late -- the stack is already popped. It tries to return
at that point, and Boom! At least that's what seems to be happening.
Unless someone can correct me...

Still I'm not sure about your situation. What do you mean by
"sitting" and "popped"? To where trying to "return"?

Can't you show simplified code?

···

--
Nobu Nakada

In article <200705160448.l4G4mPS3027878@sharui.nakada.kanuma.tochigi.jp>,

Hi,

At Wed, 16 May 2007 12:05:04 +0900,
Pete wrote in [ruby-talk:251758]:

>Have you tried rb_ensure()?
Yes. That's what I was referring to. I can 'capture' the break
with a function pointed to by an rb_ensure call, but the problem
is that the 'body' part of that call is *still* sitting on the
semaphore, so that if I release it from the ensure section, it
is too late -- the stack is already popped. It tries to return
at that point, and Boom! At least that's what seems to be happening.
Unless someone can correct me...

Still I'm not sure about your situation. What do you mean by
"sitting" and "popped"? To where trying to "return"?

Can't you show simplified code?

Guess I'm being too cryptic... (:-/) I'll try with a *very* stripped
down version of my code. Hopefully I haven't stripped out anything
relevant! Here's the skeleton:

void callback_from_handler(....){
  if (!rb_block_given_p() || !receiving || ...) return; // do nothing unless active
   VALUE bev = rb_class_new_instance(... cBMidiEvent); // event data copied to new obj
  VALUE res = rb_yield(bev); // pass event to code block
  if (RTEST(res)) { // not nil or false -- used as termination signal
    receiving = false;
    release_sem(); // tell 'do_each' to wake up
  }
}

VALUE bmidi_ensure(VALUE self) {
  if (receiving) { // only if not terminated normally (details omitted...)
    release_sem();
  }
  receiving = false;
  return self;
}

VALUE bmidi_doeach(VALUE self) {
   receiving = true;
   acquire_sem(); // sleep until woken (timeout etc omitted)
  return self;
}

VALUE bmidi_each(VALUE self) {
  // code block passed to this -- handled by ruby
  rb_ensure(RUBY_METHOD_FUNC(bmidi_doeach), self,
     RUBY_METHOD_FUNC(bmidi_ensure), self);
   return self;
}

(Note that in fact the callback is a virtual member fuction of an OS class,
and that the Ruby class for 'bmidi' wraps this with a Data_Wrap_struct.
All that seems to work, so I've left it out. 'bmidi_each' is the only
method to be published to the class (in the unshown 'Init_bmidi()').)

Now my understanding of what happens is that bmidi_each gets invoked
from the ruby level (with an associated block that gets stashed as the
"current" one). The rb_ensure() within sets up some kind of linkage
to make sure the 'ensure' section will get called, and then calls
'bmidi_doeach'. This does little but mark things as active and then
goes to sleep on the semaphore. Once this has been done, events
arriving at the callback will get passed to the current block through
rb_yield.

In *normal* operation, either the semaphore will time out (mechanism
not shown) or yield will get some 'true' value back from the block
to signal termination. Either way, the semaphore is released, bmidi_doeach
returns, bmidi_ensure is invoked (but does nothing because the flag has
been cleared), and everything is fine.

On the other hand, if there is a 'break' in the block, bmidi_ensure
gets called at once, but bmidi_doeach is still hanging on the locked
semaphore. The semaphore has to be released (otherwise the function
would block forever), but as (I am assuming) the system is now no longer
expecting a return from that function there is an immediate segmentation
fault.

Maybe I have misunderstood how Ruby handles a break-return, but the
segfault is real! Hope that makes my difficulty clearer.

          -- Pete --

···

Nobuyoshi Nakada <nobu@ruby-lang.org> wrote:

--

============================================================================
The address in the header is a Spam Bucket -- don't bother replying to it...
(If you do need to email, replace the account name with my true name.)

One thought before I quit for the night...

Thinking a bit more closely, the callback is of course in the (OS) thread
of the midi-handler, not the main thread that's running Ruby. So when
a break occurs it must go back to *that* thread, not Ruby's. This doesn't
actually seem to cause any immediate chaos -- before I put in the ensure
code, things would continue on quite normally after the break (with its
return value being reported and so on), until the semaphore actually timed out.
At which point, again, boom!

I may just be up against fundamental limits here.

          -- Pete --

···

--

The address in the header is a Spam Bucket -- don't bother replying to it...
(If you do need to email, replace the account name with my true name.)

Hi,

At Wed, 16 May 2007 15:40:08 +0900,
Pete wrote in [ruby-talk:251765]:

void callback_from_handler(....){
  if (!rb_block_given_p() || !receiving || ...) return; // do nothing unless active
   VALUE bev = rb_class_new_instance(... cBMidiEvent); // event data copied to new obj
  VALUE res = rb_yield(bev); // pass event to code block
  if (RTEST(res)) { // not nil or false -- used as termination signal
    receiving = false;
    release_sem(); // tell 'do_each' to wake up
  }
}

From where is this function called? Unless it's called as a
ruby method, you can't use rb_yield() in it. You might assume
the block to bmidi_each() is accessible, it is never guaranteed
at all.

VALUE bmidi_ensure(VALUE self) {
  if (receiving) { // only if not terminated normally (details omitted...)
    release_sem();
  }
  receiving = false;
  return self;
}

VALUE bmidi_doeach(VALUE self) {
   receiving = true;
   acquire_sem(); // sleep until woken (timeout etc omitted)
  return self;
}

Since bmidi_ensure() won't get called until bmidi_doeach()
ends, the latter seems to success to acquire the semaphore only
when the callback released it. At that time, the callback sets
receiving to false before the release, so the bmidi_ensure()
leaves the semaphore locked which acquired by bmidi_doeach(), I
guess. In short, the sequence is wrong.

Anyway, you need to call rb_yield() in bmidi_each().

void
callback_from_handler(....)
{
    VALUE bev = rb_class_new_instance(... cBMidiEvent);
    acquire_sem();
    received_bmidievent = bev; // pass event to 'each'
    release_sem(); // tell 'each' to wake up
}

VALUE
bmidi_each(VALUE self)
{
    VALUE res;

    rb_need_block();
    do {
        acquire_sem();
        VALUE bev = received_bmidievent;
        release_sem();
        res = rb_yield(bev); // pass event to code block
    } while (RTEST(res));
    return self;
}

···

--
Nobu Nakada

In article <200705160759.l4G7x8Tr014065@sharui.nakada.kanuma.tochigi.jp>,

Hi,

At Wed, 16 May 2007 15:40:08 +0900,

=46rom where is this function called? Unless it's called as a
ruby method, you can't use rb_yield() in it. You might assume
the block to bmidi_each() is accessible, it is never guaranteed
at all.

Yes, I realize this now. [In fact I realized it during my last post,
saw the real solution, started to report that, then started thinking
it was *still* wrong, and erased it! On the way to bed it dawned on
me it was correct after all.]

I was being fooled a bit by the PickAxe 'jukebox' example, which uses
a callback from external code, but of course he assumes everything is
single-threaded, so that the callback *does* derive from a ruby call
somewhere back up the call chain.

Since bmidi_ensure() won't get called until bmidi_doeach()
ends, the latter seems to success to acquire the semaphore only
when the callback released it. At that time, the callback sets
receiving to false before the release, so the bmidi_ensure()
leaves the semaphore locked which acquired by bmidi_doeach(), I
guess. In short, the sequence is wrong.

Actually, no. I think that part is correct. The callback only
releases the semaphore *if* the yield signals termination. The
idea of the ensure was to release the semaphore if the callback hadn't
-- i.e. if break bypassed the yield.

Anyway, you need to call rb_yield() in bmidi_each().

*This* is the key. I was sort of thinking in the OS's framework,
not Ruby's. At first I didn't think I could have the yield there,
but I realize that I can rearrange the semaphoring to synchronize
the two (OS) threads, and keep everything nice and orderly.

Thanks for your help
        -- Pete --

···

Nobuyoshi Nakada <nobu@ruby-lang.org> wrote:

--

The address in the header is a Spam Bucket -- don't bother replying to it...
(If you do need to email, replace the account name with my true name.)

Hi,

At Thu, 17 May 2007 03:25:05 +0900,
Pete wrote in [ruby-talk:251828]:

>Since bmidi_ensure() won't get called until bmidi_doeach()
>ends, the latter seems to success to acquire the semaphore only
>when the callback released it. At that time, the callback sets
>receiving to false before the release, so the bmidi_ensure()
>leaves the semaphore locked which acquired by bmidi_doeach(), I
>guess. In short, the sequence is wrong.
Actually, no. I think that part is correct. The callback only
releases the semaphore *if* the yield signals termination.

But receiving is left false, and then bmidi_doeach() acquires
the semaphore before bmidi_ensure() is called.

The idea of the ensure was to release the semaphore if the callback hadn't
-- i.e. if break bypassed the yield.

The ensure won't called until the callback released the
semaphore because doeach waits it.

···

--
Nobu Nakada

In article <200705170222.l4H2M9rU020152@sharui.nakada.kanuma.tochigi.jp>,

At Thu, 17 May 2007 03:25:05 +0900,
Pete wrote in [ruby-talk:251828]:

The idea of the ensure was to release the semaphore if the callback hadn't
-- i.e. if break bypassed the yield.

The ensure won't called until the callback released the
semaphore because doeach waits it.

This was *not* what I observed. I think what was happening was that
'break' was being invoked from another OS thread, so the ensure got run
too. (I had printf telltales everywhere to show me which bits were
executing, and ensure got run even though doeach was still suspended.)
Rather a mess, in other words...

Anyhow, I've rewritten the code now to be a good single-threaded Ruby
citizen, and it's now working great!

          -- Pete --

···

Nobuyoshi Nakada <nobu@ruby-lang.org> wrote:

--

The address in the header is a Spam Bucket -- don't bother replying to it...
(If you do need to email, replace the account name with my true name.)

Hi,

At Thu, 17 May 2007 15:30:08 +0900,
Pete wrote in [ruby-talk:251886]:

>>The idea of the ensure was to release the semaphore if the callback hadn't
>> -- i.e. if break bypassed the yield.
>
>The ensure won't called until the callback released the
>semaphore because doeach waits it.
This was *not* what I observed. I think what was happening was that
'break' was being invoked from another OS thread, so the ensure got run
too. (I had printf telltales everywhere to show me which bits were
executing, and ensure got run even though doeach was still suspended.)
Rather a mess, in other words...

Sorry, I misread.

···

--
Nobu Nakada