Embed Ruby or Lua

ruby-talk
1

I am writing a real-time data processing tool (data in ---> pattern
recognition ---> midi out) called rubyk (site: http://rubyk.org trac:
http://dev.rubyk.org). The first idea was to embed ruby (thus the name
rubyk).

I did some testing with a first "pure ruby" implementation of the
signal sending mechanism and it works fine but is erratic (very hard
to have stable tempo), probably due to ruby's GC messing in the way.

So I wrote a C++ core and thought I would embed ruby. But then the
garbage collector's "inconvenient pause" would happen again (with less
objects though).

Will ruby work fine for real-time midi ? Or am I better off using Lua
? I really like ruby and would prefer going this route, but I do not
want to be limited to a "rubato" style of music.

So the questions are:

1. I need to loop at a minimum of 300Hz. Will ruby GC be fast enough ?
2. Is there a way to disable GC without leaking memory ?
3. Would Lua be a better fit ?
4. Python uses reference counting so no "mark & sweep" pause. But I do
not want to use Python, so please do not tell me Python would be
great.

Many thanks for your answers.

Gaspard

2

Gaspard Bucher wrote:
--snip--

1. I need to loop at a minimum of 300Hz. Will ruby GC be fast enough ?
2. Is there a way to disable GC without leaking memory ?
3. Would Lua be a better fit ?
4. Python uses reference counting so no "mark & sweep" pause. But I do
not want to use Python, so please do not tell me Python would be
great.

Testing on old and slow computer would provide your answers.
If garbage collection disrupts tempo, it can be disabled
GC.disable
Enable and start between midi songs
GC.enable
GC.start
GC is not known to leak memory, AFAIK.

Gerald

3

Gaspard Bucher wrote:

I am writing a real-time data processing tool (data in ---> pattern
recognition ---> midi out) called rubyk (site: http://rubyk.org trac:
http://dev.rubyk.org). The first idea was to embed ruby (thus the name
rubyk).

I did some testing with a first "pure ruby" implementation of the
signal sending mechanism and it works fine but is erratic (very hard
to have stable tempo), probably due to ruby's GC messing in the way.

So I wrote a C++ core and thought I would embed ruby. But then the
garbage collector's "inconvenient pause" would happen again (with less
objects though).

Will ruby work fine for real-time midi ? Or am I better off using Lua
? I really like ruby and would prefer going this route, but I do not
want to be limited to a "rubato" style of music.

So the questions are:

1. I need to loop at a minimum of 300Hz. Will ruby GC be fast enough ?
2. Is there a way to disable GC without leaking memory ?
3. Would Lua be a better fit ?
4. Python uses reference counting so no "mark & sweep" pause. But I do
not want to use Python, so please do not tell me Python would be
great.

Many thanks for your answers.

Gaspard

I'm not quite sure *exactly* what it is you're trying to do here, but my rough guess is that it's real-time multimedia. Judging by some of the other things on the site -- support vector machines, BLAS, etc. -- I think you probably know enough to code the whole thing in C or C++, and if you can do so, that's what I'd recommend.

In fact, I'd recommend getting one of the Linux audio distributions, like JAD or Studio 64, and getting the real "hard real-time" (as hard real-time as Linux can get, anyhow) libraries up and running. I've mostly looked at Studio 64 because I have a 64-bit machine, but JAD is also good. Both are Debian Linux based and have special kernel patches and other software for low-latency audio processing.

Lua is probably a good bit faster than Ruby, but I don't know how much of the application has to be in the scripting language and how much in close-to-the-metal C/C++. I really don't think you want a garbage-collected language for this, although I'm sure it's possible to use one if the heavy lifting is done in a low-level library.

Speaking of low-level libraries, since you're looking at BLAS, LAPACK, support vector machines, etc., you might want to install Atlas (Automatically Tuned Linear Algebra Subroutines). They just released 3.8.0 and it's hand tuned to most modern chips. On my dual-core Athlon64 X2 2200+ I've gotten close to 10 GFLOPS on 32-bit operations!

4

There are, in fact, hard realtime garbage collection algorithms
available. For some reason language designers just don't seem to want
to implement them in their languages in favour of garbage collection
algorithms that are easier to write.

What I'd like to see someday is a language whose runtime explicitly
supports garbage-collection plug-ins so that you can use the appropriate
collection algorithm for the appropriate situation.

Going to the OP's question? Embed the Lua. Ruby is a great language
and great for embedding in applications, even, except in the specific
case of what you're talking about right now: anything with hard realtime
(and even soft realtime!) requirements. Lua is a much faster language
and has garbage collection that is far more easily tuned.

···

On Sun, 2007-21-10 at 15:19 +0900, M. Edward (Ed) Borasky wrote:

I really don't think you want a
garbage-collected language for this, although I'm sure it's possible to
use one if the heavy lifting is done in a low-level library.

--
Michael T. Richter <ttmrichter@gmail.com> (GoogleTalk:
ttmrichter@gmail.com)
I have to wonder why people think that when they can't manage local
personnel within easy strangling and shooting distance, then they can
manage personnel thousands of miles away that have different languages,
cultures, and business rules. (Joe Celko)

5

I'm not quite sure *exactly* what it is you're trying to do here, but my
rough guess is that it's real-time multimedia. Judging by some of the
other things on the site -- support vector machines, BLAS, etc. -- I
think you probably know enough to code the whole thing in C or C++, and
if you can do so, that's what I'd recommend.

All the core parts are coded in C. I need a scripting language for the
post-processing (simple low-pass filter, decision tree, instrument
change depending on song position, etc).

In fact, I'd recommend getting one of the Linux audio distributions,
like JAD or Studio 64, and getting the real "hard real-time" (as hard
real-time as Linux can get, anyhow) libraries up and running. I've
mostly looked at Studio 64 because I have a 64-bit machine, but JAD is
also good. Both are Debian Linux based and have special kernel patches
and other software for low-latency audio processing.

This would be great, but the machine has to include the
sound-generating application (I bought mac-minis). If the latency is
too bad, I might use Studio64 and send all the midi to a central
"audio-generation" machine.

Lua is probably a good bit faster than Ruby, but I don't know how much
of the application has to be in the scripting language and how much in
close-to-the-metal C/C++. I really don't think you want a
garbage-collected language for this, although I'm sure it's possible to
use one if the heavy lifting is done in a low-level library.

All long-time processing (SVM, PCA) will be done I C using the
Accelerate framework (fast BLAS and LAPACK tuned for Inter
processors).

Speaking of low-level libraries, since you're looking at BLAS, LAPACK,
support vector machines, etc., you might want to install Atlas
(Automatically Tuned Linear Algebra Subroutines). They just released
3.8.0 and it's hand tuned to most modern chips. On my dual-core Athlon64
X2 2200+ I've gotten close to 10 GFLOPS on 32-bit operations!

I do not know if there is a difference between Accelerate framework
and Atlas. I might install Atlas if I can gain some speed here.

Thanks for the reply, it seems like Lua will be better for the
post-processing I need to do.

Gaspard

6

Testing on old and slow computer would provide your answers.
If garbage collection disrupts tempo, it can be disabled
GC.disable
Enable and start between midi songs
GC.enable
GC.start
GC is not known to leak memory, AFAIK.

I wrote "leak" but I meant "grows", how much does the garbage grow if
it is not collected ? (I will test this to see).

Gerald

I will try this before moving to Lua. If it's fast enough and the
memory does not grow too much between garbage collections: Ruby. Else:
Lua.

Gaspard

7

Gaspard Bucher wrote:

I'm not quite sure *exactly* what it is you're trying to do here, but my
rough guess is that it's real-time multimedia. Judging by some of the
other things on the site -- support vector machines, BLAS, etc. -- I
think you probably know enough to code the whole thing in C or C++, and
if you can do so, that's what I'd recommend.

All the core parts are coded in C. I need a scripting language for the
post-processing (simple low-pass filter, decision tree, instrument
change depending on song position, etc).

In fact, I'd recommend getting one of the Linux audio distributions,
like JAD or Studio 64, and getting the real "hard real-time" (as hard
real-time as Linux can get, anyhow) libraries up and running. I've
mostly looked at Studio 64 because I have a 64-bit machine, but JAD is
also good. Both are Debian Linux based and have special kernel patches
and other software for low-latency audio processing.

This would be great, but the machine has to include the
sound-generating application (I bought mac-minis). If the latency is
too bad, I might use Studio64 and send all the midi to a central
"audio-generation" machine.

Lua is probably a good bit faster than Ruby, but I don't know how much
of the application has to be in the scripting language and how much in
close-to-the-metal C/C++. I really don't think you want a
garbage-collected language for this, although I'm sure it's possible to
use one if the heavy lifting is done in a low-level library.

All long-time processing (SVM, PCA) will be done I C using the
Accelerate framework (fast BLAS and LAPACK tuned for Inter
processors).

Speaking of low-level libraries, since you're looking at BLAS, LAPACK,
support vector machines, etc., you might want to install Atlas
(Automatically Tuned Linear Algebra Subroutines). They just released
3.8.0 and it's hand tuned to most modern chips. On my dual-core Athlon64
X2 2200+ I've gotten close to 10 GFLOPS on 32-bit operations!

I do not know if there is a difference between Accelerate framework
and Atlas. I might install Atlas if I can gain some speed here.

Thanks for the reply, it seems like Lua will be better for the
post-processing I need to do.

Gaspard

Ah ... Intel Mac Minis. I don't know Accelerate, but I know Clint Whaley and the rest of the Atlas people have wrung every possible gigaflop out of the recent Intel Core2 processors, so unless Accelerate has managed to find a cycle or two the Atlas team didn't spot, Atlas is probably the way to go.

What are you using to generate the audio? The built-in synthesizer in MacOS? I have absolutely no idea how the audio on a Mac works -- as you've probably guessed I'm a hard-core Linux geek.

8

Michael T. Richter wrote:

···

On Sun, 2007-21-10 at 15:19 +0900, M. Edward (Ed) Borasky wrote:

I really don't think you want a garbage-collected language for this, although I'm sure it's possible to use one if the heavy lifting is done in a low-level library.

There are, in fact, hard realtime garbage collection algorithms
available. For some reason language designers just don't seem to want
to implement them in their languages in favour of garbage collection
algorithms that are easier to write.

What I'd like to see someday is a language whose runtime explicitly
supports garbage-collection plug-ins so that you can use the appropriate
collection algorithm for the appropriate situation.

Going to the OP's question? Embed the Lua. Ruby is a great language
and great for embedding in applications, even, except in the specific
case of what you're talking about right now: anything with hard realtime
(and even soft realtime!) requirements. Lua is a much faster language
and has garbage collection that is far more easily tuned.

IIRC Lua was designed from the ground up to be embeddable in C code and to make interfacing with C easy. Of course, it's not Forth or Scheme, so I probably wouldn't use it ... :slight_smile:

9

If you're doing midi on a Mac, you should look into some of the Core Audio portion of the Cocoa framework.
There are a lot of things done for you. If you build the app with the Cocoa framework, you can get scriptability with very little effort.
If you're already writing in C, the Objective-C stuff should be easy. The Cocoa framework is pretty vast, and the audio stuff is too.

10

If you're doing midi on a Mac, you should look into some of the Core
Audio portion of the Cocoa framework.

I use the excellent RtMidi (http://www.music.mcgill.ca/~gary/rtmidi/\)
library so the code is portable. In fact it is just some headers for
the vendor specific libraries on Windows, Linux and Mac os x. Audio is
another story into which I might dive someday, but it's not a
priority.

There are a lot of things done for you. If you build the app with the
Cocoa framework, you can get scriptability with very little effort.

If you're already writing in C, the Objective-C stuff should be easy.
The Cocoa framework is pretty vast, and the audio stuff is too.

I have written some applications in Cocoa and it was nice. Now I
prefer to use open source libraries like agar (http://libagar.org/\)
and sdl (www.libsdl.org) instead of the proprietary Cocoa, even if
this library is really good. If I get tired of mac OS X and want to
move to Linux (for speed or whatever good reason), I want to be free
to do so. Rubyk is intended to be a long running project that I will
be using in many future works...

···

2007/10/21, John Joyce <dangerwillrobinsondanger@gmail.com>:

11

Gaspard, just a couple of comments.
I'm working on a similar soft-realtime application (autonomous drive of
a vehicle), and I implemented the high level logic as a Ruby
statemachine. I found that I can get a satisfying level of realtime
(including UDP stack access) by keeping a fixed pace loop like that:

while running do
    slp_thr = Thread.start do sleep timestep end
    #do the loop stuff here
    slp_thr.join
end

It works surprisingly fine until 300-350 Hz.
Since you're running on Macs, you could also use the scheduling control
functions of the Mach kernel. For example, something like that can be
helpful:

inline do |builder|
    builder.include "<mach/mach_init.h>"
    builder.include "<mach/thread_policy.h>"
    builder.include "<sched.h>"
    builder.c '
      int set_realtime(double period, double computation, double
constraint) {
          uint32_t HZ = 2000000000;
          struct thread_time_constraint_policy ttcpolicy;
          int ret;
          ttcpolicy.period=HZ*period; // HZ/160
          ttcpolicy.computation=HZ*computation; // HZ/3300;
          ttcpolicy.constraint=HZ*constraint; // HZ/2200;
          ttcpolicy.preemptible=1;

          if ((ret=thread_policy_set(mach_thread_self(),
              THREAD_TIME_CONSTRAINT_POLICY,
(thread_policy_t)&ttcpolicy,
              THREAD_TIME_CONSTRAINT_POLICY_COUNT)) != KERN_SUCCESS) {
                  fprintf(stderr, "set_realtime() failed.\n");
                  return 0;
          }
          return 1;
      }

      '
end

Cheers,
Paolo

···

--
Posted via http://www.ruby-forum.com/.