Ruby C extension/wrapper around C function that uses scanf and printf

ruby-talk
1

I need Ruby to access a set of external command line applications for
dataseries analysis. Each of these little apps simply reads data from
stdin and then outputs processed results to stdout (using scanf and
printf, they are all written in C).

These functions have to be accessed repeatedly when iterating over
large chunks of data, so accessing them through the pipe (popen or
popen3) doesn't work. Launching/closing an external process 3000 times
in a row results in errors. And that's not efficient. So I have to
write extension for each of those little C programs, a wrapper of
sorts.

I have learned how to write and compile an extension for Ruby. I can
import an extension, pass an array of data to it and return some data
from the extension. The problem is that I have no C experience and
dread thinking of having to replace each 'scanf' routine with my custom
array of imported data (memory allocation, writing iterator etc.). Each
input function here starts with:

     while (scanf("%lf", &y) == 1) { ... etc.

and then follows to process input data, handle cases with input buffer
is too large etc. I would very much like to leave these input()
functions as they are and just feed them my data from Ruby code.

Is it possible to fill stdin buffer with data from Ruby code and then
simply call existing C function that uses 'scanf'? I couldn't simply
assign an array to $stdin variable in Ruby, gave TypeError (File
expected)...

2

1gor wrote:

I need Ruby to access a set of external command line applications for
dataseries analysis. Each of these little apps simply reads data from
stdin and then outputs processed results to stdout (using scanf and
printf, they are all written in C).

These functions have to be accessed repeatedly when iterating over
large chunks of data, so accessing them through the pipe (popen or
popen3) doesn't work.

Do they need changing arguments? Otherwise you could just start an instance of each via popen and then reuse that throughout the whole program. Or do these programs detect end of input data via EOF on STDIN?

Launching/closing an external process 3000 times
in a row results in errors. And that's not efficient.

Yeah, true.

So I have to
write extension for each of those little C programs, a wrapper of
sorts.

I have learned how to write and compile an extension for Ruby. I can
import an extension, pass an array of data to it and return some data
from the extension. The problem is that I have no C experience and
dread thinking of having to replace each 'scanf' routine with my custom
array of imported data (memory allocation, writing iterator etc.). Each
input function here starts with:

     while (scanf("%lf", &y) == 1) { ... etc.

and then follows to process input data, handle cases with input buffer
is too large etc. I would very much like to leave these input()
functions as they are and just feed them my data from Ruby code.

Is it possible to fill stdin buffer with data from Ruby code and then
simply call existing C function that uses 'scanf'? I couldn't simply
assign an array to $stdin variable in Ruby, gave TypeError (File
expected)...

Hm, I'm not a seasoned C programmer either, so I cannot give you a definitive answer to this. However, I doubt that you can simply inject stuff into stdin.

Here are two more ideas:

1. Rewrite your programs to act as if they saw EOF from stdin when they receive a certain signal (say SIGUSR1). That way you could use a single process throughout your whole Ruby script with popen and just need to send the signal at certain points in time. You could even change behavior to use the first line read from the pipe / stdin as command line arguments.

2. If it's feasible rewrite the processing in Ruby. I guess you want to use a C program for certain reasons - these might be performance, not having to rewrite the code or whatever. If it's not performance then maybe you can replace the stuff with 10% LOC in Ruby.

Kind regards

  robert

3

Go through the source code of your external apps, and replace each
scanf() with sscanf(). (sscanf() reads from a string instead from
standard input). Then you can make the Ruby wrappers and pass a string
to each function. For example:

void old_external_app()
{
    float x;
    scanf("%f %f...",&x,...);
    ....
    printf("%f",x);
}

void new_external_app(char *input_str, char *output_str)
{
    float x;
    sscanf( input_str, "%f %f...",&x,...); // the same, except 1st arg
    ...
    sprintf( output_str, "%f", x);
}

Robert Klemme wrote:

···

1gor wrote:
> I need Ruby to access a set of external command line applications for
> dataseries analysis. Each of these little apps simply reads data from
> stdin and then outputs processed results to stdout (using scanf and
> printf, they are all written in C).
>
> These functions have to be accessed repeatedly when iterating over
> large chunks of data, so accessing them through the pipe (popen or
> popen3) doesn't work.

Do they need changing arguments? Otherwise you could just start an
instance of each via popen and then reuse that throughout the whole
program. Or do these programs detect end of input data via EOF on STDIN?

> Launching/closing an external process 3000 times
> in a row results in errors. And that's not efficient.

Yeah, true.

> So I have to
> write extension for each of those little C programs, a wrapper of
> sorts.
>
> I have learned how to write and compile an extension for Ruby. I can
> import an extension, pass an array of data to it and return some data
> from the extension. The problem is that I have no C experience and
> dread thinking of having to replace each 'scanf' routine with my custom
> array of imported data (memory allocation, writing iterator etc.). Each
> input function here starts with:
>
> while (scanf("%lf", &y) == 1) { ... etc.
>
> and then follows to process input data, handle cases with input buffer
> is too large etc. I would very much like to leave these input()
> functions as they are and just feed them my data from Ruby code.
>
> Is it possible to fill stdin buffer with data from Ruby code and then
> simply call existing C function that uses 'scanf'? I couldn't simply
> assign an array to $stdin variable in Ruby, gave TypeError (File
> expected)...

Hm, I'm not a seasoned C programmer either, so I cannot give you a
definitive answer to this. However, I doubt that you can simply inject
stuff into stdin.

Here are two more ideas:

1. Rewrite your programs to act as if they saw EOF from stdin when they
receive a certain signal (say SIGUSR1). That way you could use a single
  process throughout your whole Ruby script with popen and just need to
send the signal at certain points in time. You could even change
behavior to use the first line read from the pipe / stdin as command
line arguments.

2. If it's feasible rewrite the processing in Ruby. I guess you want to
use a C program for certain reasons - these might be performance, not
having to rewrite the code or whatever. If it's not performance then
maybe you can replace the stuff with 10% LOC in Ruby.

Kind regards

  robert