Enterprise ruby

ruby-talk
21

Roger Pack wrote:

If you wanted to avoid ever accessing freed objects at all, you'd need to create an 'allocated' list, as well, so you could just traverse the allocated list and then add those to the freelist that were freed. So about 20%/object size increase. Maybe a good trade off??? Tough to tell. If I guessed I'd say that the trade off is...worth it for large long standing processes. It would use more RAM and be faster.

Actually, although it might use more virtual address space, if done right,
it might consume *less* RAM - meaning physical, working set - simply by
not touching pages that aren't in use.

Knowledge and awareness of the VM state seems to often get neglected in
these discussions of GC, even though it's quite easy to compare the VM
effects of the various types.

Clifford Heath.

22

Maybe this was discussed before, but noone mention it.

I think Rubinius team took the idea of doing some "agnostic" GC that
they could switch backends when needed, without changing the API.

Maybe something similar could be done for MRI? Also, thinking on Boehm
Garbage Collector [1] that wasn't mention any place here, and could
reduce the "time to market" of something.

Anyway, just a shoot in the dark, since my knowledge of GC is too
limited to actually provide more constructive comments :slight_smile:

[1] http://www.hpl.hp.com/personal/Hans_Boehm/gc/

···

On Nov 11, 8:47 am, Michal Suchanek <hramr...@centrum.cz> wrote:

No, it's not dead end. However, I would expect its lifetime something
like 1-2 years. So small tweaks that bring immediate benefit are worth
it. Rewriting the GC probably not. Even if you manage to do it before
1.8 is obsolete it would get intensive use for a few moths at best.
If 2.0 succeeds (and I believe in it) there will be little incentive
to use 1.8 anymore. 2.0 will be the current actively developed
interpreter, and implementing the GC in there makes more sense.

23

Rick DeNatale wrote:
> In this implementation, Java primitives were written in Smalltalk.
> IIRC this was either before the JNI existed, or the JNI evolved to
> make this impractical, and IBM moved to a Java only VM.
>
> I know that it's difficult, and probably premature to define a
> standard extension interface which would work across the various
> emerging ruby implementations. But without that I'm afraid that the
> promise of having multiple implementations is somewhat muted.

This implies that it's only valid to call something an "extension" if
it's written in C. That's a bit narrow. JRuby has far better support
than MRI for writing extensions in Java, for example. Does it mean that
JRuby is somehow less-capable than MRI if it can't use C extensions? Of
course it doesn't.

The only thing it means is that existing extensions written in C for MRI
won't work in JRuby.

JRuby specific extension example omitted.

So I'd say it's a matter of perspective.

Can you write extensions for JRuby? Yes. Can you write them in C? Not
easily.

Can you write extensions for MRI? Yes. Can you write them in Java? Not
easily.

All of which misses my point.

Unless and until extensions can be written which are portable against
Ruby, JRuby, IronRuby, Rubinius ... the Ruby 'market' will be
fragmented.

···

On Nov 11, 2007 3:11 PM, Charles Oliver Nutter <charles.nutter@sun.com> wrote:

--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

24

Clifford Heath wrote:

Actually, although it might use more virtual address space, if done right,
it might consume *less* RAM - meaning physical, working set - simply by
not touching pages that aren't in use.

Knowledge and awareness of the VM state seems to often get neglected in
these discussions of GC, even though it's quite easy to compare the VM
effects of the various types.

platform.each do |p|
   p.vm.research
   p.phd.hire unless p.metrics.useful?
   p.tools.write unless p.tools.useful?
end

:slight_smile:

25

That's why I say, rather use an existing and highly optimized GC that exists today (meaning the JVM's GC) than invest loads of development and research time to recreate something similar. I guess several tens or hundreds of man years have gone into the JVM's GC - that's difficult to top.

Kind regards

  robert

···

On 10.11.2007 23:32, M. Edward (Ed) Borasky wrote:

As a previous email suggested, there are a couple of use cases for a garbage collector, only one of them being long-running server applications.

26

Luis Lavena wrote:

Maybe this was discussed before, but noone mention it.

I think Rubinius team took the idea of doing some "agnostic" GC that
they could switch backends when needed, without changing the API.

Maybe something similar could be done for MRI? Also, thinking on Boehm
Garbage Collector [1] that wasn't mention any place here, and could
reduce the "time to market" of something.

The ability to replace the Ruby GC with something a bit more advanced is severely hindered by--you guessed it--C extensions. Ruby's extension API allows access to the objects directly, so extensions can access the data in-memory. This is anathema to any GC that wants to move objects around, and of course it's absolutely impossible to do any GC operations in parallel if extensions might be holding direct references to memory.

So in the area of GC, at least, Ruby is being hindered by its extension API rather than helped.

- Charlie

27

"Would that be useful to anyone? Would anyone use it?"

I dont think I would use it (insofar as I dont really need it, as i use
ruby daily and thus more or less compile all ruby-addons using ruby
scripts anyway) but it sure sounds an interesting project and I would be
curious to know how far it (will) go :slight_smile:

"And any other personal tweaks that people contribute. Kind of a
bleeding edge Ruby."

I think this will be a bit more interesting. I never really used facets,
but I for sure use a lot of my own methods etc... and if people all
stick and improve on one point (like in facets), it could make life a
lot easier (and thus, the project interesting). From this latter aspect,
I am even more curious than the other mention aspect (i.e svn checkout
or GC)

···

--
Posted via http://www.ruby-forum.com/\.

28

Rick DeNatale wrote:

All of which misses my point.

Unless and until extensions can be written which are portable against
Ruby, JRuby, IronRuby, Rubinius ... the Ruby 'market' will be
fragmented.

Only as far as extensions go. I don't advocate writing extensions; rather, I advocate using rich in-language capabilities to call platform libraries, as you can do in JRuby. You can extend JRuby without ever having to write what would be considered an "extension" in MRI. If MRI had a similar capability, (or perhaps if DL was use more and in better shape) there would be less fragmentation, since anything platform-specific could be wrapped by a nice Ruby library, and no C or Java code would have to be shipped.

It's the very fact that Ruby extensions are written in C and compiled with C compilers to shared libraries with very specific constraints that limits their general applicability to other implementations (and limits evolution of core Ruby development as well).

So I would propose that what you're saying is right...and that writing C-based extensions by hand just aggravates the problem, since C-based extensions are never going to be generally applicable to all implementations.

- Charlie

29

Well I never said that portable extensions needed to be written in C.

So maybe we should be rooting for Evan, and hope that we can soon be
writing extensions in a ruby-like version of Squeak's Slang.

···

On Nov 11, 2007 6:39 PM, Charles Oliver Nutter <charles.nutter@sun.com> wrote:

Rick DeNatale wrote:
> All of which misses my point.
>
> Unless and until extensions can be written which are portable against
> Ruby, JRuby, IronRuby, Rubinius ... the Ruby 'market' will be
> fragmented.

It's the very fact that Ruby extensions are written in C and compiled
with C compilers to shared libraries with very specific constraints that
limits their general applicability to other implementations (and limits
evolution of core Ruby development as well).

So I would propose that what you're saying is right...and that writing
C-based extensions by hand just aggravates the problem, since C-based
extensions are never going to be generally applicable to all
implementations.

--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

30

Rick DeNatale wrote:

Well I never said that portable extensions needed to be written in C.

So maybe we should be rooting for Evan, and hope that we can soon be
writing extensions in a ruby-like version of Squeak's Slang.

Maybe we shouldn't be "rooting" for anyone, and we should instead be lending a hand by trying to solve the problem ourselves :wink:

- Charlie

31

I believe Rubinius actually "fakes" the MRI extensions API so your
Ruby extensions should work just fine on rbx.

(Evan, Yehuda, or someone else smarter than I please chime in if
that's not true...)

--Jeremy

···

On Nov 11, 2007 7:13 PM, Rick DeNatale <rick.denatale@gmail.com> wrote:

On Nov 11, 2007 6:39 PM, Charles Oliver Nutter <charles.nutter@sun.com> wrote:
> Rick DeNatale wrote:
> > All of which misses my point.
> >
> > Unless and until extensions can be written which are portable against
> > Ruby, JRuby, IronRuby, Rubinius ... the Ruby 'market' will be
> > fragmented.

> It's the very fact that Ruby extensions are written in C and compiled
> with C compilers to shared libraries with very specific constraints that
> limits their general applicability to other implementations (and limits
> evolution of core Ruby development as well).
>
> So I would propose that what you're saying is right...and that writing
> C-based extensions by hand just aggravates the problem, since C-based
> extensions are never going to be generally applicable to all
> implementations.

Well I never said that portable extensions needed to be written in C.

So maybe we should be rooting for Evan, and hope that we can soon be
writing extensions in a ruby-like version of Squeak's Slang.

--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

--
http://www.jeremymcanally.com/

My books:
Ruby in Practice

My free Ruby e-book

My blogs:

http://www.rubyinpractice.com/

32

Rick DeNatale wrote:

Rick DeNatale wrote:

Unless and until extensions can be written which are portable against
Ruby, JRuby, IronRuby, Rubinius ... the Ruby 'market' will be
fragmented.

It's the very fact that Ruby extensions are written in C and compiled
with C compilers to shared libraries with very specific constraints that
limits their general applicability to other implementations (and limits
evolution of core Ruby development as well).

So I would propose that what you're saying is right...and that writing
C-based extensions by hand just aggravates the problem, since C-based
extensions are never going to be generally applicable to all
implementations.

Well I never said that portable extensions needed to be written in C.

So maybe we should be rooting for Evan, and hope that we can soon be
writing extensions in a ruby-like version of Squeak's Slang.

Welll ... there are only a few reasons why one would write an extension in the first place:

1. To work with a data type not native to the language. As far as I know, the only commonly-used data type that doesn't have native operations in MRI is large multidimensional arrays of floating point (possibly complex) numeric data. So we need NArray and Ruby interfaces to C code like GSL, and Ruby becomes a "control language" rather than a data processing language.

2. To use existing functionality already written in another language. This is why most extensions get written, and since most of the functionality is in C/C++, most of the extensions are written in C. In jRuby, of course, rather than use existing functionality in C, one uses existing functionality written in Java. Again, Ruby becomes a "control language" rather than a data processing language.

3. To get more processor- or memory-efficient operation.

4. Because it's open source and we can. :slight_smile:

I think, of all these reasons, 4 is the only *legitimate* reason to write an extension in a language other than Ruby. For 1, as I noted, as far as I know there's only one missing data type, and I think it could be easily integrated into any of the implementations. It's just a matter of the community deciding that it should be there and coming up with a syntax and semantics workable in all of the implementations.

For 2, given that Ruby operates on most platforms, it's almost always possible to accomplish 2 using "loose coupling" with little or no penalty. For 3, I think YARV and jRuby have both shown that one can stay in Ruby and still gain significant efficiency. And Rubinius has as its goal minimizing the size of the C core without losing efficiency, and I think they're well on their way to that.

So that leaves 4 ... because it's open source and we can.

···

On Nov 11, 2007 6:39 PM, Charles Oliver Nutter <charles.nutter@sun.com> wrote:

33

Possible, yes. (Again, Turing-complete and all that.) But it depends what
you mean by "penalty".

The very first thing I think of when I think "Ruby and native extensions"
is RMagick/ImageMagick for a web site. And that's notoriously difficult to
build even in a C-based, MRI environment, although I understand that's
supposed to be better now in 2.0 (I haven't tried it yet).

What would the "right answer" be in JRuby? I just did some Googling, and
from what I can tell, there are a few ways you could go:

- JAI, which a bunch of people claim is too abstracted and buggy, and which
a bunch of others claim is fine and the first people are whiners

- Mistral, which provides an opaque abstraction over JAI, probably making
the first two bunches of people look like Red Sox and Yankees fans in a bar
with Scottish soccer fans

- ImageMagick via JNI (!)

- ImageMagick via exec (!!)

- ImageJ, which is pure Java but apparently has a lot of problems running
on a headless server due to its use of AWT; there's now a headless version
of the ImageJA fork, but apparently if you use any plugins you have to go
stub out all their AWT calls as wel, or something like that. Definitely
not a mature solution yet.

None of these sound all that appealing. You talk about loose coupling, and
when I think loose coupling, I think messaging. Which sounds good, and
obviously lends itself to parallel computing and multicore and grids and
all that, but as the Java world certainly knows, wrapping something like
ImageMagick in a messaging server would be a huge rat's-nest-infested
undertaking - and something that only someone with good experience in Ruby
AND Java AND image processing AND messaging could do.

As someone who's running about 1.5 out of 4 there, I'd probably just go
create a DRb/Rinda/whatever server for the ImageMagick stuff and deal with
having both a Java and a C environment going. But I wouldn't be very happy
about it, and it's certainly not generalizable.

You almost want some kind of messaging-oriented SWIG. Except SWIG itself
isn't even that easy to use (every time I think about using it, I end up
drifting toward Ruby::Inline instead; I think you only get the SWIG
advantage if you really target multiple languages.) So you want a
messaging-oriented BETTER SWIG. Without any of the complexity of SWIG. Or
messaging.

Thoughts?

···

On Sun, 11 Nov 2007 22:32:14 -0500, M. Edward (Ed) Borasky wrote:

For 2, given that Ruby operates on most platforms, it's almost always
possible to accomplish 2 using "loose coupling" with little or no
penalty.

--
Jay Levitt |
Boston, MA | My character doesn't like it when they
Faster: jay at jay dot fm | cry or shout or hit.
http://www.jay.fm | - Kristoffer

34

Jay Levitt wrote:

For 2, given that Ruby operates on most platforms, it's almost always possible to accomplish 2 using "loose coupling" with little or no penalty.

Possible, yes. (Again, Turing-complete and all that.) But it depends what
you mean by "penalty".

The very first thing I think of when I think "Ruby and native extensions"
is RMagick/ImageMagick for a web site. And that's notoriously difficult to
build even in a C-based, MRI environment, although I understand that's
supposed to be better now in 2.0 (I haven't tried it yet).

I actually thought about ImageMagick when I wrote that. Doesn't ImageMagick have a command line interface? Can't Ruby (and jRuby) execute command lines? That's what I had in mind when I wrote "loose coupling". :slight_smile:

[snip]

You almost want some kind of messaging-oriented SWIG. Except SWIG itself
isn't even that easy to use (every time I think about using it, I end up
drifting toward Ruby::Inline instead; I think you only get the SWIG
advantage if you really target multiple languages.) So you want a
messaging-oriented BETTER SWIG. Without any of the complexity of SWIG. Or
messaging.

SWIG requires deep knowledge of the library you are attempting to engulf in your scripting language. Indeed, the payoff is primarily if you want one library to serve many scripting languages, not just Ruby. However, if you want to declare Ruby the one true language, Ruby::Inline is the way to go.

Thoughts?

Well ... I'm sticking with "because it's open source and we can" as the best reason for extending a language. Implicit in that is a plea *to* extend the language to the primitive data types necessary for, say, signal and image processing, which are -- surprise -- multidimensional arrays of numeric data packed contiguously in RAM.

I'm genuinely curious about this -- MATLAB is a (very expensive closed source) package that caters to numerical processing, including signal and image processing. Can MATLAB process regular expressions? Can Mathematica? Or does one need a two-language solution -- MATLAB for the number crunching and Perl, Python or Ruby for the data extraction?

···

On Sun, 11 Nov 2007 22:32:14 -0500, M. Edward (Ed) Borasky wrote:

35

Jay Levitt wrote:

For 2, given that Ruby operates on most platforms, it's almost always possible to accomplish 2 using "loose coupling" with little or no penalty.

Possible, yes. (Again, Turing-complete and all that.) But it depends what
you mean by "penalty".

The very first thing I think of when I think "Ruby and native extensions"
is RMagick/ImageMagick for a web site. And that's notoriously difficult to
build even in a C-based, MRI environment, although I understand that's
supposed to be better now in 2.0 (I haven't tried it yet).

What would the "right answer" be in JRuby? I just did some Googling, and
from what I can tell, there are a few ways you could go:

The truth is there's probably another dozen options beyond those listed here, so it's hard to say what the best approach would be. There's even a Java-based ImageMagick that either wraps or duplicates IM's functionality.

So I guess there's an important introductory question to ask before we would start investigating:

Is it ImageMagick's exact specifications or general image processing we want to provide? The former is obviously much harder, while the latter can be done through any number of libraries.

I think you'd be hard-pressed to find any library in the C world that doesn't have a reasonably good equivalent in the Java world (sometimes that just wrap those libraries from the C world). So it's certainly possible and reasonable to say most extensions could be provided in a similar-if-not-identical form under JRuby with just a bit of plumbing. And thankfully JRuby's integration with Java means that plumbing should be less than equivalent in the C world (i.e. the C world is fraught with peril, as RMagick's occasional instability has shown).

- Charlie

···

On Sun, 11 Nov 2007 22:32:14 -0500, M. Edward (Ed) Borasky wrote:

36

M. Edward (Ed) Borasky wrote:

I'm genuinely curious about this -- MATLAB is a (very expensive closed
source) package that caters to numerical processing, including signal
and image processing. Can MATLAB process regular expressions? Can
Mathematica? Or does one need a two-language solution -- MATLAB for the
number crunching and Perl, Python or Ruby for the data extraction?

String processing in MATLAB (Octave, really) is agony, so I tend to use
a hybrid of Perl and Octave. I've tried Ruby plus NArray/rb-gsl and
Perl plus PDL, but neither has Octave's huge number of linear algebra
functions (SVD, least-squares, eigenvalues, ...) or plotting support.
Compact arrays are crucial, but they're only a small part of the story.

···

--
Posted via http://www.ruby-forum.com/\.

37

The very first thing I think of when I think "Ruby and native extensions"
is RMagick/ImageMagick for a web site. And that's notoriously difficult to
build even in a C-based, MRI environment, although I understand that's
supposed to be better now in 2.0 (I haven't tried it yet).

What would the "right answer" be in JRuby? I just did some Googling, and
from what I can tell, there are a few ways you could go:

The truth is there's probably another dozen options beyond those listed
here, so it's hard to say what the best approach would be. There's even
a Java-based ImageMagick that either wraps or duplicates IM's functionality.

The only Java-ImageMagick library I can find is JMagick, a JNI bridge to
ImageMagick (which I'm guessing won't build in JRuby).

So I guess there's an important introductory question to ask before we
would start investigating:

Is it ImageMagick's exact specifications or general image processing we
want to provide? The former is obviously much harder, while the latter
can be done through any number of libraries.

Well, in theory, the latter; in reality, the former. ImageMagick, as
unwieldy as it's been, seems to be a de facto standard in the web
community. I must have installed it five zillion times by now (that's in
metric, you understand) for various PHP, Perl, and Ruby packages.

Only a subset of ImageMagick is really used there - basically enough to get
thumbnails, text rendering, and maybe the equivalent of Amazon's image
processing features (http://aaugh.com/imageabuse.html\). But the API oughta
Just Work. There are drop-in replacements for RMagick, like ImageScience -
but even that needs FreeImage (binary!) and Ruby::Inline (C!).

···

On Mon, 12 Nov 2007 13:03:42 -0500, Charles Oliver Nutter wrote:

-
Jay Levitt |
Boston, MA | My character doesn't like it when they
Faster: jay at jay dot fm | cry or shout or hit.
http://www.jay.fm | - Kristoffer

38

I actually thought about ImageMagick when I wrote that. Doesn't
ImageMagick have a command line interface? Can't Ruby (and jRuby)
execute command lines? That's what I had in mind when I wrote "loose
coupling". :slight_smile:

Ah, yeah, that's VERY loose coupling! And there are people doing that
today; there's a recommendation on an O'Reilly blog to skip the JNI JMagick
wrapper and just use Runtime.exec().

Aside from the obvious performance hit - a new process for every image! -
it just feels too much like the old days of MIS-written batch-processed
no-error-checking copy-a-file-into-the-queue-directory scripts. I'm
probably being religious there, though. But the performance issue remains.

Well ... I'm sticking with "because it's open source and we can" as the
best reason for extending a language. Implicit in that is a plea *to*
extend the language to the primitive data types necessary for, say,
signal and image processing, which are -- surprise -- multidimensional
arrays of numeric data packed contiguously in RAM.

I realize that you're talking mostly about scientific software, and that
the image-processing would be a side benefit. But I do want to point out
that, to a non-comp-sci guy like myself, image processing has nothing to do
with a multidimensional array of numeric data packed contiguously in RAM.
It has to do with thumbnails and resizing and drawing lines and circles and
fonts. I'm looking for an iPod, not an FPGA. Unless someone wanted to
reinvent ImageMagick as a pure-ruby solution based on those new primitive
types, they wouldn't do me any good for this example.

···

On Mon, 12 Nov 2007 10:02:21 -0500, M. Edward (Ed) Borasky wrote:

--
Jay Levitt |
Boston, MA | My character doesn't like it when they
Faster: jay at jay dot fm | cry or shout or hit.
http://www.jay.fm | - Kristoffer

39

Sean Surname wrote:

M. Edward (Ed) Borasky wrote:

I'm genuinely curious about this -- MATLAB is a (very expensive closed
source) package that caters to numerical processing, including signal
and image processing. Can MATLAB process regular expressions? Can
Mathematica? Or does one need a two-language solution -- MATLAB for the
number crunching and Perl, Python or Ruby for the data extraction?

String processing in MATLAB (Octave, really) is agony, so I tend to use a hybrid of Perl and Octave. I've tried Ruby plus NArray/rb-gsl and Perl plus PDL, but neither has Octave's huge number of linear algebra functions (SVD, least-squares, eigenvalues, ...) or plotting support. Compact arrays are crucial, but they're only a small part of the story.

GSL doesn't have SVD, least-squares and eigenvalues? Well then -- you need (pregnant pause) RSRuby!! It hooks into the whole R language shared library, and I think it can get to the BLAS and LAPACK libraries in R too. But I really thought LAPACK was part of GSL.

40

Jay Levitt wrote:

The only Java-ImageMagick library I can find is JMagick, a JNI bridge to
ImageMagick (which I'm guessing won't build in JRuby).

Actually, it probably would work fine, but I'm not sure anyone's tried it. Of course, JNI brings along its own considerations, but nothing in JRuby prevents JNI from working; it's just more painful to wire libraries in from scratch using JNI.

Well, in theory, the latter; in reality, the former. ImageMagick, as
unwieldy as it's been, seems to be a de facto standard in the web
community. I must have installed it five zillion times by now (that's in
metric, you understand) for various PHP, Perl, and Ruby packages.

Only a subset of ImageMagick is really used there - basically enough to get
thumbnails, text rendering, and maybe the equivalent of Amazon's image
processing features (http://aaugh.com/imageabuse.html\). But the API oughta
Just Work. There are drop-in replacements for RMagick, like ImageScience -
but even that needs FreeImage (binary!) and Ruby::Inline (C!).

I guess my specific problem with RMagick (beyond those that e.g. inspired ImageScience) is that like most Ruby extensions there's no abstraction going on. It's little more than a wrapper around a C library, resulting in apps being tightly coupled to one specific image-processing library. If that weren't the case, and if the operations were a bit more abstracted, it would be a lot easier to provide the same functionality from any other library.

- Charlie