since you seem to be online matz 
any thoughts on a good approach for people that want to do some memory sharing
between ruby objects? say strings and mmaps and narrays and images all
sharing some memory regions. any do's and dont's or other thoughts?
cheers.
-a
Hi,
In message "Re: sharing memory in ruby" on Thu, 8 Sep 2005 09:48:48 +0900, "Ara.T.Howard" <Ara.T.Howard@noaa.gov> writes:
any thoughts on a good approach for people that want to do some memory sharing
between ruby objects? say strings and mmaps and narrays and images all
sharing some memory regions. any do's and dont's or other thoughts?
I don't thing you need to share memory within same processes. Just
refer to a same object. mmap seems reasonable solution to share
memory between processes. The another idea is using some kind of RPC,
such as dRuby instead of sharing memory.
matz.
attm. i'm writing some code to process images for katrina, check out
http://dmsp.ngdc.noaa.gov/interest/katrina.html
i sometimes must shared memory between an mmap, ruby string, and an narray to
i can work on it in memory. our machines have 8gb, but a couple channels and
some duplicate data and i run out... bascially i have an extension to narray
that allows it to share it's storage with an mmap in the same process but,
apparently, this is frowned upon? bascially i allow any string type object to
be passed to an narray and it uses that string's memory without copying it -
in this way i can
na = NArray::str mmap.to_s, samples, lines, NArray::SINT
any comments on this approach?
my narray.c patch is below...
the really critical bit is this:
+ ary->ptr = RSTRING(str)->ptr;
+ ary->ref = Qfalse;
the 'ary->ref' means it will not be free'd...
so now the NArray and String, which is itself an Mmap are all sharing memory.
the neat thing is that i can do this:
na = na + 1
and a the file backing is affected with no io on my part. also i can load
image larger than memory and, therfore, many large images at once. is this a
terrible idea? if not what alternative might i use? perhaps a generic
Pointer (shudder) object that behaves like a String might be useful...
cheers.
-a
On Thu, 8 Sep 2005, Yukihiro Matsumoto wrote:
Hi,
In message "Re: sharing memory in ruby" > on Thu, 8 Sep 2005 09:48:48 +0900, "Ara.T.Howard" <Ara.T.Howard@noaa.gov> writes:
>any thoughts on a good approach for people that want to do some memory sharing
>between ruby objects? say strings and mmaps and narrays and images all
>sharing some memory regions. any do's and dont's or other thoughts?I don't thing you need to share memory within same processes. Just
refer to a same object. mmap seems reasonable solution to share
memory between processes. The another idea is using some kind of RPC,
such as dRuby instead of sharing memory.
email :: ara [dot] t [dot] howard [at] noaa [dot] gov
phone :: 303.497.6469
Your life dwells amoung the causes of death
Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================
--- narray.c.org 2004-10-15 20:57:34.000000000 -0600
+++ narray.c 2004-10-15 21:40:21.000000000 -0600
@@ -757,6 +757,54 @@
/* singleton method: + NArray.str( string, type, size1,size2,...,sizeN )
+*/
+static VALUE
+ na_s_str(int argc, VALUE *argv, VALUE klass)
+{
+ struct NARRAY *ary;
+ VALUE v;
+ VALUE str;
+ int i, type, len=1, str_len, *shape, rank=argc-2;
+
+ if (argc < 1)
+ rb_raise(rb_eArgError, "String Argument required");
+
+ if (argc < 2)
+ rb_raise(rb_eArgError, "Type and Size Arguments required");
+
+ type = na_get_typecode(argv[1]);
+
+ str = rb_funcall(argv[0], rb_intern("to_str"), 0);
+
+ str_len = RSTRING(str)->len;
+
+ if (argc == 2) {
+ rank = 1;
+ shape = ALLOCA_N(int,rank);
+ if ( str_len % na_sizeof[type] != 0 )
+ rb_raise(rb_eArgError, "string size mismatch");
+ shape[0] = str_len / na_sizeof[type];
+ }
+ else {
+ shape = ALLOCA_N(int,rank);
+ for (i=0; i<rank; i++)
+ len *= shape[i] = NUM2INT(argv[i+2]);
+ len *= na_sizeof[type];
+ if ( len != str_len )
+ rb_raise(rb_eArgError, "size mismatch");
+ }
+ + v = na_make_object( type, rank, shape, cNArray );
+ GetNArray(v,ary);
+ ary->ptr = RSTRING(str)->ptr;
+ ary->ref = Qfalse;
+
+ return v;
+}
+
+/* singleton method:
NArray[object]
*/
static VALUE
@@ -1189,6 +1237,7 @@
rb_define_singleton_method(cNArray,"to_na",na_s_to_na,-1);
rb_define_singleton_method(cNArray,"to_narray",na_s_to_na,-1);
rb_define_singleton_method(cNArray,"",na_s_bracket,-1);
+ rb_define_singleton_method(cNArray,"str",na_s_str,-1);
/* methods */
rb_define_method(cNArray, "", na_aref,-1);
Hi,
Ara wrote:
+ ary->ref = Qfalse;the 'ary->ref' means it will not be free'd...
This is not what I have intended.
'ary->ref' is an object that is referred from NArray object
so that the original String object is not garbage-collected.
So here should be `ary->ref = str;'.
+ ary->ptr = RSTRING(str)->ptr;
This has a serious caveat.
s = "0000"
na = NArray.str(s,2,NArray::SINT)
s << "1"*100 # The memory of 's' will be reallocated.
puts na[0] # accesses an invalid memory.
However, I think it would be quite useful if Ruby have something like
a common memory framework, which Ara suggested, for handling huge memory.
Masahiro Tanaka
Hi,
In message "Re: sharing memory in ruby" on Thu, 8 Sep 2005 12:55:51 +0900, Masahiro TANAKA <masa@ir.isas.ac.jp> writes:
However, I think it would be quite useful if Ruby have something like
a common memory framework, which Ara suggested, for handling huge memory.
Perhaps. If you have any requirement/proposal/suggestion for that
kind of memory framework, I'd love to hear. I am not the right person
to make up the memory sharing, since I am no expert on the field.
matz.
Hi,
Ara wrote:
+ ary->ref = Qfalse;the 'ary->ref' means it will not be free'd...
This is not what I have intended.
'ary->ref' is an object that is referred from NArray object
so that the original String object is not garbage-collected.
So here should be `ary->ref = str;'.
hmmm. i did not understand that. i'll make the mod and report back...
thanks for the insight.
+ ary->ptr = RSTRING(str)->ptr;
This has a serious caveat.
s = "0000"
na = NArray.str(s,2,NArray::SINT)
s << "1"*100 # The memory of 's' will be reallocated.
puts na[0] # accesses an invalid memory.
yes i'm aware of that. but, in my case, the string is actually a memory map
using guy's mmap and i hold it as a private variable. true, it could be
changed, but in fact it never is since the map is pinned to a specific
region/length of memory and i never tough the mmap object. i think the danger
is worth it since programs which normally could not even run run easily and
can be written in only a few lines. so, in my case the alternative has a
serious caveat - namely NoMemoryError it __is__ important to realize
this danger however.
However, I think it would be quite useful if Ruby have something like a
common memory framework, which Ara suggested, for handling huge memory.
any thoughts on what the ruby would look like?
kind regards.
-a
email :: ara [dot] t [dot] howard [at] noaa [dot] gov
phone :: 303.497.6469
Your life dwells amoung the causes of death
Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================
hi masa-
here's the result:
jib:~/build/narray-0.5.7p2 > cat a.rb
On Thu, 8 Sep 2005, Masahiro TANAKA wrote:
Hi,
Ara wrote:
+ ary->ref = Qfalse;the 'ary->ref' means it will not be free'd...
This is not what I have intended.
'ary->ref' is an object that is referred from NArray object
so that the original String object is not garbage-collected.
So here should be `ary->ref = str;'.
#
# the narray lib below is patched with your suggested mod
#
require 'mmap'
require './narray'
data, format = './data', 'c*'
#
# create a file
#
open(data, 'w'){|f| f.write [42,0,0,42].flatten.pack(format)}
#
# map in in
#
mmap = Mmap::new 'data', 'rw', Mmap::MAP_SHARED
#
# modify mmap via mmap
#
na = NArray::str mmap.to_str, NArray::BYTE
na = 42
#
# original backing file is changed
#
p IO::read(data).unpack(format)
jib:~/build/narray-0.5.7p2 > ruby a.rb
[42, 42, 42, 42]
looks good. how would you feel about making a release with the patch?
it's inlined below.
cheers.
email :: ara [dot] t [dot] howard [at] noaa [dot] gov
phone :: 303.497.6469
Your life dwells amoung the causes of death
Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================
--- narray.c.org 2004-10-15 20:57:34.000000000 -0600
+++ narray.c 2004-10-15 21:40:21.000000000 -0600
@@ -757,6 +757,54 @@
/* singleton method: + NArray.str( string, type, size1,size2,...,sizeN )
+*/
+static VALUE
+ na_s_str(int argc, VALUE *argv, VALUE klass)
+{
+ struct NARRAY *ary;
+ VALUE v;
+ VALUE str;
+ int i, type, len=1, str_len, *shape, rank=argc-2;
+
+ if (argc < 1)
+ rb_raise(rb_eArgError, "String Argument required");
+
+ if (argc < 2)
+ rb_raise(rb_eArgError, "Type and Size Arguments required");
+
+ type = na_get_typecode(argv[1]);
+
+ str = rb_funcall(argv[0], rb_intern("to_str"), 0);
+
+ str_len = RSTRING(str)->len;
+
+ if (argc == 2) {
+ rank = 1;
+ shape = ALLOCA_N(int,rank);
+ if ( str_len % na_sizeof[type] != 0 )
+ rb_raise(rb_eArgError, "string size mismatch");
+ shape[0] = str_len / na_sizeof[type];
+ }
+ else {
+ shape = ALLOCA_N(int,rank);
+ for (i=0; i<rank; i++)
+ len *= shape[i] = NUM2INT(argv[i+2]);
+ len *= na_sizeof[type];
+ if ( len != str_len )
+ rb_raise(rb_eArgError, "size mismatch");
+ }
+ + v = na_make_object( type, rank, shape, cNArray );
+ GetNArray(v,ary);
+ ary->ptr = RSTRING(str)->ptr;
+ ary->ref = str;
+
+ return v;
+}
+
+/* singleton method:
NArray[object]
*/
static VALUE
@@ -1189,6 +1237,7 @@
rb_define_singleton_method(cNArray,"to_na",na_s_to_na,-1);
rb_define_singleton_method(cNArray,"to_narray",na_s_to_na,-1);
rb_define_singleton_method(cNArray,"",na_s_bracket,-1);
+ rb_define_singleton_method(cNArray,"str",na_s_str,-1);
/* methods */
rb_define_method(cNArray, "", na_aref,-1);