[ANN] na_str-0.0.0 : combining narray with mmap for persistant numerical data

ruby-talk
1

NAME

   na_str (c extension)

   nmap (reference ruby class which combines narray and mmap)

URIS

   http://codeforpeople.com/lib/ruby/na_str/

SYNOPSIS

   na_str is designed to allow data sharing between narray object and other ruby
   objects. the shared data is that returned by the objects to_s or to_str method
   and rb_string_new4. using this technique memory mapped (using guy's mmap
   extensions) data can be altered with no explicit io on the users part and
   partial changed to numerical grids can occur very quickly and persistently.

   the nmap.rb reference impl is a concrete example of this usage and is
   included in the distribution

EXAMPLE USAGE

     jib:~/eg/ruby/na_str > cat a.rb

···

#
     # the nmap extension is installed along with na_str
     #
       require 'nmap'
     #
     # the NMap ctor interface is similar to NArray's, but a backing file must also
     # be specified
     #
       path, x, y = 'int.data', 3, 4
       nmap = NMap.int path, x, y
     #
     # copy the last row to the first
     #
       nmap.na[true, 0] = nmap.na[true, 3]
     #
     # set the last row to be the current time. if you run this a few times you'll
     # notice that changes to the narray are automatically written to the to
     # backing file via the magic of mmap
     #
       nmap.na[true, 3] = Time.now.to_i
     #
     # show the narray
     #
       p nmap.na

   jib:~/eg/ruby/na_str > ruby a.rb
   NArray(ref).int(3,4):
   [ [ 0, 0, 0 ],
     [ 0, 0, 0 ],
     [ 1151439012, 1151439012, 1151439012 ] ]

   jib:~/eg/ruby/na_str > ruby a.rb
   NArray(ref).int(3,4):
   [ [ 1151439012, 1151439012, 1151439012 ],
     [ 0, 0, 0 ],
     [ 1151439014, 1151439014, 1151439014 ] ]

SPEED

   jib:~/eg/ruby/na_str > cat a.rb
   #
   # reference impl installed with na_str
   #
     require 'nmap'
   #
   # setup a narray grid containing 1 gb of ints
   #
     gb = 2 ** 30
     mb = 2 ** 20
     sizeof_int = [42].pack('i').size
     gig_of_ints = gb / sizeof_int
     nmap = NMap.int '1GB', gig_of_ints
   #
   # set about 1 million of the ints to 42
   #
     nmap.na[0 .. mb] = 42
   #
   # show that the data was written
   #
     p nmap.na[mb - 1]

   jib:~/eg/ruby/na_str > time ruby a.rb
   42

   real 0m0.078s
   user 0m0.020s
   sys 0m0.020s

   jib:~/eg/ruby/na_str > ls -ltar 1GB
   -rw-rw-r-- 1 ahoward ahoward 1073741824 Jun 27 14:28 1GB

this an experimental package - thank to matz for the tip on rb_str_new4.

enjoy.

-a
--
suffering increases your inner strength. also, the wishing for suffering
makes the suffering disappear.
- h.h. the 14th dali lama

2

Ara, for the uninitiated among us, could you please explain what
rb_str_new4 does exactly? Also, what is the ELTS_SHARED flag
signifies and in what circumstances should it be used?

Blessings,
TwP

···

On 6/27/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:

this an experimental package - thank to matz for the tip on rb_str_new4.

enjoy.

3

This is very cool, but won't the name be confusing?

$ man nmap
...
       nmap - Network exploration tool and security scanner
...

···

ara.t.howard@noaa.gov wrote:

NAME

  na_str (c extension)

  nmap (reference ruby class which combines narray and mmap)

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

4

Hi,

···

In message "Re: [ANN] na_str-0.0.0 : combining narray with mmap for persistant numerical data" on Thu, 29 Jun 2006 02:14:34 +0900, "Tim Pease" <tim.pease@gmail.com> writes:

Ara, for the uninitiated among us, could you please explain what
rb_str_new4 does exactly? Also, what is the ELTS_SHARED flag
signifies and in what circumstances should it be used?

rb_str_new4() creates a new string that shares internal memory region
with the original. ELTS_SHARED means the string is sharing memory
region with other string object.

              matz.

5

i open to suggestion - ideas? this first one was just a whack at it anyhow -
but it's already pretty useful.

cheers.

-a

···

On Fri, 30 Jun 2006, Joel VanderWerf wrote:

ara.t.howard@noaa.gov wrote:

NAME

  na_str (c extension)

  nmap (reference ruby class which combines narray and mmap)

This is very cool, but won't the name be confusing?

$ man nmap
...
      nmap - Network exploration tool and security scanner
...

--
suffering increases your inner strength. also, the wishing for suffering
makes the suffering disappear.
- h.h. the 14th dali lama

6

maybe na_map?

-a

···

On Fri, 30 Jun 2006, Joel VanderWerf wrote:

ara.t.howard@noaa.gov wrote:

NAME

  na_str (c extension)

  nmap (reference ruby class which combines narray and mmap)

This is very cool, but won't the name be confusing?

$ man nmap
...
      nmap - Network exploration tool and security scanner
...

--
suffering increases your inner strength. also, the wishing for suffering
makes the suffering disappear.
- h.h. the 14th dali lama

7

ara.t.howard@noaa.gov writes:

NAME

  na_str (c extension)

  nmap (reference ruby class which combines narray and mmap)

This is very cool, but won't the name be confusing?

$ man nmap
...
      nmap - Network exploration tool and security scanner
...

maybe na_map?

What about ActiveMMappedNArray? *scnr*.

···

On Fri, 30 Jun 2006, Joel VanderWerf wrote:

ara.t.howard@noaa.gov wrote:

-a

--
Christian Neukirchen <chneukirchen@gmail.com> http://chneukirchen.org

8

heh.

i'll count that as na_map +1 then :wink:

-a

···

On Sat, 1 Jul 2006, Christian Neukirchen wrote:

ara.t.howard@noaa.gov writes:

On Fri, 30 Jun 2006, Joel VanderWerf wrote:

ara.t.howard@noaa.gov wrote:

NAME

  na_str (c extension)

  nmap (reference ruby class which combines narray and mmap)

This is very cool, but won't the name be confusing?

$ man nmap
...
      nmap - Network exploration tool and security scanner
...

maybe na_map?

What about ActiveMMappedNArray? *scnr*.

--
suffering increases your inner strength. also, the wishing for suffering
makes the suffering disappear.
- h.h. the 14th dali lama