[ANN] fairy: A framework for distributed processing

ruby-talk
#1

We are very happy to announce that fairy 0.6.0 has just been released!

fairy is a framework for distributed processing in Ruby, originally
designed at Rakuten Institute of Technology with Yukihiro Matsumoto,
the founder of Ruby.

Although fairy was inspired by MapReduce model, a well-known
programming model for distributed processing, it's more flexible and
suitable for wider use. That's due to fairy's programming model,
called filter IF, and various built-in filters.

fairy is implemented in Ruby and inherits its high productivity and
simplicity. fairy's API is quite similar to Ruby. Therefore most
programmers who know Ruby can easily understand and use it.

Although fairy is still experimental, its basic functions work well.

Please visit us and try it:
* <http://code.google.com/p/fairy-prj/>

== Example: Distributed Word Count

require 'rubygems'
require 'fairy'

fairy = Fairy::Fairy.new
input = fairy.input "testdata.vf"

maped = input.mapf(%{|ln|
ln.split
})

grouped = map.group_by(%{|w| w})

counted = grouped.map(%q{|bag|
"#{bag.key}\t#{bag.size}"
})

counted.output "wc-out.txt"

Enjoy!

···

--
Hajime Masuda
Rakuten Institute of Technology Rakuten, Inc.
email: hajime.masuda@mail.rakuten.co.jp

#2

hi Hajime,

its quite paranoid on 1.9.2

require 'fairy'
/usr/local/lib/ruby/gems/1.9.1/gems/DeepConnect-0.4.06.006/lib/deep-connect/deep-space.rb:399:
warning: redefining `__send__' may cause serious problems
#=> true

and doc seems to fails slightly.

$ sudo gem install fairy
Successfully installed DeepConnect-0.4.06.006
Successfully installed fiber-mon-0.1.0
Successfully installed fairy-0.6.0
3 gems installed
Installing ri documentation for DeepConnect-0.4.06.006...
Installing ri documentation for fiber-mon-0.1.0...
Installing ri documentation for fairy-0.6.0...
Before reporting this, could you check that the file you're documenting
compiles cleanly--RDoc is not a full Ruby parser, and gets confused easily if
fed invalid programs.

The internal error was:

  (ArgumentError) invalid byte sequence in UTF-8

ERROR: While executing gem ... (ArgumentError)
    invalid byte sequence in UTF-8

thanks for fairy and best regards
-botp

···

On Mon, Dec 6, 2010 at 4:43 PM, Hajime Masuda <hazimem@gmail.com> wrote:

We are very happy to announce that fairy 0.6.0 has just been released!

#3

Looks like the API could use Serializable Proc.

···

On Dec 6, 3:43 am, Hajime Masuda <hazi...@gmail.com> wrote:

We are very happy to announce that fairy 0.6.0 has just been released!

fairy is a framework for distributed processing in Ruby, originally
designed at Rakuten Institute of Technology with Yukihiro Matsumoto,
the founder of Ruby.

Although fairy was inspired by MapReduce model, a well-known
programming model for distributed processing, it's more flexible and
suitable for wider use. That's due to fairy's programming model,
called filter IF, and various built-in filters.

fairy is implemented in Ruby and inherits its high productivity and
simplicity. fairy's API is quite similar to Ruby. Therefore most
programmers who know Ruby can easily understand and use it.

Although fairy is still experimental, its basic functions work well.

Please visit us and try it:
* <http://code.google.com/p/fairy-prj/>

== Example: Distributed Word Count

require 'rubygems'
require 'fairy'

fairy = Fairy::Fairy.new
input = fairy.input "testdata.vf"

maped = input.mapf(%{|ln|
ln.split

})

grouped = map.group_by(%{|w| w})

counted = grouped.map(%q{|bag|
"#{bag.key}\t#{bag.size}"

})

counted.output "wc-out.txt"

#4

Hi, botp

Thank you for your comment :slight_smile:

its quite paranoid on 1.9.2

Sorry, it should be fixed.

and doc seems to fails slightly.

I've never had such a problem.
Our test servers are Debian 5.0 and their locales are ja_JP.UTF-8.
Can you tell me your environment, such as OS, locale and so forth?

br,
Hajime

#5

Hi, Intransition

Looks like the API could use Serializable Proc.

I agree with you.
Actually, the API using String is a little uncool.
Do you know any so-called 'Serializable Proc' libraries that work stably?

And I hope original Ruby's Proc becomes serializable.

br,
Hajime

#6

$ locale; lsb_release -a; uname -an; ruby -v; gem -v
LANG=en_PH.UTF-8
LC_CTYPE="en_PH.UTF-8"
LC_NUMERIC="en_PH.UTF-8"
LC_TIME="en_PH.UTF-8"
LC_COLLATE="en_PH.UTF-8"
LC_MONETARY="en_PH.UTF-8"
LC_MESSAGES="en_PH.UTF-8"
LC_PAPER="en_PH.UTF-8"
LC_NAME="en_PH.UTF-8"
LC_ADDRESS="en_PH.UTF-8"
LC_TELEPHONE="en_PH.UTF-8"
LC_MEASUREMENT="en_PH.UTF-8"
LC_IDENTIFICATION="en_PH.UTF-8"
LC_ALL=
LSB Version: core-2.0-ia32:core-3.0-ia32:core-3.1-ia32:core-3.2-ia32:core-4.0-ia32:core-2.0-noarch:core-3.0-noarch:core-3.1-noarch:core-3.2-noarch:core-4.0-noarch
Distributor ID: Ubuntu
Description: Ubuntu 8.04.4 LTS
Release: 8.04
Codename: hardy
Linux jedi2 2.6.24-28-server #1 SMP Wed Nov 24 10:04:43 UTC 2010 i686 GNU/Linux
ruby 1.9.2p0 (2010-08-18) [i686-linux]
1.3.7

thank you and best regards
-botp

···

On Tue, Dec 7, 2010 at 1:21 PM, Hajime Masuda <hazimem@gmail.com> wrote:

Can you tell me your environment, such as OS, locale and so forth?