Camping or Merb

ruby-talk
1

Hi folks,

We will be creating a heavy traffic site that uses Ajax, Flash video
and has heavy persistent database connectivity. Database will be most
probably PostgreSQL.

Usage requires speed close to real time, bandwidth or latency will not
be an issue. I'm pretty sure we can accomplish all of this if we stick
to pure Ruby. But we would like to create it using a web framework and
take advantage of Swiftiply, Erubis, Mongrel etc. .

My research shows that we will be needing Swiftiplied Mongrel and
should avoid Activerecord (use Og?). The closest frameworks in mind
are Camping and Merb. Which of these two would be the best option for
us?

        pǝ

2

My research shows that we will be needing Swiftiplied Mongrel and
should avoid Activerecord (use Og?). The closest frameworks in mind
are Camping and Merb. Which of these two would be the best option for
us?

Have a look at Sequel, a lightweight ORM library for Ruby:

  Google Code Archive - Long-term storage for Google Code Project Hosting.

Sharon

3

Eduardo Tongson wrote:

Hi folks,

We will be creating a heavy traffic site that uses Ajax, Flash video
and has heavy persistent database connectivity. Database will be most
probably PostgreSQL.

Usage requires speed close to real time, bandwidth or latency will not
be an issue. I'm pretty sure we can accomplish all of this if we stick
to pure Ruby. But we would like to create it using a web framework and
take advantage of Swiftiply, Erubis, Mongrel etc. .

My research shows that we will be needing Swiftiplied Mongrel and
should avoid Activerecord (use Og?). The closest frameworks in mind
are Camping and Merb. Which of these two would be the best option for
us?

        pǝ

Camping is very hard to modify, so merb is probably better choice, swiftiply is nice but it's new solution and hard to tell how well it works yet.

anyway i would recommend nginx as load balancer and for serving static files and flv's, evented mongrel for backend, merb for backend, erubis for templates, and sequel fro postgresql database - supports connection pooling and persistent connections.

i tested my own micro-framework based on evented mongrel handlers and it was working 20 (yes 20) times faster then standard rails setup.

if you are interested in high availbility and perfomance tunning please contact me.

4

I'd suggest looking at Merb, or maybe Ramaze, or my framework, IOWA. IOWA is definitely the fastest of the bunch. Merb is #2 in speed, is flexible, and has an active community. Ramaze is a newcomer. It's kind of to Nitro what Merb is to Rails. It's slower than Merb, but is a very nice framework with some good ideas.

For ORM layers, you should do some benchmarks. AR is not particularly fast. Og...might be faster. Maybe not. I'm not really sure. Also check Sequel and you might try my ORM, Kansas (http://withruby.com/kansas for a more recent snapshot). It's fairly lightweight and fast, and quite capable, and has been in active use on production sites for years.

I originally wrote Swiftiply to use with IOWA, so it does work very well with it. Regardless of what you choose to do, though, feel free to contact me if you have any questions or need any help.

Kirk Haines

···

On Wed, 12 Sep 2007, Eduardo Tongson wrote:

We will be creating a heavy traffic site that uses Ajax, Flash video
and has heavy persistent database connectivity. Database will be most
probably PostgreSQL.

Usage requires speed close to real time, bandwidth or latency will not
be an issue. I'm pretty sure we can accomplish all of this if we stick
to pure Ruby. But we would like to create it using a web framework and
take advantage of Swiftiply, Erubis, Mongrel etc. .

My research shows that we will be needing Swiftiplied Mongrel and
should avoid Activerecord (use Og?). The closest frameworks in mind
are Camping and Merb. Which of these two would be the best option for
us?

5

Eduardo Tongson wrote:

We will be creating a heavy traffic site that uses Ajax, Flash video
and has heavy persistent database connectivity. Database will be most
probably PostgreSQL.

Usage requires speed close to real time, bandwidth or latency will not
be an issue. I'm pretty sure we can accomplish all of this if we stick
to pure Ruby. But we would like to create it using a web framework and
take advantage of Swiftiply, Erubis, Mongrel etc. .

My research shows that we will be needing Swiftiplied Mongrel and
should avoid Activerecord (use Og?). The closest frameworks in mind
are Camping and Merb. Which of these two would be the best option for
us?

I strongly recommend *not* using swiftiply. It's basically unmaintained.
I posted a reproducible bug a long time ago, and it was ignored.

There's little point going with Merb, because that's basically
unmaintained now as well - it's been merged into Rails 3.

Camping is an example of a "micro framework". It uses too much
metaprogramming magic for my taste. If you want a micro framework, I'd
suggest Sinatra instead.

Rails is obviously an example of a big framework ("macro framework"?).
Whether you use that will depend on how much you want to make use of the
structuring and helper methods it provides. It's a love-or-hate thing,
so if you love it, use it. I would not worry about performance when
using either of these: they both sit on top of Rack, and if you have
specific methods which require the lowest possible latency, you can
write them as Rack methods (under Rails they call this "Rails Metal")

Mongrel works fine but there are newer derivatives you should look at -
thin, unicorn, rainbows!

With any of those you'll probably want an apache or nginx proxy sitting
in front. Alternatively look at Phusion Passenger which runs directly
within apache or nginx.

HTH,

Brian.

···

--
Posted via http://www.ruby-forum.com/\.

6

The most severe problem I had with swiftiply was when I mounted an app
on a subdirectory with :prefix. If a request came outside that prefix
then it crashed immediately. Details at
http://rubyforge.org/tracker/index.php?func=detail&aid=14949&group_id=3637&atid=14008

I also submitted two minor patches:
http://rubyforge.org/tracker/index.php?group_id=3637&atid=14010
The pidfile fix certainly hasn't been applied in the current git code;
the other patch looks like it's no longer relevant.

I actually quite like the concept of swiftiply, where you can have a
dynamic pool of workers which can "pull" HTTP requests from a
multiplexor, but it's a non-standard protocol and IMO ties you too
heavily to this one implementation. If you use regular HTTP proxying
then at least it's HTTP, and you have a whole range of both proxies and
HTTP servers to choose from. If you want a dynamic worker pool you have
at least Passenger and FastCGI to choose from.

···

--
Posted via http://www.ruby-forum.com/.

7

so you prefer nginx over swiftiply? Have you tried your mockup with
swiftiply? Just wondering.
-Roger

···

--
Posted via http://www.ruby-forum.com/.

8

Hi Brian,

btw.. What was the Swiftiply bug?

MarkT

9

If you need SSL, then I believe you will need something like nginx (apache,
lighttpd, pound, etc) to handle the certificates. Also I doubt that you will
beat the speed of nginx for serving static content, so that would also be a
consideration.

Jeff

···

On 9/14/07, Roger Pack <rogerpack2005@gmail.com> wrote:

so you prefer nginx over swiftiply? Have you tried your mockup with
swiftiply? Just wondering.

10

Me?

I wrote Swiftiply, and yes, I use it in lieu of nginx or Apache on several sites, now.

Kirk Haines

···

On Sat, 15 Sep 2007, Roger Pack wrote:

so you prefer nginx over swiftiply? Have you tried your mockup with
swiftiply? Just wondering.

11

Swiftiply has SSL on the horizon. EventMachine will handle SSL connections, so that is not a problem. There are just a few config management issues in the way of full fledged support. I have it on my roadmap for the 0.7.0 release.

Kirk Haines

···

On Sat, 15 Sep 2007, Jeff Barczewski wrote:

If you need SSL, then I believe you will need something like nginx (apache,
lighttpd, pound, etc) to handle the certificates. Also I doubt that you will
beat the speed of nginx for serving static content, so that would also be a
consideration.

12

Apologies for the back to back posts. On static file handling, you are right. nginx is about as fast as it gets. But that's fine. Swiftiply's static file handling doesn't have to be that fast to be fast enough. My target range is 8000 to 9000 small static files per second, and at least 100 megabytes per second of data transfer for large static files, on my test machine. That is the area the threshold that I am working to continue to meet or beat as I work on Swiftiply. It's not nginx speeds, but it's better than Apache speeds (out of a single process, too), and it's fast enough to saturate a gigabit port if one is doing big files, and a 100mbs port if one is doing small files, so it's fast enough for the vast majority of deployments.

Kirk Haines

···

On Sat, 15 Sep 2007, Jeff Barczewski wrote:

If you need SSL, then I believe you will need something like nginx (apache,
lighttpd, pound, etc) to handle the certificates. Also I doubt that you will
beat the speed of nginx for serving static content, so that would also be a
consideration.

13

so you prefer nginx over swiftiply? Have you tried your mockup with
swiftiply? Just wondering.

Me?

I wrote Swiftiply, and yes, I use it in lieu of nginx or Apache on
several
sites, now.

Kirk Haines

I guess the concern would be that using swiftiply to handle static
content might be a little slower than nginx. Any thoughts?

If this were a speed problem, a possible fix might be to add to
EventMachine a call along the lines of 'here's a file's path, I want you
to pipe that file to this socket' or something. Then maybe swiftiply
could get about as fast as C based. If it were slower. Just some
random thoughts. Thanks Kirk.

-Roger

···

--
Posted via http://www.ruby-forum.com/\.

14

How do you use Swiftcore IOWA? Is it better than using IOWA with HA
Proxy with evented mongrel?

BTW http://iowa.swiftcore.org is not responding as of Sat Sep 15
02:16:36 UTC 2007.

···

On 9/15/07, khaines@enigo.com <khaines@enigo.com> wrote:

On Sat, 15 Sep 2007, Jeff Barczewski wrote:

> If you need SSL, then I believe you will need something like nginx (apache,
> lighttpd, pound, etc) to handle the certificates. Also I doubt that you will
> beat the speed of nginx for serving static content, so that would also be a
> consideration.

Apologies for the back to back posts. On static file handling, you are
right. nginx is about as fast as it gets. But that's fine. Swiftiply's
static file handling doesn't have to be that fast to be fast enough. My
target range is 8000 to 9000 small static files per second, and at least
100 megabytes per second of data transfer for large static files, on my
test machine. That is the area the threshold that I am working to
continue to meet or beat as I work on Swiftiply. It's not nginx speeds,
but it's better than Apache speeds (out of a single process, too), and
it's fast enough to saturate a gigabit port if one is doing big files, and
a 100mbs port if one is doing small files, so it's fast enough for the
vast majority of deployments.

Kirk Haines

15

EventMachine has that feature now. Kirk and I were discussing maybe to add
an in-memory cache for smaller static files and a few other things too.

EM has some advanced features in it that are intended to improve the
responsiveness and fairness of sites that serve really huge streaming HTTP
responses. But of course that's a different thing. I've used those to
improve the performance of my evented LDAP server.

···

On 9/17/07, Roger Pack <rogerpack2005@gmail.com> wrote:

If this were a speed problem, a possible fix might be to add to
EventMachine a call along the lines of 'here's a file's path, I want you
to pipe that file to this socket' or something. Then maybe swiftiply
could get about as fast as C based. If it were slower. Just some
random thoughts. Thanks Kirk.

-Roger
-

16

A little slower than nginx can still be very fast.

My design goals are to be in the 8k-9k files per second range on my development hardware, for small files, and preferably > 120 megabytes per second for large files.

It is already a somewhat optimized path that lets it be that fast. Part of what I have been working on over the last month (ugh), along with HTTP caching support, are some changes to further streamline and optimize that path. However, after a point, additional speed is mostly irrelevant because one has already exceeded the bandwidth of one's port. If you have a 100mbs port, you're going to stream large files at a small percentage of 120 megabytes per second, anyway, and even the typical small image files that are common on web sites are going to saturate your bandwidth at a speed less than 8k/second, so the fact that nginx can do 16k/second isn't so meaningful, most of the time.

That said....I _think_ small files handling may bump up into the 11k to 12k/second range on my hardware. Large file streaming will remain about the same. I am tinkering with it as I type this, in fact.

I'm also tinkering with a microframework, tentatively labeled Wisteria (might be something up at wisteria.swiftcore.org later this week), based off a lot of this work. It's aimed at things which need to be done very fast, or which require a bunch of long running connections (file uploads and downloads).

At its simplest, an earlier version of it that I was playing with last week would do 16k "Hello World" type requests per second. I think that if this pans out (it's rather directly derived from other Swiftiply work), it'll probably end up doing around 11 thousand to 12 thousand simple requests per second, and should be able to saturate a gigabit port, for upload or download handling, on one or maybe two processes.

Kirk Haines

···

On Tue, 18 Sep 2007, Roger Pack wrote:

I guess the concern would be that using swiftiply to handle static
content might be a little slower than nginx. Any thoughts?

If this were a speed problem, a possible fix might be to add to
EventMachine a call along the lines of 'here's a file's path, I want you
to pipe that file to this socket' or something. Then maybe swiftiply
could get about as fast as C based. If it were slower. Just some
random thoughts. Thanks Kirk.

17

I'll try to reply to few previous post here.

I didn't test swiftyply very much, and i didn't do benchmarks comparing to nginx so i have no idea how speed comparsion looks.

But i DID test nginx vs. apache vs. apache2 vs. light and nginx always wins :slight_smile:

now mongrel vs. evented mongrel - evented mongrel wins BIG

clasic mongrel - forking mongrel (my own modification that makes mongrel act like apache - fork on request) - a little bit slower then classic, but for low traffic sites - much better choice then cluster - you only need one mongrel instance in your memory

mongrel vs. nginx on static files - nginx wins - but it's power is only needed when working with lots of small files. i also have moded mongrel that is designed to handle static pages - it allows caching most used files in memory - like css styles and main page, i'm planing to improve it with sendfile syscall for speed on files that are not in cache and mmap on things that are in cache.
BUT all this improving is only for my fun and pleasure becouse you are more likely to run out of bandwidth before speed will be an issue

now frameworks - i didn't test all and mostly I'm basing this list on my old benchmarks, what khaines said, and other things found on network
Order from slowest to fastest

- rails
- nitro
- ramazee
- merb - almost 15 times faster then rails in my tests
- iowa

somewhere between merb and iowa probably would be my micro-framework, still unreleased - but i'm geting into it and it'll be availble this week i think for alfa

now ORM
- AR
- Og
- Sequel / Kansas - no idea which one is faster i didn't test them

sequel is 2-3 times faster then AR on small and frequent queries, but have nice features like connection pooling for postgres which gives it even more advantege.

Okey that would be all in terms of speed, but remember that usually lower level you work on - more time you spent and more bugs you make, we were considering using somethning other then rails in our projects and we found out that if project doesn't REALLY need speed, costs of adopting new technology, training, testing etc. exceed what we would save on hardware.

cheers

18

A little slower than nginx can still be very fast.

My design goals are to be in the 8k-9k files per second range on my
development hardware, for small files, and preferably > 120 megabytes
per
second for large files.

Wow that would be fast: 120MB/s~1Gb/s.

That said....I _think_ small files handling may bump up into the 11k to
12k/second range on my hardware. Large file streaming will remain about
the same. I am tinkering with it as I type this, in fact.

Let us know if it works out and maybe swiftiply really will become the
default for Rails serving :slight_smile:
Cheers.
-Roger

···

--
Posted via http://www.ruby-forum.com/\.