Ruby game server (concurrency and speed)

Hello there,

I'm writing a game server for flash based game. Currently there is MySQL
+ activerecord + activesupport + eventmachine.

At timed intervals (lets say 1 minute) actions should be done:

* buildings construction status should be updated.
* resource entries should be updated.
* technologies should be updated.
* unit construction status should be updated.
* unit movement status should be updated.

All those operations do heavy-duty actions with SQL and objects so they
take a while.

For example with 500 000 resource entries updating them all takes about
27 seconds.

500 000 resource entries are needed for 33 000 players.

I imagine that other status updates will add up to this time
considerably.

While eventmachine allows you to spawn green ruby threads, it seems that
mysql gem is blocking.

So far I had these ideas:
1) run things in linear fashion, ensure that everything runs pretty
speedy (under 1-2 seconds) because incoming requests would be blocked by
eventmachine. Buy another server if it doesn't.
2) run things in concurrent fashion, ensure that everything fits into
period (1 minute). Run updaters as separate ruby processes. Pretty
complex to code.

Perhaps there would be any other ideas that I've missed?

Thank you.

···

--
Posted via http://www.ruby-forum.com/.

There is an asynchronous mysql gem, but I haven't done much with it.

Link: http://www.espace.com.eg/neverblock/blog/2008/08/28/neverblock-mysql-support/

-Jonathan Nielsen

···

2010/2/3 Artūras Šlajus <x11@arturaz.net>:

While eventmachine allows you to spawn green ruby threads, it seems that
mysql gem is blocking.

A few observations based on my own experiences of writing real-time systems. YMMV.

To start with there are many architectural assumptions implicit in the scenario you've outlined that you should reconsider to see if they're justified, not least of which is that you're apparently trying to do all of this through a single server application. Even if you were doing this in C/C++ I'd view that as a huge disaster waiting to happen.

The rule of thumb with any "real-time" system is to break it down into as many discrete parts as you can and to have these running as separate services so that individual bottlenecks can be identified and scaled out to separate hardware when load demands.

Is it really necessary that all of these updates happen concurrently? And if not how would you prioritise the updates? Is it possible to limit the updates to just those players who are logged on and active? Or perhaps a slightly larger group of which they're a subset? Perhaps you can even partition the game world into zones where updates to an individual zone can be scheduled in isolation.

Examine your query logs and figure out how much overhead you're paying for each type of query you run. It could be that for some updates you should abandon ActiveRecord and use embedded SQL where you can tailor the query to your specific requirements.

You should also look at message queues for handling your updates. This will allow you to move the heavy data processing work into separate processes and make scaling much easier to get your head around. With a bit of thought you can probably turn what appears to be an atomic update process into a staged pipeline which more effectively uses the underlying computing resources at your disposal.

Depending on how much control you have over your database configuration you could also look at sharding (the zone-splitting already mentioned is a simple example of this), data denormalisation, and decoupling the database behind a series of simple web services.

Finally don't treat writes as immediate updates of the live data as that's a very difficult contract to fulfil. Instead consider each update a discrete 'turn'. Read the wikipedia article on Shannon-Nyquist and you'll see that with the right duty cycle you can make any digital system appear continuous :slight_smile:

Also, based on the architecture you've described I'm not sure EventMachine is a good approach to your problems. It's really designed for handling i/o traffic where latency is less of a consideration than the need to handle many concurrent requests in a mostly timely fashion.

I hope there are some ideas in there that help. You might also find various useful bits of code in the presentations linked from my signature.

Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

···

On 3 Feb 2010, at 14:52, Artūras Šlajus wrote:

Hello there,

I'm writing a game server for flash based game. Currently there is MySQL
+ activerecord + activesupport + eventmachine.

At timed intervals (lets say 1 minute) actions should be done:

* buildings construction status should be updated.
* resource entries should be updated.
* technologies should be updated.
* unit construction status should be updated.
* unit movement status should be updated.

All those operations do heavy-duty actions with SQL and objects so they
take a while.

For example with 500 000 resource entries updating them all takes about
27 seconds.

500 000 resource entries are needed for 33 000 players.

I imagine that other status updates will add up to this time
considerably.

While eventmachine allows you to spawn green ruby threads, it seems that
mysql gem is blocking.

So far I had these ideas:
1) run things in linear fashion, ensure that everything runs pretty
speedy (under 1-2 seconds) because incoming requests would be blocked by
eventmachine. Buy another server if it doesn't.
2) run things in concurrent fashion, ensure that everything fits into
period (1 minute). Run updaters as separate ruby processes. Pretty
complex to code.

Perhaps there would be any other ideas that I've missed?

----
raise ArgumentError unless @reality.responds_to? :reason

While eventmachine allows you to spawn green ruby threads, it seems that
mysql gem is blocking.

Yeah check out mysqlplus, and possibly

-r

···

--
Posted via http://www.ruby-forum.com/\.

I use mysqlplus instead of mysql gem. It's like a replacement, but it allows
async operations (non-blocking) so a thread performing an async DB query
doesn't block other threads.

···

El Miércoles, 3 de Febrero de 2010, Jonathan Nielsen escribió:

2010/2/3 Artūras Šlajus <x11@arturaz.net>:
> While eventmachine allows you to spawn green ruby threads, it seems that
> mysql gem is blocking.

There is an asynchronous mysql gem, but I haven't done much with it.

Link:
http://www.espace.com.eg/neverblock/blog/2008/08/28/neverblock-mysql-suppo
rt/

--
Iñaki Baz Castillo <ibc@aliax.net>

Another idea my friend suggested was to only calculate actual values
when requested.

So instead of recalculating all the resource entries each minute only
recalc those, which are requested (and that is not very often).

It sounds pretty reasonable for me.

The only thing being on_complete callbacks for units, technologies and
buildings.

···

--
Posted via http://www.ruby-forum.com/.

You can calculate when actions start the ETA for completion and then have a timed queue which check their completion at that point in time and then reschedule if necessary. These kinds of scheduling systems have all kinds of applicability elsewhere as well.

Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net
http://www.linkedin.com/in/eleanormchugh

···

On 3 Feb 2010, at 17:18, Artūras Šlajus wrote:

Another idea my friend suggested was to only calculate actual values
when requested.

So instead of recalculating all the resource entries each minute only
recalc those, which are requested (and that is not very often).

It sounds pretty reasonable for me.

The only thing being on_complete callbacks for units, technologies and
buildings.