# EventMachine.defer and ActiveRecord connection pool? **URL:** \ **Category:** ruby-talk\ **Created:** [9 September 2009 20:37 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313 "2009-09-09T20:37:54Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Arturas\_Slajus](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@Arturas\_Slajus](https://rubytalk.org/u/Arturas_Slajus)\ **Post date:** [9 September 2009 20:37 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313/1 "2009-09-09T20:37:54Z")
Hello, I'm writing a server with EM. I'm having some long-running tasks, so I'm deferring those requests using EM.defer. However, after 5 such deferred requests AR suddently stops working... I'm using AR 2.3.4, so concurrency shouldn't be a problem. Gems configuration: REQUIRED\_GEMS = [   ['mysql', '\>=2.7.0'],   ['activerecord', '=2.3.4'],   ['activesupport', '=2.3.4'],   ['eventmachine', '=0.12.6'],   ['json', '\>=1.1.6'],   "daemons" ] Anyway, as you can see it happens after 5 requests (just simulated action, one select query): [2009-09-09 23:32:35|main|info] Starting server... [2009-09-09 23:32:36|main|info] [Resources manager] [Resources manager END] Time taken: 0.031 seconds [2009-09-09 23:32:37|main|info] [Resources manager] [Resources manager END] Time taken: 0.016 seconds [2009-09-09 23:32:38|main|info] [Resources manager] [Resources manager END] Time taken: 0.000 seconds [2009-09-09 23:32:39|main|info] [Resources manager] [Resources manager END] Time taken: 0.016 seconds [2009-09-09 23:32:40|main|info] [Resources manager] [Resources manager END] Time taken: 0.000 seconds [2009-09-09 23:32:41|main|info] [Resources manager] [Resources manager END with EXCEPTION] #\ I'm running this code: EventMachine::run {   EventMachine::start\_server "0.0.0.0", 843, FlashPolicyServer   EventMachine::start\_server "0.0.0.0", CONFIG['port'], GameServer   # Resources manager   EventMachine::PeriodicTimer.new(CONFIG['resources\_manager.period']) do     SingletonBlock.run("Resources manager", nil, nil, :defer =\> true) do       "Time taken: %4.3f seconds" % Benchmark.realtime { ResourcesEntry.tick }     end   end } Where SingletonBlock is: # Class for running one block of same name at a time class SingletonBlock   @@running = {}   @@lock = Mutex.new   class \<\< self     def run(name, operation=nil, callback=nil, options={}, &block)       unless running?(name)         operation ||= block         options.reverse\_merge!(:defer =\> true)         if options[:defer]           EventMachine.defer(             proc do               SingletonBlock.started(name)               LOGGER.block(name) { operation.call }             end,             proc do               LOGGER.block("#{name} CALLBACK") { callback.call } if callback               SingletonBlock.finished(name)             end           )         else           SingletonBlock.started(name)           LOGGER.block(name) { operation.call }           LOGGER.block("#{name} CALLBACK") { callback.call } if callback           SingletonBlock.finished(name)         end       end     end     def running?(name)       @@lock.synchronize do         ! @@running[name].nil?       end     end     def started(name)       @@lock.synchronize do         @@running[name] = true       end     end     def finished(name)       @@lock.synchronize do         @@running.delete(name)       end     end   end end If I don't use EM.defer - everything works fine. Googled - nothing found. Any ideas? > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Chuck\_Remes](https://avatars.discourse-cdn.com/v4/letter/c/3be4f8/32.png) [@Chuck\_Remes](https://rubytalk.org/u/Chuck_Remes)\ **Post date:** [9 September 2009 21:07 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313/2 "2009-09-09T21:07:34Z")
I have a guess. I guess that your CONFIG['resources\_manager.period'] is set to 0 or quite close to it. If so, the PeriodicTimer is probably firing on every "crank" of the EM reactor which is deferring more and more of your singleton blocks. There isn't any time left over to clean up connections to the ActiveRecord connection pool and you are running out. Trying making your 'resources\_manager.period' 1 second and see if it still blows up. BTW, EM#defer uses a threadpool (size of 20). It enqueues each #defer request to a Queue (thread safe). As each thread in the pool completes its task it pops the next one off the queue. If you flood the queue (which I suspect) then the newest tasks are going to starve while waiting for the previous ones to complete. As each task completes, try outputting the number of tasks in your @@running hash and see if it is a large number. cr > **···** > > On Sep 9, 2009, at 3:37 PM, Artūras Šlajus wrote: > > > Hello, > > > > I'm writing a server with EM. I'm having some long-running tasks, so I'm > > deferring those requests using EM.defer. > > > > However, after 5 such deferred requests AR suddently stops working... > > I'm using AR 2.3.4, so concurrency shouldn't be a problem. Gems > > configuration: > > > > REQUIRED\_GEMS = [ > > ['mysql', '\>=2.7.0'], > > ['activerecord', '=2.3.4'], > > ['activesupport', '=2.3.4'], > > ['eventmachine', '=0.12.6'], > > ['json', '\>=1.1.6'], > > "daemons" > > ] > > > > Anyway, as you can see it happens after 5 requests (just simulated > > action, one select query): > > > > [2009-09-09 23:32:35|main|info] Starting server... > > [2009-09-09 23:32:36|main|info] [Resources manager] > > [Resources manager END] Time taken: 0.031 seconds > > [2009-09-09 23:32:37|main|info] [Resources manager] > > [Resources manager END] Time taken: 0.016 seconds > > [2009-09-09 23:32:38|main|info] [Resources manager] > > [Resources manager END] Time taken: 0.000 seconds > > [2009-09-09 23:32:39|main|info] [Resources manager] > > [Resources manager END] Time taken: 0.016 seconds > > [2009-09-09 23:32:40|main|info] [Resources manager] > > [Resources manager END] Time taken: 0.000 seconds > > [2009-09-09 23:32:41|main|info] [Resources manager] > > [Resources manager END with EXCEPTION] > > #\ > ould not obtain a database connection within 5 seconds. The max pool > > size is cu > > rrently 5; consider increasing it.\> ---
**Author:** ![Arturas\_Slajus](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@Arturas\_Slajus](https://rubytalk.org/u/Arturas_Slajus)\ **Post date:** [9 September 2009 21:14 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313/3 "2009-09-09T21:14:01Z")
Chuck Remes wrote: > I have a guess. I guess that your CONFIG['resources\_manager.period'] > is set to 0 or quite close to it. If so, the PeriodicTimer is probably > firing on every "crank" of the EM reactor which is deferring more and > more of your singleton blocks. There isn't any time left over to clean > up connections to the ActiveRecord connection pool and you are running > out. Nop, it's 1 second for testing. It happens even if you set it to 1 minute. Loads of time to clean up and still... > BTW, EM#defer uses a threadpool (size of 20). It enqueues each #defer > request to a Queue (thread safe). As each thread in the pool completes > its task it pops the next one off the queue. > > If you flood the queue (which I suspect) then the newest tasks are > going to starve while waiting for the previous ones to complete. As > each task completes, try outputting the number of tasks in your > @@running hash and see if it is a large number. The whole purpose of @@running hash is to ensure that same task won't be running in parallel. I don't need two resource managers messing up values with database, so it skips that block completely if same task is still running. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Chuck\_Remes](https://avatars.discourse-cdn.com/v4/letter/c/3be4f8/32.png) [@Chuck\_Remes](https://rubytalk.org/u/Chuck_Remes)\ **Post date:** [9 September 2009 21:28 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313/4 "2009-09-09T21:28:55Z")
Hmm, that is curious. Then here's another guess... I haven't used rails in a long time, but I recall that AR required a call to verify\_connections! or something crazy when running with threads. This is because each call opens its own connection to the database but there is (or was) no automatic close. So it could be that your threaded calls are opening up db connections and never closing them. You need to figure out how to close the connection or get AR to figure out that the others are stale or should be garbage collected. cr > **···** > > On Sep 9, 2009, at 4:14 PM, Artūras Šlajus wrote: > > > Chuck Remes wrote: > > > > > I have a guess. I guess that your CONFIG['resources\_manager.period'] > > > is set to 0 or quite close to it. If so, the PeriodicTimer is probably > > > firing on every "crank" of the EM reactor which is deferring more and > > > more of your singleton blocks. There isn't any time left over to clean > > > up connections to the ActiveRecord connection pool and you are running > > > out. > > > > Nop, it's 1 second for testing. It happens even if you set it to 1 > > minute. Loads of time to clean up and still... ---
**Author:** ![Arturas\_Slajus](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@Arturas\_Slajus](https://rubytalk.org/u/Arturas_Slajus)\ **Post date:** [9 September 2009 21:43 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313/5 "2009-09-09T21:43:22Z")
Chuck Remes wrote: > > Nop, it's 1 second for testing. It happens even if you set it to 1 > > minute. Loads of time to clean up and still... > > Hmm, that is curious. Then here's another guess... > > I haven't used rails in a long time, but I recall that AR required a > call to verify\_connections! or something crazy when running with > threads. Tried this with no luck 🙂 Thou it gave me some idea for googling... > This is because each call opens its own connection to the > database but there is (or was) no automatic close. So it could be that > your threaded calls are opening up db connections and never closing > them. You need to figure out how to close the connection or get AR to > figure out that the others are stale or should be garbage collected. It seems that ActiveRecord::Base.clear\_reloadable\_connections! in:           EventMachine.defer(             proc do               SingletonBlock.started(name)               LOGGER.block(name) { operation.call }             end,             proc do               LOGGER.block("#{name} CALLBACK") { callback.call } if callback               SingletonBlock.finished(name)               ActiveRecord::Base.clear\_reloadable\_connections!             end           ) actually helped 🙂 ([http://coderrr.wordpress.com/2009/01/12/rails-22-activerecord-connection-cleanup/\](http://coderrr.wordpress.com/2009/01/12/rails-22-activerecord-connection-cleanup/%5C)) Thanks for ideas and help! 😉 > **···** > > > On Sep 9, 2009, at 4:14 PM, Artūras Šlajus wrote: > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Chuck\_Remes](https://avatars.discourse-cdn.com/v4/letter/c/3be4f8/32.png) [@Chuck\_Remes](https://rubytalk.org/u/Chuck_Remes)\ **Post date:** [9 September 2009 21:59 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313/6 "2009-09-09T21:59:01Z")
You are welcome. This made me curious so I looked a little deeper and found a better choice for you. Look up #release\_connection in ActiveRecord::ConnectionAdapters::ConnectionPool. It does exactly what you need whereas that call to #clear\_reloadable\_connections! might be doing unnecessary work. Good luck. cr > **···** > > On Sep 9, 2009, at 4:43 PM, Artūras Šlajus wrote: > > > Chuck Remes wrote: > > > > > On Sep 9, 2009, at 4:14 PM, Artūras Šlajus wrote: > > > > > > > Nop, it's 1 second for testing. It happens even if you set it to 1 > > > > minute. Loads of time to clean up and still... > > > > > > Hmm, that is curious. Then here's another guess... > > > > > > I haven't used rails in a long time, but I recall that AR required a > > > call to verify\_connections! or something crazy when running with > > > threads. > > > > Tried this with no luck 🙂 Thou it gave me some idea for googling... > > > > > This is because each call opens its own connection to the > > > database but there is (or was) no automatic close. So it could be that > > > your threaded calls are opening up db connections and never closing > > > them. You need to figure out how to close the connection or get AR to > > > figure out that the others are stale or should be garbage collected. > > > > It seems that ActiveRecord::Base.clear\_reloadable\_connections! in: > > > >          EventMachine.defer( > >            proc do > >              SingletonBlock.started(name) > >              LOGGER.block(name) { operation.call } > >            end, > >            proc do > >              LOGGER.block("#{name} CALLBACK") { callback.call } if > > callback > >              SingletonBlock.finished(name) > >              ActiveRecord::Base.clear\_reloadable\_connections! > >            end > >          ) > > > > actually helped 🙂 > > ([http://coderrr.wordpress.com/2009/01/12/rails-22-activerecord-connection-cleanup/\](http://coderrr.wordpress.com/2009/01/12/rails-22-activerecord-connection-cleanup/%5C)) > > > > Thanks for ideas and help! 😉 ---
**Author:** ![Arturas\_Slajus](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@Arturas\_Slajus](https://rubytalk.org/u/Arturas_Slajus)\ **Post date:** [10 September 2009 12:27 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313/7 "2009-09-10T12:27:24Z")
Chuck Remes wrote: > Look up #release\_connection in > ActiveRecord::ConnectionAdapters::ConnectionPool. It does exactly what > you need whereas that call to #clear\_reloadable\_connections! might be > doing unnecessary work. Unfortunately that doesn't work 🙂           EventMachine.defer(             proc do               SingletonBlock.started(name)               LOGGER.block(name) { operation.call }             end,             proc do               LOGGER.block("#{name} CALLBACK") { callback.call } if callback               SingletonBlock.finished(name)               ActiveRecord::Base.connection\_pool.release\_connection               #ActiveRecord::Base.clear\_reloadable\_connections!             end           ) Still hangs 🙂 > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Pete\_Brumm](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@Pete\_Brumm](https://rubytalk.org/u/Pete_Brumm)\ **Post date:** [16 April 2010 19:59 UTC](https://rubytalk.org/t/eventmachine-defer-and-activerecord-connection-pool/55313/8 "2010-04-16T19:59:07Z")
Artūras Šlajus wrote: > Chuck Remes wrote: > > > Look up #release\_connection in > > ActiveRecord::ConnectionAdapters::ConnectionPool. It does exactly what > > you need whereas that call to #clear\_reloadable\_connections! might be > > doing unnecessary work. > > Unfortunately that doesn't work 🙂 > >           EventMachine.defer( >             proc do >               SingletonBlock.started(name) >               LOGGER.block(name) { operation.call } >             end, >             proc do >               LOGGER.block("#{name} CALLBACK") { callback.call } if > callback >               SingletonBlock.finished(name) >               ActiveRecord::Base.connection\_pool.release\_connection >               #ActiveRecord::Base.clear\_reloadable\_connections! >             end >           ) > > Still hangs 🙂 using ActiveRecord::Base.connection\_pool.release\_connection with jruby and with delayed\_jobs running in same jvm I was seeing hanging on long running methods with lots of queries. I changed my invoke\_job to the following and no more lockups     def invoke\_job       begin         payload\_object.perform       ensure         ActiveRecord::Base.connection\_pool.release\_connection       end     end Active Record's connection pool uses the thread\_id to checkout connections. if event machine is switching threads as it sees fit than this probably won't work. you may have to rework the connection pool to allow a better checkout checkin process. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).