# Strange interference between LSAPI, popen3 and MySQL **URL:** \ **Category:** ruby-talk\ **Created:** [4 March 2007 18:39 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855 "2007-03-04T18:39:51Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Andreas\_S1](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@Andreas\_S1](https://rubytalk.org/u/Andreas_S1)\ **Post date:** [4 March 2007 18:39 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/1 "2007-03-04T18:39:51Z")
Hi, I am experiencing a very strange problem in my Rails app. I use popen3 to execute an external program. Everything works fine in Mongrel, but when I use the app with LSAPI, after popen3 all MySQL queries fail with "Mysql::Error (MySQL server has gone away)". Once I comment out popen3 it works fine again. Does anyone have the slightest idea about what goes wrong here? Thanks Andreas > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Andreas\_S1](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@Andreas\_S1](https://rubytalk.org/u/Andreas_S1)\ **Post date:** [4 March 2007 18:57 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/2 "2007-03-04T18:57:38Z")
The problem is easy to reproduce in any Rails app using LSAPI and MySQL. Add the following to a controller action:     require 'open3'     Open3.popen3('/usr/bin/ls') {|stdout, stdin, stderr| stdin.read } The stdin.read is important. > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Ara.T.Howard6](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Ara.T.Howard6](https://rubytalk.org/u/Ara.T.Howard6)\ **Post date:** [4 March 2007 19:49 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/3 "2007-03-04T19:49:44Z")
guessing, but popen3 uses fork. forking with and open db handle is undefined for almost any db lib - of course people ignore this. i did, and had all sorts of issues. you can read about them here    [Linux Clustering with Ruby Queue: Small Is Beautiful | Linux Journal](http://www.linuxjournal.com/article/7922) i bet what happens is that the parent (rails) has setup an at\_exit handler to close the db on exit. when popen forks and does it's thing in the child, the child's death kills the db handle (at\_exit handlers being inherited - of course). this shows what i'm talking about    harp:~ \> cat a.rb    at\_exit{ puts 'close connection \_\_A\_\_' }    fork{ }    Process.wait    class Object      alias\_method '\_\_fork\_\_', 'fork'      def fork \*a, &b        at\_exit{ exit! }        \_\_fork\_\_ \*a, &b      end    end    module Kernel      alias\_method '\_\_fork\_\_', 'fork'      def fork \*a, &b        at\_exit{ exit! }        \_\_fork\_\_ \*a, &b      end    end    at\_exit{ puts 'close connection \_\_B\_\_' }    fork{ }    Process.wait    harp:~ \> ruby a.rb    close connection \_\_A\_\_ i'd try putting this    class Object      alias\_method '\_\_fork\_\_', 'fork'      def fork \*a, &b        at\_exit{ exit! }        \_\_fork\_\_ \*a, &b      end    end    module Kernel      alias\_method '\_\_fork\_\_', 'fork'      def fork \*a, &b        at\_exit{ exit! }        \_\_fork\_\_ \*a, &b      end    end in environment.rb or something. another idea, i'm grasping, is some sort of buffer flushing thing. you could try    4242.times do |fd|      begin        IO.for\_fd(fd).flush      rescue Errno::EBADF      end    end right before the popen. wild ass guess though. good luck. -a > **···** > > On Mon, 5 Mar 2007, Andreas Schwarz wrote: > > > Hi, > > > > I am experiencing a very strange problem in my Rails app. I use popen3 > > to execute an external program. Everything works fine in Mongrel, but > > when I use the app with LSAPI, after popen3 all MySQL queries fail with > > "Mysql::Error (MySQL server has gone away)". Once I comment out popen3 > > it works fine again. > > > > Does anyone have the slightest idea about what goes wrong here? > > -- > be kind whenever possible... it is always possible. > - the dalai lama ---
**Author:** ![Andreas\_S1](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@Andreas\_S1](https://rubytalk.org/u/Andreas_S1)\ **Post date:** [4 March 2007 20:43 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/4 "2007-03-04T20:43:04Z")
> > I am experiencing a very strange problem in my Rails app. I use popen3 > > to execute an external program. Everything works fine in Mongrel, but > > when I use the app with LSAPI, after popen3 all MySQL queries fail with > > "Mysql::Error (MySQL server has gone away)". Once I comment out popen3 > > it works fine again. > > > > Does anyone have the slightest idea about what goes wrong here? > > guessing, but popen3 uses fork. forking with and open db handle is undefined > for almost any db lib - of course people ignore this. i did, and had all > sorts of issues. you can read about them here OK, good to know. I can close the MySQL connection for the popen3 call and reopen it after it to avoid problems. > i bet what happens is that the parent (rails) has setup an at\_exit handler to > close the db on exit. That makes sense, although I couldn't find any reference to at\_exit in Rails or ruby-lsapi. > i'd try putting this > >    class Object >      alias\_method '\_\_fork\_\_', 'fork' >      def fork \*a, &b >        at\_exit{ exit! } >        \_\_fork\_\_ \*a, &b >      end >    end >    module Kernel >      alias\_method '\_\_fork\_\_', 'fork' >      def fork \*a, &b >        at\_exit{ exit! } >        \_\_fork\_\_ \*a, &b >      end >    end > > in environment.rb or something. This fixes the MySQL problem (as does manually disconnecting/reconnecting), but now I have another problem: popen3 just doesn't do anything when used within LSAPI. The program is never executed. Any ideas? Who would have thought that simply calling an external program could be so difficult... Thanks Andreas > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Ara.T.Howard6](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Ara.T.Howard6](https://rubytalk.org/u/Ara.T.Howard6)\ **Post date:** [4 March 2007 21:12 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/5 "2007-03-04T21:12:47Z")
> and reopen it after it to avoid problems. > > > i bet what happens is that the parent (rails) has setup an at\_exit handler to > > close the db on exit. > > That makes sense, although I couldn't find any reference to at\_exit in > Rails or ruby-lsapi. huh - maybe it's the actuall c mysql driver. > > i'd try putting this > >    class Object > >      alias\_method '\_\_fork\_\_', 'fork' > >      def fork \*a, &b > >        at\_exit{ exit! } > >        \_\_fork\_\_ \*a, &b > >      end > >    end > >    module Kernel > >      alias\_method '\_\_fork\_\_', 'fork' > >      def fork \*a, &b > >        at\_exit{ exit! } > >        \_\_fork\_\_ \*a, &b > >      end > >    end > > > > in environment.rb or something. > > This fixes the MySQL problem (as does manually > disconnecting/reconnecting), but now I have another problem: popen3 just > doesn't do anything when used within LSAPI. The program is never > executed. Any ideas? hmmm. no. must be related though. i'm staring to thinks it's lsapi which has setup all the funky at\_exit handlers, and that this is causing you grief... > Who would have thought that simply calling an external program could be > so difficult... me 😉 ultimately, in ruby queue, i had to setup a drb process before setting up db connections. then i used that drb process to do all forking/waiting for me. here's another idea, try systemu, it does not use fork at all    gem install systemu    [http://rubyforge.org/frs/?group\_id=1024&release\_id=7721](http://rubyforge.org/frs/?group_id=1024&release_id=7721)    [http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README](http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README) unless you need realtime control over stdin/stdout/stderr it'll do the trick. regards. -a > **···** > > On Mon, 5 Mar 2007, Andreas Schwarz wrote: > -- > be kind whenever possible... it is always possible. > - the dalai lama ---
**Author:** ![Andreas\_S1](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@Andreas\_S1](https://rubytalk.org/u/Andreas_S1)\ **Post date:** [4 March 2007 21:33 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/6 "2007-03-04T21:33:09Z")
Ara wrote: > > >      def fork \*a, &b > > > > executed. Any ideas? > > hmmm. no. must be related though. i'm staring to thinks it's lsapi > which has > setup all the funky at\_exit handlers, and that this is causing you > grief... I grepped for at\_exit in the LSAPI source code, but didn't find anything. Maybe it's the stdout/stderr/stdin redirection stuff... > > Who would have thought that simply calling an external program could be > > so difficult... > > me 😉 > > ultimately, in ruby queue, i had to setup a drb process before setting > up db > connections. then i used that drb process to do all forking/waiting for > me. > > here's another idea, try systemu, it does not use fork at all > >    gem install systemu > >    [http://rubyforge.org/frs/?group\_id=1024&release\_id=7721](http://rubyforge.org/frs/?group_id=1024&release_id=7721) >    [http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README](http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README) It seems to use popen, isn't this based on fork, too? The reason I want to use popen3 is not that I need stdin/stderr, but that there is a clean way to pass arguments without having to build a command line (potential escaping problems etc.). > **···** > > > On Mon, 5 Mar 2007, Andreas Schwarz wrote: > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Ezra\_Zygmuntowicz](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@Ezra\_Zygmuntowicz](https://rubytalk.org/u/Ezra_Zygmuntowicz)\ **Post date:** [4 March 2007 23:16 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/7 "2007-03-04T23:16:04Z")
Hi~ > Ara wrote: > > > > >      def fork \*a, &b > > > > > > executed. Any ideas? > > > > hmmm. no. must be related though. i'm staring to thinks it's lsapi > > which has > > setup all the funky at\_exit handlers, and that this is causing you > > grief... > > I grepped for at\_exit in the LSAPI source code, but didn't find > anything. Maybe it's the stdout/stderr/stdin redirection stuff... > > > > Who would have thought that simply calling an external program could be > > > so difficult... > > > > me 😉 > > > > ultimately, in ruby queue, i had to setup a drb process before setting > > up db > > connections. then i used that drb process to do all forking/waiting for > > me. > > > > here's another idea, try systemu, it does not use fork at all > > > >    gem install systemu > > > >    [http://rubyforge.org/frs/?group\_id=1024&release\_id=7721](http://rubyforge.org/frs/?group_id=1024&release_id=7721) > >    [http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README](http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README) > > It seems to use popen, isn't this based on fork, too? > > The reason I want to use popen3 is not that I need stdin/stderr, but > that there is a clean way to pass arguments without having to build a > command line (potential escaping problems etc.). > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).   LSAPI does its own preforking of Rails processes and disconnects and reconnects the database handles after every fork. The way they manage ruby processes causes weird issues like the one you are seeing. The manager code that does the fork management is closed source so it's hard to know exactly what is going on in there. Cheers- -- Ezra Zygmuntowicz-- Lead Rails Evangelist -- ez@engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) > **···** > > On Mar 4, 2007, at 1:33 PM, Andreas Schwarz wrote: > > > > On Mon, 5 Mar 2007, Andreas Schwarz wrote: ---
**Author:** ![George\_Wang](https://avatars.discourse-cdn.com/v4/letter/g/f0a364/32.png) [@George\_Wang](https://rubytalk.org/u/George_Wang)\ **Post date:** [9 March 2007 20:37 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/8 "2007-03-09T20:37:45Z")
OK, we figured out what is wrong. "reopen" method need to be defined for "LSAPI" class since popen3 calls "reopen". LSAPI new release will address this. However, the root cause of broken MySQL connection is not LSAPI's fault, actually a bug in ruby's "open3.rb", the fix is like: def popen3(\*cmd)     pw = IO::pipe # pipe[0] for read, pipe[1] for write     pr = IO::pipe     pe = IO::pipe     pid = fork{       # child       fork{   # grandchild + begin   pw[1].close   STDIN.reopen(pw[0])   pw[0].close   pr[0].close   STDOUT.reopen(pr[1])   pr[1].close   pe[0].close   STDERR.reopen(pe[1])   pe[1].close         exec(\*cmd) + ensure + exit!(0) + end       }       exit!(0)     }     pw[0].close     pr[1].close     pe[1].close     Process.waitpid(pid)     pi = [pw[1], pr[0], pe[0]]     pw[1].sync = true     if defined? yield       begin   return yield(\*pi)       ensure   pi.each{|p| p.close unless p.closed?}       end     end     pi   end   module\_function :popen3 end Whenever an error raise in the grandchild process, this process will not be terminated properly, and DB connection was closed in this process, thus the DB connection error in the main process. It happens to both mongrel and LSAPI. With above fix, DB connection is in good shape even without the LSAPI fix, just the command was not getting executed. Best Regards, George Wang [http://www.litespeedtech.com/](http://www.litespeedtech.com/) > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Jeremy\_Kemper1](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@Jeremy\_Kemper1](https://rubytalk.org/u/Jeremy_Kemper1)\ **Post date:** [4 March 2007 23:45 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/9 "2007-03-04T23:45:57Z")
It's all under a BSD-looking license:   gem install ruby-lsapi jeremy > **···** > > On 3/4/07, Ezra Zygmuntowicz \ wrote: > > >         LSAPI does its own preforking of Rails processes and disconnects > > and > > reconnects the database handles after every fork. The way they manage > > ruby processes causes weird issues like the one you are seeing. The > > manager code that does the fork management is closed source so it's > > hard to know exactly what is going on in there. ---
**Author:** ![Ezra\_Zygmuntowicz](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@Ezra\_Zygmuntowicz](https://rubytalk.org/u/Ezra_Zygmuntowicz)\ **Post date:** [5 March 2007 00:15 UTC](https://rubytalk.org/t/strange-interference-between-lsapi-popen3-and-mysql/35855/10 "2007-03-05T00:15:38Z")
You're right, my bad. I didn't realize they distributed the C code that does the management as well. -- Ezra Zygmuntowicz-- Lead Rails Evangelist -- ez@engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) > **···** > > On Mar 4, 2007, at 3:45 PM, Jeremy Kemper wrote: > > > On 3/4/07, Ezra Zygmuntowicz \ wrote: > > > > >         LSAPI does its own preforking of Rails processes and disconnects > > > and > > > reconnects the database handles after every fork. The way they manage > > > ruby processes causes weird issues like the one you are seeing. The > > > manager code that does the fork management is closed source so it's > > > hard to know exactly what is going on in there. > > > > It's all under a BSD-looking license: > > gem install ruby-lsapi > > > > jeremy