Sandboxing eval'd code

I'm working on a web app with complicated and ever changing business
rules.

On any given day, a new rule may need to be introduced - quickly.
There's not enough commonality between them to factor out and just make
configuration changes. So the developers are always adding and
modifying, which of course makes reliability and testing difficult.

I thought of the following solution: New business rules can be coded in
Ruby and serialized into the database, along with configuration options
as to when they apply. (The rules look at a bunch of things and return
either continue or abort - so making hooks for them is easy. They
should never modify *anything* - just return a value.)
The app will load them and run them in a sandbox, catching all
exceptions and also timeing them out if they fail to return.
Developers can add/modify these rules easily, without touching the core
app, and, should one have a bug, although it may give the wrong result,
it won't bring the rest of the system down.

One concern I have is that I know that eval'd code can modify class
definitions, and access other objects via ObjectSpace. Is there anyway
to eval code so that it can't change Classes and the like? A true
Sandbox.

I'm not familiar enough with $SAFE to know what it can do (although
I've heard it is not reliable). http://tryruby.hobix.com/ probably
does this, but source doesn't seem available.
http://approximity.com/cgi-bin/rubybuch_wiki/wpage.rb?nd=214 looks
relevant, but I can't figure it out.

In general, comments about this greatly appreciated.

class String
  def de_inspect
    Thread.new do
      $SAFE = 4

      de_inspect_unsafe
    end.value
  end

  def de_inspect_unsafe # But fast...
    eval(self, Module.new.module_eval{binding})
  end

  def evil_eval # Don't use it !!!
    eval(self)
  end
end

def foo ; p [self, 1] ; end

foo

"def foo ; p [self, 2] ; end".evil_eval

foo

"def foo ; p [self, 3] ; end".de_inspect_unsafe

foo

"def foo ; p [self, 4] ; end".de_inspect # Will fail...

foo

class String
  def de_inspect
    Thread.new do
      $SAFE = 4

      de_inspect_unsafe
    end.value
  end

  def de_inspect_unsafe # But fast...
    eval(self, Module.new.module_eval{binding})
  end

  def evil_eval # Don't use it !!!
    eval(self)
  end
end

def foo ; p [self, 1] ; end

foo

"def foo ; p [self, 2] ; end".evil_eval

foo

"def foo ; p [self, 3] ; end".de_inspect_unsafe

foo

"def foo ; p [self, 4] ; end".de_inspect # Will fail...

foo

eastcoastcoder@gmail.com wrote:

One concern I have is that I know that eval'd code can modify class
definitions, and access other objects via ObjectSpace. Is there anyway
to eval code so that it can't change Classes and the like? A true
Sandbox.

Try Ruby uses fork. I setup the basic environment and then fork new interpreters. I then remove all the constants and methods that could be used maliciously. remove_const and remove_method, that's the secret. However, there is some sticky business involved in trying to close up the `ancestors' hole. I don't yet have an answer that doesn't involve a hacked Ruby.

I don't use $SAFE because I didn't want the intrepid student to ever see the word `taint' in her's session. Catch my drift? (Give it a sec, it's drifting down to you right now.)

I don't use threads because I want the student to have all the power to screw up her's environment without troubling other students. In addition, I can use FreeBSD's process management to control individual sessions. To prevent fork bombs and endless loops.

But I do use threads inside each session. One thread for an IRB process, one thread for passing data through the FastCGI socket.

_why

Could you elaborate a little more on what you are showing here?
Specifically, why is it called "de_inspect_unsafe" and what is the
array in "p [self, 1]", "p [self, 2]" etc for?

I have also wondered about this issue myself and would like to see what
people have done. I'd love to see why's "try ruby" code but that is
probably a pipe dream ...

Erik Veenstra wrote:

···

class String
  def de_inspect
    Thread.new do
      $SAFE = 4

      de_inspect_unsafe
    end.value
  end

  def de_inspect_unsafe # But fast...
    eval(self, Module.new.module_eval{binding})
  end

  def evil_eval # Don't use it !!!
    eval(self)
  end
end

def foo ; p [self, 1] ; end

foo

"def foo ; p [self, 2] ; end".evil_eval

foo

"def foo ; p [self, 3] ; end".de_inspect_unsafe

foo

"def foo ; p [self, 4] ; end".de_inspect # Will fail...

foo

hello,

i have been looking for a solution to almost the same problem. but have not
found anything yet except for a solution that uses threads. just like this
code. which i have doubts about because creating threads in a web
application does not seem to scale too much.

konstantin

"Erik Veenstra" <google@erikveen.dds.nl> wrote in message
news:1139422614.523641.62400@g14g2000cwa.googlegroups.com...

···

class String
def de_inspect
   Thread.new do
     $SAFE = 4

     de_inspect_unsafe
   end.value
end

def de_inspect_unsafe # But fast...
   eval(self, Module.new.module_eval{binding})
end

def evil_eval # Don't use it !!!
   eval(self)
end
end

def foo ; p [self, 1] ; end

foo

"def foo ; p [self, 2] ; end".evil_eval

foo

"def foo ; p [self, 3] ; end".de_inspect_unsafe

foo

"def foo ; p [self, 4] ; end".de_inspect # Will fail...

foo

Hello everyone,

I'm fairly new to ruby and wanted to write a very simple cgi program,
which uploads a file and returns a converted file. Because it is so easy
(and I want to learn about ruby) I tried to do it with the standard
cgi.rb library. But I can't get the multipart_form (I need it for the
file upload) to work right. Everything works fine using the normal form
method, but with the multipart_form I always get an EOF Error 'no
content body' in the apache log. As soon as there is any input from an
multipart_form, an error occurs at: checkfile = CGI.new

Below you'll find my code (I boiled it down to the important stuff) and the
corresponding part of my apache2 error_log. I searched the web, several
mailing lists and the ruby docs, but couldn't find an answer. I also
asked on #ruby-lang in freenode, but no one there knew the cgi.rb
library very well.

Any help, on why this server error is happening, is very much
appreciated.

Bye,
     Michael <-- who hopes, that this is the right list for the question

My code:

···

------------------------------------------------------------------
#!/usr/bin/ruby -w

require "cgi"

# make_form create a very simple html form
def make_form
  make_form = CGI.new("html4")
  make_form.out {
    make_form.html {
      make_form.body { "\n" +
# make_form.form{ "\n" + # everything works using the normal form
        make_form.multipart_form{ "\n" + # but not with multipart_form to upload a file
        make_form.file_field("filename",40,500000) + "\n" +
        make_form.br +
        make_form.submit("test it") + "\n"
        }
      }
    }
  }
end

# answer should show the filename of the uploaded file
def answer(file_in)

  answer = CGI.new("html4")
  answer.out {
    answer.html {
      answer.body{ "\n" +
      answer.p{ "This should be the filename:" + file_in + "\n" } +
      answer.br
      }
    }
  }
end

begin
  checkfile = CGI.new
  parameter = checkfile['filename']
# parameter = checkfile.params['filename'][0] # this doesn't work either

rescue
  answer("ERROR: " + $!.to_s) # send eror message to browser if possible
end

# create form if no data was send to the script, otherwise answer
if parameter == ""
  make_form()
else
  answer(parameter.to_s)
end
------------------------------------------------------------------

Apache log:
------------------------------------------------------------------
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] /usr/lib/ruby/1.8/cgi.rb:979:in `read_multipart', referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] : , referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] no content body, referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] (, referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] EOFError, referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] ), referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom /usr/lib/ruby/1.8/cgi.rb:1104:in `initialize_query', referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom /usr/lib/ruby/1.8/cgi.rb:2270:in `initialize', referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom /home/httpd/cgi-bin/multiform-test.rb:25:in `answer', referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom /home/httpd/cgi-bin/multiform-test.rb:52, referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] Premature end of script headers: multiform-test.rb, referer: http://localhost/cgi-bin/multiform-test.rb
------------------------------------------------------------------

--
Michael Schilling

eMail : ruby@untiefe.de
URL : http://glcu.sf.net/

"Change my name I remain the same." - Moloko

Could you elaborate a little more on what you are showing
here?

I'm showing 3 ways of doing what you want to achieve, in
increasing order of safety. (Although the definitions of the
methods are in decreasing order of safety...) That' all.

Specifically, why is it called "de_inspect_unsafe"

Because it's not as safe as "de_inspect". And I want to trigger
some alarm bells in case one want to use it... :slight_smile:

and what is the array in "p [self, 1]", "p [self, 2]" etc
for?

If you don't get this, well, get any other string of Ruby code.
It's not important.

I have also wondered about this issue myself and would like
to see what people have done. I'd love to see why's "try
ruby" code but that is probably a pipe dream ...

Just ask him...

gegroet,
Erik V. - http://www.erikveen.dds.nl/

···

On Wed, 08 Feb 2006 10:46:54 -0800, m4dc4p wrote:

i have been looking for a solution to almost the same problem. but have not found anything yet except for a solution that uses threads. just like this code. which i have doubts about because creating threads in a web application does not seem to scale too much.

time ruby -e '100000.times { Thread.new { }.join }'

real 0m1.130s
user 0m1.124s
sys 0m0.008s

Regards,

Bill

···

From: "konsu" <konsu@hotmail.com>

hello,

try running your script from the command line. but set the url and method
environment variables first. on windows i do this:

set REQUEST_URI=/path
set REQUEST_METHOD=GET
ruby muscript.cgi .

this will allow you to avoid dealing with apache for a while, and will show
more errors too.

konstantin

"Michael Schilling" <ruby@untiefe.de> wrote in message
news:20060208203511.GA16112@calvin.physik.uni-bielefeld.de...

···

Hello everyone,

I'm fairly new to ruby and wanted to write a very simple cgi program,
which uploads a file and returns a converted file. Because it is so easy
(and I want to learn about ruby) I tried to do it with the standard
cgi.rb library. But I can't get the multipart_form (I need it for the
file upload) to work right. Everything works fine using the normal form
method, but with the multipart_form I always get an EOF Error 'no
content body' in the apache log. As soon as there is any input from an
multipart_form, an error occurs at: checkfile = CGI.new

Below you'll find my code (I boiled it down to the important stuff) and
the
corresponding part of my apache2 error_log. I searched the web, several
mailing lists and the ruby docs, but couldn't find an answer. I also
asked on #ruby-lang in freenode, but no one there knew the cgi.rb
library very well.

Any help, on why this server error is happening, is very much
appreciated.

Bye,
    Michael <-- who hopes, that this is the right list for the question

My code:
------------------------------------------------------------------
#!/usr/bin/ruby -w

require "cgi"

# make_form create a very simple html form
def make_form
make_form = CGI.new("html4")
make_form.out {
   make_form.html {
     make_form.body { "\n" +
# make_form.form{ "\n" + # everything works using the normal
form
       make_form.multipart_form{ "\n" + # but not with multipart_form
to upload a file
       make_form.file_field("filename",40,500000) + "\n" +
       make_form.br +
       make_form.submit("test it") + "\n"
       }
     }
   }
}
end

# answer should show the filename of the uploaded file
def answer(file_in)

answer = CGI.new("html4")
answer.out {
   answer.html {
     answer.body{ "\n" +
     answer.p{ "This should be the filename:" + file_in + "\n" } +
     answer.br
     }
   }
}
end

begin
checkfile = CGI.new
parameter = checkfile['filename']
# parameter = checkfile.params['filename'][0] # this doesn't work either

rescue
answer("ERROR: " + $!.to_s) # send eror message to browser if possible
end

# create form if no data was send to the script, otherwise answer
if parameter == ""
make_form()
else
answer(parameter.to_s)
end
------------------------------------------------------------------

Apache log:
------------------------------------------------------------------
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1]
/usr/lib/ruby/1.8/cgi.rb:979:in `read_multipart', referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] : , referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] no content body,
referer: http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] (, referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] EOFError, referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] ), referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
/usr/lib/ruby/1.8/cgi.rb:1104:in `initialize_query', referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
/usr/lib/ruby/1.8/cgi.rb:2270:in `initialize', referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
/home/httpd/cgi-bin/multiform-test.rb:25:in `answer', referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
/home/httpd/cgi-bin/multiform-test.rb:52, referer:
http://localhost/cgi-bin/multiform-test.rb
[Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] Premature end of
script headers: multiform-test.rb, referer:
http://localhost/cgi-bin/multiform-test.rb
------------------------------------------------------------------

--
Michael Schilling

eMail : ruby@untiefe.de
URL : http://glcu.sf.net/

"Change my name I remain the same." - Moloko

could you elaborate a bit? are these numbers good or bad? what about memory
consumption? may be i am missing something important about ruby threads, but
my impression was that spawning a thread on every request is bad.

konstantin

"Bill Kelly" <billk@cts.com> wrote in message
news:08be01c62cf2$6bd84080$6442a8c0@musicbox...

···

From: "konsu" <konsu@hotmail.com>

i have been looking for a solution to almost the same problem. but have
not found anything yet except for a solution that uses threads. just like
this code. which i have doubts about because creating threads in a web
application does not seem to scale too much.

time ruby -e '100000.times { Thread.new { }.join }'

real 0m1.130s
user 0m1.124s
sys 0m0.008s

Regards,

Bill

$ time ruby -e '(1..100000).collect{Thread.new{sleep 1}}.collect{|t| t.join}'

real 0m42.583s
user 0m37.693s
sys 0m1.828s

$ time ruby -e '(1..100000).collect{Thread.new{}.join}'

real 0m4.683s
user 0m3.984s
sys 0m0.245s

In the latter situation, the threads are not living side-by-side.
The second thread is started when the first one has finished.

The first example starts a lot of threads concurrently.

gegroet,
Erik V. -- http://www.erikveen.dds.nl/

Hi,

thank you for the suggestions. I set the variables as you suggested
(with export REQUEST_URI=/path in Linux) but always only get the form. I
don't exactly know how a string from the multipart_form looks like, so
I couldn't really test the behaviour.

When I use the normal form method, and do not set the variables as you
suggested, I get the expected result and no error from
'myscript.rb filename=something'. With the multipart_form I also get no
error, but also no output -> only the form.

I also thought, that this problem might be a misconfiguration of apache,
but a guy on #ruby-lang told me, that it is very unlikely when the
normal form method works fine.

So I can't reproduce the error without using the browser, but don't get
the expected result either.

Thanks for any help.

Bye, Michael

···

konsu wrote on thursday the 9th of February 2006: > try running your script from the command line. but set the url and method > environment variables first. on windows i do this:

set REQUEST_URI=/path
set REQUEST_METHOD=GET
ruby muscript.cgi .

this will allow you to avoid dealing with apache for a while, and will show
more errors too.

konstantin

"Michael Schilling" <ruby@untiefe.de> wrote in message
news:20060208203511.GA16112@calvin.physik.uni-bielefeld.de...
> Hello everyone,
>
> I'm fairly new to ruby and wanted to write a very simple cgi program,
> which uploads a file and returns a converted file. Because it is so easy
> (and I want to learn about ruby) I tried to do it with the standard
> cgi.rb library. But I can't get the multipart_form (I need it for the
> file upload) to work right. Everything works fine using the normal form
> method, but with the multipart_form I always get an EOF Error 'no
> content body' in the apache log. As soon as there is any input from an
> multipart_form, an error occurs at: checkfile = CGI.new
>
> Below you'll find my code (I boiled it down to the important stuff) and
> the
> corresponding part of my apache2 error_log. I searched the web, several
> mailing lists and the ruby docs, but couldn't find an answer. I also
> asked on #ruby-lang in freenode, but no one there knew the cgi.rb
> library very well.
>
> Any help, on why this server error is happening, is very much
> appreciated.
>
> Bye,
> Michael <-- who hopes, that this is the right list for the question
>
>
>
> My code:
> ------------------------------------------------------------------
> #!/usr/bin/ruby -w
>
> require "cgi"
>
> # make_form create a very simple html form
> def make_form
> make_form = CGI.new("html4")
> make_form.out {
> make_form.html {
> make_form.body { "\n" +
> # make_form.form{ "\n" + # everything works using the normal
> form
> make_form.multipart_form{ "\n" + # but not with multipart_form
> to upload a file
> make_form.file_field("filename",40,500000) + "\n" +
> make_form.br +
> make_form.submit("test it") + "\n"
> }
> }
> }
> }
> end
>
> # answer should show the filename of the uploaded file
> def answer(file_in)
>
> answer = CGI.new("html4")
> answer.out {
> answer.html {
> answer.body{ "\n" +
> answer.p{ "This should be the filename:" + file_in + "\n" } +
> answer.br
> }
> }
> }
> end
>
>
>
> begin
> checkfile = CGI.new
> parameter = checkfile['filename']
> # parameter = checkfile.params['filename'][0] # this doesn't work either
>
> rescue
> answer("ERROR: " + $!.to_s) # send eror message to browser if possible
> end
>
>
> # create form if no data was send to the script, otherwise answer
> if parameter == ""
> make_form()
> else
> answer(parameter.to_s)
> end
> ------------------------------------------------------------------
>
>
> Apache log:
> ------------------------------------------------------------------
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1]
> /usr/lib/ruby/1.8/cgi.rb:979:in `read_multipart', referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] : , referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] no content body,
> referer: http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] (, referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] EOFError, referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] ), referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
> /usr/lib/ruby/1.8/cgi.rb:1104:in `initialize_query', referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
> /usr/lib/ruby/1.8/cgi.rb:2270:in `initialize', referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
> /home/httpd/cgi-bin/multiform-test.rb:25:in `answer', referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
> /home/httpd/cgi-bin/multiform-test.rb:52, referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] Premature end of
> script headers: multiform-test.rb, referer:
> http://localhost/cgi-bin/multiform-test.rb
> ------------------------------------------------------------------
>
>
>
> --
> Michael Schilling
>
> eMail : ruby@untiefe.de
> URL : http://glcu.sf.net/
>
>
> "Change my name I remain the same." - Moloko
>
>

time ruby -e '100000.times { Thread.new { }.join }'

real 0m1.130s
user 0m1.124s
sys 0m0.008s

could you elaborate a bit? are these numbers good or bad? what about memory consumption? may be i am missing something important about ruby threads, but my impression was that spawning a thread on every request is bad.

Hi, well, being able to create, execute, and join 100,000 threads
in about one second seemed to me to indicate that Ruby has pretty
low overhead for creating threads. (2.8GHz Pentium Xeon)

How many requests per second does your server need to handle?

I don't know if there's an absolute answer to whether creating
a thread per request is good or bad. It just looks like it doesn't
take very much time. (I don't think it takes much memory either.)

Unless your server is handling thousands of requests per second, it looks like the overhead for creating a thread is fairly
insignificant. (Remember, Ruby currently has "green" threads,
so these are not actual Operating System threads being created.)

Regards,

Bill

···

From: "konsu" <konsu@hotmail.com>

"Bill Kelly" <billk@cts.com> wrote

$ time ruby -e '(1..100000).collect{Thread.new{sleep 1}}.collect{|t| t.join}'

real 0m42.583s
user 0m37.693s
sys 0m1.828s

$ time ruby -e '(1..100000).collect{Thread.new{}.join}'

real 0m4.683s
user 0m3.984s
sys 0m0.245s

In the latter situation, the threads are not living side-by-side.
The second thread is started when the first one has finished.

The first example starts a lot of threads concurrently.

Right. I didn't mean to imply having thousands of live threads
was equivalent to having one thread created and deleted serially.

In the $SAFE/eval situation, I presumed the usage would be like
the former: create one thread, and ask for its value.

Regards,

Bill

···

From: "Erik Veenstra" <pan@erikveen.dds.nl>

i was just suggesting to mimic the web server's behaviour and simulate form
submission using environment variables since cgi relies just on these
variables and nothing more. to eliminate any apache misconfiguration. in my
earlier response i suggested to set REQUEST_METHOD=GET but to thest the form
submission you really need REQUEST_METHOD=POST and also you need to set the
corresponding variables that carry the request body. i do not remember the
names. the source for cgi.rb in the ruby lib directory might help you.

konstantin

"Michael Schilling" <ruby@untiefe.de> wrote in message
news:20060208213611.GB16112@calvin.physik.uni-bielefeld.de...

···

Hi,

thank you for the suggestions. I set the variables as you suggested
(with export REQUEST_URI=/path in Linux) but always only get the form. I
don't exactly know how a string from the multipart_form looks like, so
I couldn't really test the behaviour.

When I use the normal form method, and do not set the variables as you
suggested, I get the expected result and no error from
'myscript.rb filename=something'. With the multipart_form I also get no
error, but also no output -> only the form.

I also thought, that this problem might be a misconfiguration of apache,
but a guy on #ruby-lang told me, that it is very unlikely when the
normal form method works fine.

So I can't reproduce the error without using the browser, but don't get
the expected result either.

Thanks for any help.

Bye, Michael

konsu wrote on thursday the 9th of February 2006: > >> try running your script from the command line. but set the url and method >> environment variables first. on windows i do this:

set REQUEST_URI=/path
set REQUEST_METHOD=GET
ruby muscript.cgi .

this will allow you to avoid dealing with apache for a while, and will
show
more errors too.

konstantin

"Michael Schilling" <ruby@untiefe.de> wrote in message
news:20060208203511.GA16112@calvin.physik.uni-bielefeld.de...
> Hello everyone,
>
> I'm fairly new to ruby and wanted to write a very simple cgi program,
> which uploads a file and returns a converted file. Because it is so
> easy
> (and I want to learn about ruby) I tried to do it with the standard
> cgi.rb library. But I can't get the multipart_form (I need it for the
> file upload) to work right. Everything works fine using the normal form
> method, but with the multipart_form I always get an EOF Error 'no
> content body' in the apache log. As soon as there is any input from an
> multipart_form, an error occurs at: checkfile = CGI.new
>
> Below you'll find my code (I boiled it down to the important stuff) and
> the
> corresponding part of my apache2 error_log. I searched the web, several
> mailing lists and the ruby docs, but couldn't find an answer. I also
> asked on #ruby-lang in freenode, but no one there knew the cgi.rb
> library very well.
>
> Any help, on why this server error is happening, is very much
> appreciated.
>
> Bye,
> Michael <-- who hopes, that this is the right list for the question
>
>
>
> My code:
> ------------------------------------------------------------------
> #!/usr/bin/ruby -w
>
> require "cgi"
>
> # make_form create a very simple html form
> def make_form
> make_form = CGI.new("html4")
> make_form.out {
> make_form.html {
> make_form.body { "\n" +
> # make_form.form{ "\n" + # everything works using the normal
> form
> make_form.multipart_form{ "\n" + # but not with
> multipart_form
> to upload a file
> make_form.file_field("filename",40,500000) + "\n" +
> make_form.br +
> make_form.submit("test it") + "\n"
> }
> }
> }
> }
> end
>
> # answer should show the filename of the uploaded file
> def answer(file_in)
>
> answer = CGI.new("html4")
> answer.out {
> answer.html {
> answer.body{ "\n" +
> answer.p{ "This should be the filename:" + file_in + "\n" } +
> answer.br
> }
> }
> }
> end
>
>
>
> begin
> checkfile = CGI.new
> parameter = checkfile['filename']
> # parameter = checkfile.params['filename'][0] # this doesn't work
> either
>
> rescue
> answer("ERROR: " + $!.to_s) # send eror message to browser if possible
> end
>
>
> # create form if no data was send to the script, otherwise answer
> if parameter == ""
> make_form()
> else
> answer(parameter.to_s)
> end
> ------------------------------------------------------------------
>
>
> Apache log:
> ------------------------------------------------------------------
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1]
> /usr/lib/ruby/1.8/cgi.rb:979:in `read_multipart', referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] : , referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] no content
> body,
> referer: http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] (, referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] EOFError,
> referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] ), referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
> /usr/lib/ruby/1.8/cgi.rb:1104:in `initialize_query', referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
> /usr/lib/ruby/1.8/cgi.rb:2270:in `initialize', referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
> /home/httpd/cgi-bin/multiform-test.rb:25:in `answer', referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
> /home/httpd/cgi-bin/multiform-test.rb:52, referer:
> http://localhost/cgi-bin/multiform-test.rb
> [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] Premature end
> of
> script headers: multiform-test.rb, referer:
> http://localhost/cgi-bin/multiform-test.rb
> ------------------------------------------------------------------
>
>
>
> --
> Michael Schilling
>
> eMail : ruby@untiefe.de
> URL : http://glcu.sf.net/
>
>
> "Change my name I remain the same." - Moloko
>
>

Hello again,

as I couldn't reproduce the described multipart_form error (see
description below) directly on the command-line I looked into the cgi.rb
source and found the chunk of code that raises the exception:

Original code in cgi.rb

···

----------------------------------------------------
    def read_multipart(boundary, content_length)
      params = Hash.new()
      boundary = "--" + boundary
      buf = ""
      bufsize = 10 * 1024

      # start multipart/form-data
      stdinput.binmode if defined? stdinput.binmode
      boundary_size = boundary.size + EOL.size
      content_length -= boundary_size
      status = stdinput.read(boundary_size)
      if nil == status
        raise EOFError, "no content body"
      elsif boundary + EOL != status
        raise EOFError, "bad content body"
      end
---------------------------------------------------

After playing around with this part of the code, I have found more
questions than answers. When I negate the if clause before the exception
(see code below), the expression is still true (even so, it was false
for my program, before the change in cgi.rb), and the status variable
will even be written into the apache logs with something, that looks
right (see below for the logs - in the line with 'no content body'):

So whatever I do, I can't get rid of the exception. Something seems to
be very wrong here, but I have no clue, what it could be. Is this some
kind of bug in cgi.rb?

Maybe I should try a different approach, but I like a good challenge :wink:
Any more help is very much appreciated!

Bye, Michael

Changed code in cgi.rb
---------------------------------------------------
    def read_multipart(boundary, content_length)
      params = Hash.new()
      boundary = "--" + boundary
      buf = ""
      bufsize = 10 * 1024

      # start multipart/form-data
      stdinput.binmode if defined? stdinput.binmode
      boundary_size = boundary.size + EOL.size
      content_length -= boundary_size
      status = stdinput.read(boundary_size)
      if nil != status # set this to _not_ equal
        raise EOFError, "no content body. Status: " + status.to_s # and add here the status as a string
      elsif boundary + EOL != status
        raise EOFError, "bad content body"
      end
---------------------------------------------------

apache error log with original cgi.rb code:
-----------------------------------------------------------
[Fri Feb 10 03:56:33 2006] [error] [client 129.70.170.3] /usr/lib/ruby/1.8/cgi.rb:979:in `read_multipart', referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
[Fri Feb 10 03:56:33 2006] [error] [client 129.70.170.3] : , referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
[Fri Feb 10 03:56:33 2006] [error] [client 129.70.170.3] no content body. Status: ;, referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
[Fri Feb 10 03:56:33 2006] [error] [client 129.70.170.3] (, referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
[Fri Feb 10 03:56:33 2006] [error] [client 129.70.170.3] EOFError, referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
-----------------------------------------------------------

apache error log with changed cgi.rb code:
-----------------------------------------------------------
[Fri Feb 10 03:57:02 2006] [error] [client 129.70.170.3] /usr/lib/ruby/1.8/cgi.rb:979:in `read_multipart', referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
[Fri Feb 10 03:57:02 2006] [error] [client 129.70.170.3] : , referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
[Fri Feb 10 03:57:02 2006] [error] [client 129.70.170.3] no content body. Status: Content-Disposition: form-data; name="sowas";, referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
[Fri Feb 10 03:57:02 2006] [error] [client 129.70.170.3] (, referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
[Fri Feb 10 03:57:02 2006] [error] [client 129.70.170.3] EOFError, referer: http://calvin.physik.uni-bielefeld.de/cgi-bin/pdf2eps.rb
-----------------------------------------------------------

------------- ORIGINAL MAILS ------------------------------------

konsu wrote on Thursday the 9th of February 2006:

i was just suggesting to mimic the web server's behaviour and simulate form
submission using environment variables since cgi relies just on these
variables and nothing more. to eliminate any apache misconfiguration. in my
earlier response i suggested to set REQUEST_METHOD=GET but to thest the form
submission you really need REQUEST_METHOD=POST and also you need to set the
corresponding variables that carry the request body. i do not remember the
names. the source for cgi.rb in the ruby lib directory might help you.

konstantin

"Michael Schilling" <ruby@untiefe.de> wrote in message
news:20060208213611.GB16112@calvin.physik.uni-bielefeld.de...
> Hi,
>
> thank you for the suggestions. I set the variables as you suggested
> (with export REQUEST_URI=/path in Linux) but always only get the form. I
> don't exactly know how a string from the multipart_form looks like, so
> I couldn't really test the behaviour.
>
> When I use the normal form method, and do not set the variables as you
> suggested, I get the expected result and no error from
> 'myscript.rb filename=something'. With the multipart_form I also get no
> error, but also no output -> only the form.
>
> I also thought, that this problem might be a misconfiguration of apache,
> but a guy on #ruby-lang told me, that it is very unlikely when the
> normal form method works fine.
>
> So I can't reproduce the error without using the browser, but don't get
> the expected result either.
>
>
> Thanks for any help.
>
>
> Bye, Michael
>
>
>
>
> konsu wrote on thursday the 9th of February 2006: > > > >> try running your script from the command line. but set the url and method > >> environment variables first. on windows i do this:
>>
>>
>> set REQUEST_URI=/path
>> set REQUEST_METHOD=GET
>> ruby muscript.cgi .
>
>>
>> this will allow you to avoid dealing with apache for a while, and will
>> show
>> more errors too.
>>
>> konstantin
>>
>>
>> "Michael Schilling" <ruby@untiefe.de> wrote in message
>> news:20060208203511.GA16112@calvin.physik.uni-bielefeld.de...
>> > Hello everyone,
>> >
>> > I'm fairly new to ruby and wanted to write a very simple cgi program,
>> > which uploads a file and returns a converted file. Because it is so
>> > easy
>> > (and I want to learn about ruby) I tried to do it with the standard
>> > cgi.rb library. But I can't get the multipart_form (I need it for the
>> > file upload) to work right. Everything works fine using the normal form
>> > method, but with the multipart_form I always get an EOF Error 'no
>> > content body' in the apache log. As soon as there is any input from an
>> > multipart_form, an error occurs at: checkfile = CGI.new
>> >
>> > Below you'll find my code (I boiled it down to the important stuff) and
>> > the
>> > corresponding part of my apache2 error_log. I searched the web, several
>> > mailing lists and the ruby docs, but couldn't find an answer. I also
>> > asked on #ruby-lang in freenode, but no one there knew the cgi.rb
>> > library very well.
>> >
>> > Any help, on why this server error is happening, is very much
>> > appreciated.
>> >
>> > Bye,
>> > Michael <-- who hopes, that this is the right list for the question
>> >
>> >
>> >
>> > My code:
>> > ------------------------------------------------------------------
>> > #!/usr/bin/ruby -w
>> >
>> > require "cgi"
>> >
>> > # make_form create a very simple html form
>> > def make_form
>> > make_form = CGI.new("html4")
>> > make_form.out {
>> > make_form.html {
>> > make_form.body { "\n" +
>> > # make_form.form{ "\n" + # everything works using the normal
>> > form
>> > make_form.multipart_form{ "\n" + # but not with
>> > multipart_form
>> > to upload a file
>> > make_form.file_field("filename",40,500000) + "\n" +
>> > make_form.br +
>> > make_form.submit("test it") + "\n"
>> > }
>> > }
>> > }
>> > }
>> > end
>> >
>> > # answer should show the filename of the uploaded file
>> > def answer(file_in)
>> >
>> > answer = CGI.new("html4")
>> > answer.out {
>> > answer.html {
>> > answer.body{ "\n" +
>> > answer.p{ "This should be the filename:" + file_in + "\n" } +
>> > answer.br
>> > }
>> > }
>> > }
>> > end
>> >
>> >
>> >
>> > begin
>> > checkfile = CGI.new
>> > parameter = checkfile['filename']
>> > # parameter = checkfile.params['filename'][0] # this doesn't work
>> > either
>> >
>> > rescue
>> > answer("ERROR: " + $!.to_s) # send eror message to browser if possible
>> > end
>> >
>> >
>> > # create form if no data was send to the script, otherwise answer
>> > if parameter == ""
>> > make_form()
>> > else
>> > answer(parameter.to_s)
>> > end
>> > ------------------------------------------------------------------
>> >
>> >
>> > Apache log:
>> > ------------------------------------------------------------------
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1]
>> > /usr/lib/ruby/1.8/cgi.rb:979:in `read_multipart', referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] : , referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] no content
>> > body,
>> > referer: http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] (, referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] EOFError,
>> > referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] ), referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
>> > /usr/lib/ruby/1.8/cgi.rb:1104:in `initialize_query', referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
>> > /usr/lib/ruby/1.8/cgi.rb:2270:in `initialize', referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
>> > /home/httpd/cgi-bin/multiform-test.rb:25:in `answer', referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] \tfrom
>> > /home/httpd/cgi-bin/multiform-test.rb:52, referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > [Wed Feb 08 21:08:57 2006] [error] [client 191.168.1.1] Premature end
>> > of
>> > script headers: multiform-test.rb, referer:
>> > http://localhost/cgi-bin/multiform-test.rb
>> > ------------------------------------------------------------------
>> >
>> >
>> >
>> > --
>> > Michael Schilling
>> >
>> > eMail : ruby@untiefe.de
>> > URL : http://glcu.sf.net/
>> >
>> >
>> > "Change my name I remain the same." - Moloko

the problem is in your source i'm afraid. you can call CGI.new more than once
in a script : each call attempts to parse ENV and STDIN according to the cgi
spec. check out the docs for details. the other issue is that if, and only
if, you create a form using multipar the cgi params are either StringIO or
Tempfile objects. this is a good thing. consider a 10GB file upload. you
need a way to determine this. following is one way:

   harp:~> cat a.rb

   #! /usr/local/bin/ruby

   require "cgi"

   class Upload
     attr_accessor 'cgi'

     def initialize
       self.cgi = CGI::new 'html4'
       file = get 'file'
       file ? answer : make_form
     end

     def make_form
      cgi.out {
        cgi.html {
          cgi.body { "\n" +
            cgi.multipart_form{ "\n" +
            cgi.file_field("file",40,500000) + "\n" +
            cgi.br +
            cgi.submit("test it") + "\n"
            }
          }
        }
      }
     end

     def answer
      cgi.out {
        cgi.html {
          cgi.body{ "\n" +
          cgi.p{ "This should be the filename : " + param('file').original_filename + "\n" } +
          cgi.br
          }
        }
      }
     end

     def get key
       value = param key
       value = value.read.strip if value.respond_to?('read')
       value
     end

     def param key
       cgi.params[key].first
     end
   end

   Upload::new if $0 == __FILE__

this should run with ruby 1.8 up.

inlined below is a totally simple upload script i sometimes use for personal
stuff. it allows upload/download and some other stuff.

regards.

-a

···

On Fri, 10 Feb 2006, Michael Schilling wrote:

Hello again,

as I couldn't reproduce the described multipart_form error (see
description below) directly on the command-line I looked into the cgi.rb
source and found the chunk of code that raises the exception:

--
happiness is not something ready-made. it comes from your own actions.
- h.h. the 14th dali lama

#! /usr/local/bin/ruby

require 'cgi'
require 'tempfile'
require 'ftools'
require 'uri'

cgi = CGI::new
user = ENV['REMOTE_USER'] || 'anonymous'
headers = {}
content = ''

begin
   mode = cgi.params['m'].first || 'index'
   mode = mode.read.strip if mode.respond_to? 'read'

   case mode
     when /index|^$/

       files = Dir['**/*'].select{|f| f !~ %r/html|cgi/}
       dirname = File::dirname cgi.script_name

       files.sort!{|a,b| File::stat(b).mtime <=> File::stat(a).mtime}

       table_rows =
         files.collect do |f|
           stat = File::stat f
           uri = 'http://' << cgi.host << File::join(dirname, f)
           # <a href=#{ URI::escape f } title="right click -> save as">download</a>
           <<-html
             <tr valign=top aligh=left>
               <td>
                 #{ CGI::escapeHTML f }
               </td>
               <td>
                 #{ CGI::escapeHTML stat.mtime.to_s }
               </td>
               <td>
                 #{ CGI::escapeHTML stat.size.to_s }
               </td>
               <td>
                 <a href=#{ uri } title="right click -> save (page) as">#{ uri }</a>
               </td>
             </tr>
           html
         end

       table =
         <<-html
           <table width=100%>
             <tr valign=top aligh=left>
               <td> <b><u> file </b></u> </td>
               <td> <b><u> modification time </b></u> </td>
               <td> <b><u> size </b></u> </td>
               <td> <b><u> download </b></u> </td>
             </tr>
             #{ table_rows }
           </table>
         html

       content = <<-html
         <html>
           <head>
             <title>#{ user }::files</title>
           </head>
           <body>
             <h3>#{ user }::files</h3>

             <table cellpadding=1 align=right width=100%>
               <tr align=right>
                 <td>
                   <i>to download - right click -> save as</i>
                 </td>
               </tr>
             </table>

             <hr>

             <hr>
             </table>
             <hr>
               #{ table }
             <hr>
             <h3><a href=#{ cgi.script_name }?m=upload>upload files</a></h3>
           </body>
         </html>
       html

     when /upload/
       buttons =
       8.times do
         buttons << <<-html
           <input size=100 name=file type=file>
           <hr width=30% align=left>
         html
       end
       content = <<-html
         <html>
           <head>
             <title>#{ user }::upload</title>
           </head>
           <body>
             <h3>#{ user }::upload</h3>
             <hr>
             <form action=#{ cgi.script_name } method=POST enctype=multipart/form-data>
               <input size=100 name=file type=file>
               <hr width=30% align=left>
               #{ buttons }

               <input type=submit value=upload>
               <hr width=30% align=left>

               <input type=reset value=clear>
               <hr width=30% align=left>

               <input type=hidden name=m value=put>
             <form>
             <hr>
             <h3><a href=#{ cgi.script_name }?m=index>index</a></h3>
           </body>
         </html>
       html

     when /put/
       uploaded =
       tempfiles = cgi.params['file']
       tempfiles.each do |tempfile|
         org_path = tempfile.original_filename

         local = "#{ org_path }"
         local.gsub! %r|[/\\]|,'/'
         local.gsub! %r|\s+|,'_'
         local.downcase!

         next if local =~ /^\s*$/

         begin
           File::mkpath(File::dirname(local)) rescue nil
           if tempfile.path
             File::copy tempfile.path, local
           else
             open(local,'w'){|f| f.write tempfile.read}
           end
           File::chmod 0777, local
           uploaded << [org_path, true]
         rescue
           raise
           raise tempfile.methods.inspect
           uploaded << [org_path, false]
         end
       end

       table_rows =
         uploaded.collect do |u|
           f, status = u
           s = (status ? 'success' : 'failed')
           <<-html
             <tr valign=top aligh=left>
               <td>
                 #{ CGI::escapeHTML f.inspect }
               </td>
               <td>
                 #{ s }
               </td>
             </tr>
           html
         end

       table =
         <<-html
           <table width=100%>
             <tr valign=top aligh=left>
               <td> <b><u> file </b></u> </td>
               <td> <b><u> status </b></u> </td>
             </tr>
             #{ table_rows }
           </table>
         html

       content = <<-html
         <html>
           <head>
             <title>#{ user }::uploaded</title>
           </head>
           <body>
             <h3>#{ user }::uploaded</h3>
             </table>
             <hr>
               #{ table }
             <hr>
             <h3><a href=#{ cgi.script_name }?m=index>index</a></h3>
           </body>
         </html>
       html

     else
       content = "#{ cgi.inspect }"
       content << '<hr>'
       content << cgi.params.inspect
       content << '<hr>'
       content << cgi.params.map{|key,value| [key.inspect, value.map{|v| [v.inspect, v.class]}.inspect]}.inspect
   end

rescue Exception => e
   content = "<b>#{ e.message } - (#{ e.class })</b><hr>#{ e.backtrace.join '<br>' }"
ensure
   cgi.out{ content.to_s }
end

Hi,

···

ara.t.howard@noaa.gov wrote on Friday the 10th of February 2006: > >as I couldn't reproduce the described multipart_form error (see > >description below) directly on the command-line I looked into the cgi.rb > >source and found the chunk of code that raises the exception:

the problem is in your source i'm afraid. you can call CGI.new more than
once
in a script : each call attempts to parse ENV and STDIN according to the cgi
spec.

Thank you very much, for pointing to the error and for sending two very
nice solutions! It works great now!
Sometimes I'm so far down in the libraries, that I can't see the whole
picture anymore.

Bye, Michael

--
Michael Schilling

eMail : ruby@untiefe.de
URL : http://glcu.sf.net/

"Change my name I remain the same." - Moloko