Web services and larger files

Hi all,

This is an addendum to my earlier thread about web services and ruby,
wherein I was forced to accept the inevitability of using threads.

I've successfully got my client and server talking to each other for
small tasks, but I'm trying to expand to being able to pass (sometimes
large) files around and I've run into what seems almost like a
fundamental problem.

Basically, I'm trying to send a file over the wire, and for files above
about 20k in size I get an indefinite hang.

I've tried this with soap4r and webrick (I initially thought it was a
problem passing binary around, because the binary files I was testing
were the bigger ones), and I just implemented the same functionality in
xmlrpc and its own builtin server. In both cases, I get basically the
same behaviour: Smaller files go over just fine, larger files hang
indefinitely on a 'write' call:

write(4, "<?xml version=\"1.0\" ?><methodCal"..., 1389384

I've verified that I'm depending on essentially none of the same
libraries in my two implementations (other than builtin classes, of
course), yet I get the same behaviour in both cases.

Anyone have any ideas?

I'm currently actually passing the file contents as an argument to the
method call; should I instead somehow attach the file somehow? Is there
an accepted way to pass files around in XML that I don't know about?

Any help would be greatly appreciated.

···

--
I went to a restaurant that serves "breakfast at anytime". So I
ordered French Toast during the Renaissance. -- Stephen Wright
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://config.sage.org

Sorry, forgot the code.

Here's the actual code:

http://reductivelabs.com/svn/puppet/language/trunk/lib/puppet/filebucket.rb

And the test script:

http://reductivelabs.com/svn/puppet/language/trunk/test/bucket/tc_bucket.rb

···

--
Barrow's first law:
   Any Universe simple enough to be understood is too simple to produce
   a mind able to understand it.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://config.sage.org

Boy. I'm not sure if I am going to be able to help you, but I'll take a look
at the code you posted links to in your followup message.

However, I did want to pipe up and say that I have code that is regularly
being used to pass files over the wire as large as 100 megabytes, with no
problems. This is a web based application, and works just fine in three
different scenarios that I have tested:

1) Webrick -> application thread
2) Apache/mod_ruby -> seperate ruby process
3) Lighttpd -> FCGI proxy (written in Ruby) -> seperate ruby process

Kirk Haines

···

On Thursday 07 July 2005 3:57 pm, Luke Kanies wrote:

I've successfully got my client and server talking to each other for
small tasks, but I'm trying to expand to being able to pass (sometimes
large) files around and I've run into what seems almost like a
fundamental problem.

Hi,

Luke Kanies wrote:

This is an addendum to my earlier thread about web services and ruby,
wherein I was forced to accept the inevitability of using threads.

I've successfully got my client and server talking to each other for
small tasks, but I'm trying to expand to being able to pass (sometimes
large) files around and I've run into what seems almost like a
fundamental problem.

Basically, I'm trying to send a file over the wire, and for files above
about 20k in size I get an indefinite hang.

Hmm. I tried sending 2M< size binary file but it seems to work. Can I
see an example which causes indefinite hang?

I'm currently actually passing the file contents as an argument to the
method call; should I instead somehow attach the file somehow? Is there
an accepted way to pass files around in XML that I don't know about?

Do you see a sample of "SOAP with Attachment" at
http://dev.ctor.org/soap4r/browser/trunk/sample/soap/swa/ ?

Regards,
// NaHi

Boy. I'm not sure if I am going to be able to help you, but I'll take a look
at the code you posted links to in your followup message.

I would definitely appreciate it.

However, I did want to pipe up and say that I have code that is regularly
being used to pass files over the wire as large as 100 megabytes, with no
problems. This is a web based application, and works just fine in three
different scenarios that I have tested:

1) Webrick -> application thread
2) Apache/mod_ruby -> seperate ruby process
3) Lighttpd -> FCGI proxy (written in Ruby) -> seperate ruby process

Do you have any code you'd be willing to share?

I'm somewhat convinced that it's a problem with what I've written, so I
would not be surprised if others aren't having this problem, but, well,
I'm having this problem and I can't seem to figure out what the deal is.
I'm especially concerned that the behaviour is a bit weird -- just
hanging, instead of throwing an actual error -- as that makes it more
likely I'm doing something wrong.

Thanks,
Luke

···

On Fri, 8 Jul 2005, Kirk Haines wrote:

--
Censorship, like charity, should begin at home; but, unlike charity, it
should end there. --Clare Booth Luce
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://config.sage.org

Because you don't mention XMLRPC here, I figure it might be of value to
point to the webrick/soap4r versions of my code:

http://reductivelabs.com/cgi-bin/puppet.cgi/file/language/trunk/lib/puppet/filebucket.rb?rev=331

http://reductivelabs.com/cgi-bin/puppet.cgi/file/language/trunk/test/bucket/tc_bucket.rb?rev=331

···

On Fri, 8 Jul 2005, Kirk Haines wrote:

However, I did want to pipe up and say that I have code that is regularly
being used to pass files over the wire as large as 100 megabytes, with no
problems. This is a web based application, and works just fine in three
different scenarios that I have tested:

1) Webrick -> application thread
2) Apache/mod_ruby -> seperate ruby process
3) Lighttpd -> FCGI proxy (written in Ruby) -> seperate ruby process

--
One of the Ten Commandments for Technicians
        (7) Work thou not on energized equipment, for if thou dost, thy
            fellow workers will surely buy beers for thy widow and
            console her in other ways.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://config.sage.org