Proxy Server

I’ve been playing around with a project that’s a proxy and cache
server for a Debian apt repository. (apt-proxy in ruby basically) I’ve
got a simple example working using Webrick, but have one annoying
problem I’d like to solve.

When an apt client requests a package file from the proxy, the client
receives no feedback until the proxy has completely downloaded the
file from the backend repository. I’d like to start sending the file
to the client while the backend download is ongoing. I’m hoping this
would also have the side benefit of not having to load the entire
package file into interpreter memory.

The amazing thing is the the original Debian apt-proxy code does this
in a shell-script. I could adopt the apt-proxy methods and use
system calls to perform this, but I was hoping for a more rubyesque
solution.

Any ideas?

  • alan
···


Alan Chen
Digikata LLC

quick guess:

read file only in chunks, pass off each chuck to another thread (don’t
ask me much about this, i’m not sure. check out mutexes) this other
thread of course then passes it on to the user.

like i said my first thoughts.

cheers
transami

···

On Wed, 2002-09-11 at 11:36, Alan Chen wrote:

I’ve been playing around with a project that’s a proxy and cache
server for a Debian apt repository. (apt-proxy in ruby basically) I’ve
got a simple example working using Webrick, but have one annoying
problem I’d like to solve.

When an apt client requests a package file from the proxy, the client
receives no feedback until the proxy has completely downloaded the
file from the backend repository. I’d like to start sending the file
to the client while the backend download is ongoing. I’m hoping this
would also have the side benefit of not having to load the entire
package file into interpreter memory.

The amazing thing is the the original Debian apt-proxy code does this
in a shell-script. I could adopt the apt-proxy methods and use
system calls to perform this, but I was hoping for a more rubyesque
solution.

Any ideas?

  • alan


Alan Chen
Digikata LLC
http://digikata.com


tom sawyer, aka transami
transami@transami.net

A difficulty come from Net::HTTP#get, which is used by WEBrick::HTTPProxy
internally. HTTP#get{} doesn’t tell http response header to its given
block. So WEBrick can’t return the header to user agent until HTTP#get
returns.

I’m planing to modify WEBrick#HTTPProxy to use WEBrick’s own
HTTPClient instead of Net::HTTP to solve this problem. But anything
about its detail have not decided still now. If you have any idea on
design of HTTPClient, please discuss at webricken list.
http://www.notwork.org/ipr/webrick/#ml

Thanks,

– Gotoken

···

At Thu, 12 Sep 2002 02:36:37 +0900, Alan Chen wrote:

When an apt client requests a package file from the proxy, the client
receives no feedback until the proxy has completely downloaded the
file from the backend repository. I’d like to start sending the file
to the client while the backend download is ongoing. I’m hoping this
would also have the side benefit of not having to load the entire
package file into interpreter memory.

Sorry if I was unclear, you’ve caught onto the general idea, but what
I was mostly looking for was a ruby class or library which already
handles this.

···

On Thu, Sep 12, 2002 at 02:51:02AM +0900, Tom Sawyer wrote:

quick guess:

read file only in chunks, pass off each chuck to another thread (don’t
ask me much about this, i’m not sure. check out mutexes) this other
thread of course then passes it on to the user.

like i said my first thoughts.

cheers
transami


Alan Chen
Digikata LLC

I’ve been peeking at libcurl which implements an optional callback
every time a block of data is received on the socket. This allows
you to write your own callback to route the data block. So we could
initiate a download, and have a callback write the received blocks
to multiple destinations at once - in my case, to the file system and
the requesting apt-get client.

···

On Thu, Sep 12, 2002 at 11:20:46AM +0900, GOTO Kentaro wrote:

I’m planing to modify WEBrick#HTTPProxy to use WEBrick’s own
HTTPClient instead of Net::HTTP to solve this problem. But anything
about its detail have not decided still now. If you have any idea on
design of HTTPClient, please discuss at webricken list.
http://www.notwork.org/ipr/webrick/#ml

Thanks,

– Gotoken


Alan Chen
Digikata LLC