I've been working on a proxy server implementation for a project idea
I have. But I've found that it only seems to work on certain pages,
and not on others. I've distilled the problem down to a simple
implementation, but it still occurs. The code below will return a
(mostly) correct website if pointed at 'www.msn.com' but if pointed at
say 'slashdot.org' the browser refuses to display it, in both cases
the page is loaded correctly. If anyone has any ideas I'd really
appreciate it.
host = 'slashdot.org'
server = TCPServer.new(local_host,local_port)
while true
Thread.start(server.accept) do |s|
data = ''
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
p "====Received: #{data}"
net = Net::HTTP.new(host,80)
response = net.get('/')
s << "HTTP/#{response.http_version}
#{response.code}/#{response.message} \r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end
I gave it a quick run. I had to change
local_host = nil
so that it would bind and fixed the line wrap in the posted code. Telnetted to port 8080 on localhost and hit enter twice and it came back with a lot of html from slashdot. So it works on
Can't see anything obviously wrong. Do you have a transparent proxy or anything between you and the site as this can cause havoc?
J.
···
On 22/04/2005, at 10:54 AM, Tanner Burson wrote:
I've been working on a proxy server implementation for a project idea
I have. But I've found that it only seems to work on certain pages,
and not on others. I've distilled the problem down to a simple
implementation, but it still occurs. The code below will return a
(mostly) correct website if pointed at 'www.msn.com' but if pointed at
say 'slashdot.org' the browser refuses to display it, in both cases
the page is loaded correctly. If anyone has any ideas I'd really
appreciate it.
I've been working on a proxy server implementation for a project idea
I have. But I've found that it only seems to work on certain pages,
and not on others. I've distilled the problem down to a simple
implementation, but it still occurs. The code below will return a
(mostly) correct website if pointed at 'www.msn.com' but if pointed at
say 'slashdot.org' the browser refuses to display it, in both cases
the page is loaded correctly. If anyone has any ideas I'd really
appreciate it.
I am behind a proxy server at work, yet got this to work. I commented
out your Net::HTTP.new line and added what I believe to be the Proxy
equivalent. Then I set Mozilla Firefox's proxy to be localhost:8080.
NOTE: I do not have the ability to use another machine to test as I
am very restricted with net access here. I launched slashdot.org and
monitored the proxy.rb window while all of the HTML flew by. It came
up and rendered properly in the browser.
I am on Windows 2000
C:\>ruby -v
ruby 1.8.2 (2004-12-25) [i386-mswin32]
host = 'slashdot.org'
server = TCPServer.new(local_host, local_port)
while true
Thread.start(server.accept) do |s|
data = ''
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
p "====Received: #{data}"
# net = Net::HTTP.new(host,80)
net = Net::HTTP::Proxy("proxy", "80").start(host)
response = net.get('/')
s <<
"HTTP/#{response.http_version}#{response.code}/#{response.message}
\r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end
···
On 4/21/05, Tanner Burson <tanner.burson@gmail.com> wrote:
I've been working on a proxy server implementation for a project idea
I have. But I've found that it only seems to work on certain pages,
and not on others. I've distilled the problem down to a simple
implementation, but it still occurs. The code below will return a
(mostly) correct website if pointed at 'www.msn.com' but if pointed at
say 'slashdot.org' the browser refuses to display it, in both cases
the page is loaded correctly. If anyone has any ideas I'd really
appreciate it.
host = 'slashdot.org'
server = TCPServer.new(local_host,local_port)
while true
Thread.start(server.accept) do |s|
data = ''
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
p "====Received: #{data}"
net = Net::HTTP.new(host,80)
response = net.get('/')
s << "HTTP/#{response.http_version}
#{response.code}/#{response.message} \r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end
====end
--
===Tanner Burson===
tanner.burson@gmail.com http://tannerburson.com <---Might even work one day...
I gave it a quick run. I had to change
local_host = nil
so that it would bind and fixed the line wrap in the posted code. Telnetted to port 8080 on localhost and hit enter twice and it came back with a lot of html from slashdot.
I managed the same, on win32, but could not get the results to render in Firefox or IE. It fetches the page OK (judging by the debugging p calls), but whatever is sent back to the browser is off in some way.
Not that I'm aware of. I should have been more specific the first
time, try setting it as a proxy in your browser settings, I've yet to
get it to load correctly in a browser.
···
On 4/21/05, jm <jeffm@ghostgun.com> wrote:
I gave it a quick run. I had to change
local_host = nil
so that it would bind and fixed the line wrap in the posted code.
Telnetted to port 8080 on localhost and hit enter twice and it came
back with a lot of html from slashdot. So it works on
I neglected to mention that if you use the proxy version of Net::HTTP,
you can get it to work without a proxy by putting in "" for the proxy
and port. Here's the proxy version:
net = Net::HTTP::Proxy("proxy", "80").start(host)
and the non-proxy version:
net = Net::HTTP::Proxy("", "").start(host)
HTH,
Craig
···
On 4/25/05, Craig Moran <craig.m.moran.ruby@gmail.com> wrote:
I am behind a proxy server at work, yet got this to work. I commented
out your Net::HTTP.new line and added what I believe to be the Proxy
equivalent. Then I set Mozilla Firefox's proxy to be localhost:8080.
NOTE: I do not have the ability to use another machine to test as I
am very restricted with net access here. I launched slashdot.org and
monitored the proxy.rb window while all of the HTML flew by. It came
up and rendered properly in the browser.
I am on Windows 2000
C:\>ruby -v
ruby 1.8.2 (2004-12-25) [i386-mswin32]
host = 'slashdot.org'
server = TCPServer.new(local_host, local_port)
while true
Thread.start(server.accept) do |s|
data = ''
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
p "====Received: #{data}"
# net = Net::HTTP.new(host,80)
net = Net::HTTP::Proxy("proxy", "80").start(host)
response = net.get('/')
s <<
"HTTP/#{response.http_version}#{response.code}/#{response.message}
\r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end
On 4/21/05, Tanner Burson <tanner.burson@gmail.com> wrote:
> I've been working on a proxy server implementation for a project idea
> I have. But I've found that it only seems to work on certain pages,
> and not on others. I've distilled the problem down to a simple
> implementation, but it still occurs. The code below will return a
> (mostly) correct website if pointed at 'www.msn.com' but if pointed at
> say 'slashdot.org' the browser refuses to display it, in both cases
> the page is loaded correctly. If anyone has any ideas I'd really
> appreciate it.
>
> ===proxy.rb
>
> require 'socket'
> require 'net/http'
> local_host = '192.168.0.6'
> local_port = 8080
>
> host = 'slashdot.org'
> server = TCPServer.new(local_host,local_port)
> while true
> Thread.start(server.accept) do |s|
> data = ''
> while s.gets && !$_.nil?
> data += $_
> break if $_ == "\r\n" || $_ == "\n"
> end
> p "====Received: #{data}"
> net = Net::HTTP.new(host,80)
> response = net.get('/')
> s << "HTTP/#{response.http_version}
> #{response.code}/#{response.message} \r\n"
> p "====Got back"
> response.each_header do |key,value|
> tmp = key + ": " + value + "\r\n"
> p tmp
> s << tmp
> end
> p response.body
> s << "\r\n" << response.body
> s.close # We're done with this request, bring on the next.
> end
> end
>
> ====end
> --
> ===Tanner Burson===
> tanner.burson@gmail.com
> http://tannerburson.com <---Might even work one day...
>
>
jm wrote:
> I gave it a quick run. I had to change
> local_host = nil
> so that it would bind and fixed the line wrap in the posted code.
> Telnetted to port 8080 on localhost and hit enter twice and it came back
> with a lot of html from slashdot.
I managed the same, on win32, but could not get the results to render in
Firefox or IE. It fetches the page OK (judging by the debugging p
calls), but whatever is sent back to the browser is off in some way.
This so far has been exactly the problem. Some sites (ie: msn.com )
work just fine, but others won't work at all. I've compared the
output of both requests but haven't been able to determine any
difference other than possibly the amount of content being sent back.
I'm about at my wit's end over this one, any advice would be great.
···
On 4/21/05, James Britt <james_b@neurogami.com> wrote:
James
--
===Tanner Burson===
tanner.burson@gmail.com http://tannerburson.com <---Might even work one day...
Alright. this is wierd I just tried it with firefox on macosx and it rendered find, except for all the broken images, etc due to the reference to '/', ie
net = Net::HTTP.new(host,80)
response = net.get('/')
Made a few modification. Really needs a clean up and there's probably libraries out there which will do this cleaner. I've been staring at perl code most of the day wishing it wasn't . Anyway, I've included the full script below.
I gave it a quick run. I had to change
local_host = nil
so that it would bind and fixed the line wrap in the posted code. Telnetted to port 8080 on localhost and hit enter twice and it came back with a lot of html from slashdot.
I managed the same, on win32, but could not get the results to render in Firefox or IE. It fetches the page OK (judging by the debugging p calls), but whatever is sent back to the browser is off in some way.
host = 'slashdot.org'
server = TCPServer.new(local_host,local_port)
while true
Thread.start(server.accept) do |s|
header = ''
data = ''
# this is a little crude
header = s.gets # added
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
header = header.split(' ') # added
proto,host,path = url_split(header[1]) # added
p "==== header[1]: #{header[1]}" # added
p "==== #{proto} #{host} #{path}" # added
p "====Received: #{data}"
net = Net::HTTP.new(host,80)
response = net.get(path) # changed
s << "HTTP/#{response.http_version} #{response.code}/#{response.message} \r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end
I neglected to mention that if you use the proxy version of Net::HTTP,
you can get it to work without a proxy by putting in "" for the proxy
and port. Here's the proxy version:
net = Net::HTTP::Proxy("proxy", "80").start(host)
and the non-proxy version:
net = Net::HTTP::Proxy("", "").start(host)
This hangs until the timeout is reached. But if I use nil instead of
empty string it continues, but still fails loading correctly in the
browser. But I think it may have to do with the headers sent with a
proxied request, can you send me the sent headers portion of the
output when used with a working proxy server?
···
On 4/25/05, Craig Moran <craig.m.moran.ruby@gmail.com> wrote:
Have you tried setting your HTTP_USER_AGENT to something like:
Mozilla/5.0
When I'm using the full script (not included for brevity), it runs as
a true proxy and relays all the request headers straight on through to
the server. The code I provided is just a simple sample that still
portrays the same problems. It doesn't appear to be anything related
to the request itself, as the response comes back successfully. There
is just something about it the browser doesn't like.
My first thought was that it didn't like receiving the whole page at
once, so I tried reading the body in chunks and sending it on, but it
didn't seem to solve it either.
···
On 4/21/05, Jim Freeze <jim@freeze.org> wrote:
--
Jim Freeze
Code Red. Code Ruby
--
===Tanner Burson===
tanner.burson@gmail.com http://tannerburson.com <---Might even work one day...
* jm <jeffm@ghostgun.com> [2005-04-22 12:50:56 +0900]:
Alright. this is wierd I just tried it with firefox on macosx and it
rendered find, except for all the broken images, etc due to the
reference to '/', ie
Uhh, this may be another dumb suggestion, but are you
by chance sending inspect data with #p instead of #puts
to the browser?
···
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.