Using an HTTP proxy to record web traffic

ruby-talk
1

As part of debugging WWW::Mechanize, I realized that the best way to
compare what my browser is doing to what mechanize is doing would be to
compare their HTTP transactions incoming and outgoing. What I want is
an HTTP proxy that can record the transactions and also supports SSL.

The Ruby project doesn't have any code
http://rubyforge.org/projects/rpp/

This free program doesn't support SSL, and costs $600.
http://www.proxy-sniffer.com/download.html

This doesn't support SSL but is supposed to soon.
http://kevinlangdon.com/serviceCapture/

Any other suggestions?

2

Dan Kohn wrote:

As part of debugging WWW::Mechanize, I realized that the best way to
compare what my browser is doing to what mechanize is doing would be to
compare their HTTP transactions incoming and outgoing. What I want is
an HTTP proxy that can record the transactions and also supports SSL.

MouseHole is a scriptable proxy. <http://mousehole.rubyforge.org/&gt;

A script to record traffic would a cinch. Basically,

1. Download MouseHole from the above link. Instructions for running it are there as well.
2. Add a script named `proxyrecord.user.rb' to your ~/.mouseHole/userScripts/ directory.
    Contains:

       MouseHole.script do
           name "Proxy Record"
           namespace "http://yourdomain.com/&quot;
           description "Records all traffic, very snoopy."
           include_match %r!http://.*!
           version "1.0"

           rewrite do |req, res|
               # store contents of `req' and `res' in the database or a
               # log file or something
           end
       end
3. In your browser, setup MouseHole as your proxy (http://127.0.0.1:37004/\)
4. Visit http://127.0.0.1:37004/ and activate the Proxy Record script by checking its box.

As for SSL, I don't think proxies can actually intercept unencrypted SSL. It uses the CONNECT method.

_why

3

Very helpful, why, thanks. I presume I would need to use something
like WATIR to see the unencrypted contents of an SSL connection by
getting it directly from IE.