I have a project which involves visiting a url
and responding to several prompts to get access
to a database. Is there a way that I can use ruby
to interact with the front end to the database and
automate database queries?
For example, there is a site that gives addresses for
realtors among many other things. The dialog with the
site involves selecting through to the point where
the client gives the server a query to the realtor
database. The query may result in multiple pages.
The client continues requesting pages until it has
obtained all the realtors names.
I would like to start the process by addressing the
url of the server and then having a ruby script be
an automated client to request and collect the list
of realtors.
I can almost figure out how to do this. I know how to
use "require CGI" and I understand how to use eruby.
I suspect that this is not really a CGI problem. Maybe
it is more akin to downloading mail using pop.
Basically, I can find examples which use CGI to field
responses to request s sent to a another url. But
I have not seen a case of an automated interaction
with a url. It seems an obvious thing to have. Am I
missing something? Can some point me to examples of
scripted queries?
Thanks,
John
I can almost figure out how to do this. I know how to use "require CGI" and I understand how to use eruby. I suspect that this is not really a CGI problem. Maybe it is more akin to downloading mail using pop.
Not sure where cgi would come in, as this appears to be a client-side application.
It sounds like you want to make an HTTP request to fetch a page, parse the returned (possibly malformed) HTML (and possibly cookies,
too), create a new HTTP POST or GET, and submit the new HTTP request to proceed to the next page.
Basically, I can find examples which use CGI to field responses to request s sent to a another url. But I have not seen a case of an automated interaction with a url. It seems an obvious thing to have. Am I missing something? Can some point me to examples of scripted queries?
While I can’t point to any examples offhand, I believe this has come up before on this list, so you can perhaps check the archives
at Google groups, or maybe somebody else can provide code/reference.
A simple example of a basic fetch-n-parse is dictionary.rb. It shows how to fetch a web page and parse out specific info, but does
not create a new request based on any content:
#!/usr/local/bin/ruby
···
#--------------------------------------------------------------
Script to fetch a definition for dictionaty.com, strip the HTML, and display
the results in the cosole window. Original code written by Ralph Mason:
Yahoo | Mail, Weather, Search, Politics, News, Finance, Sports & Videos
#--------------------------------------------------------------
require ‘net/http’
class WebDictionary
def look_up( word )
site = “www.dictionary.com”
page = “/cgi-bin/dict.pl?term=#{word}”
a = Net::HTTP.get( site , page )
res = ""
if ( a=~/No entry found for/ )
puts "No entry found for '#{word}' - Suggestions:\n\n"
while $' =~/<a href="\/cgi-bin\/dict.pl\?term=\w+">(\w+)/
res << $1
end
else
while a =~ /<!-- resultItemStart -->/
$' =~ /<!-- resultItemEnd -->/
a=$'
res << $`.gsub(/<.*?>/m){ |i|
case i
when "<b>"
""
when "</b>"
"\n"
when "<p>"
""
else
""
end
}
end
end
res
end
end
if FILE == $0
if ARGV.size == 0
puts "Usage: #{$0} "
exit
end
puts WebDictionary.new().look_up( ARGV[0] )
end
I have a project which involves visiting a url and responding to several prompts to get access to a database. Is there a way that I can use ruby to interact with the front end to the database and automate database queries? For example, there is a site that gives addresses for realtors among many other things. The dialog with the site involves selecting through to the point where the client gives the server a query to the realtor database. The query may result in multiple pages. The client continues requesting pages until it has obtained all the realtors names. I would like to start the process by addressing the url of the server and then having a ruby script be an automated client to request and collect the list of realtors. I can almost figure out how to do this. I know how to use "require CGI" and I understand how to use eruby. I suspect that this is not really a CGI problem. Maybe it is more akin to downloading mail using pop. Basically, I can find examples which use CGI to field responses to request s sent to a another url. But I have not seen a case of an automated interaction with a url. It seems an obvious thing to have. Am I missing something? Can some point me to examples of scripted queries? Thanks, John
What you want is http-access2, look for it on RAA. Depending on how the site is
written, you may only need to look at the last page to figure out what to send.
I have a project which involves visiting a url and responding to several prompts to get access to a database. Is there a way that I can use ruby to interact with the front end to the database and automate database queries?
Using net/http could work well. If you’re on Windows and using IE, I’ve got
a helper lib to drive IE at my site - check it out here:
http://clabs.org/scrpware.htm (bottom of the page: cLabs IEController)
Chris