Friends-
I have seen a few blog posts where folks mention that they have gotten the ruby one click installer and watir/IE running under wine on linux. Has anyone out there done this successfully? My company would be willing to pay someone to help us get this up and running. Anyone have any tips or any more information about making this kind of thing happen?
We are using watir as a web scraper tool of sorts for scraping crusty sites that use multiple framesets and java applets and all kinds of other horrible stuff that defies any type of regular screen scraping short of using watir to actually navigate the sites and enter data into forms to get results we need from these companies sites. And we need multiple instances of watir driving the same companies site at once which is problematic because internet explorer overwrites its cookies when more then one of these scrapes is run against the same website at once. So we are trying to experiment with running watir under wine so each scaper runs in its own env and doesn't step on each others cookies.
Any win32 gurus have any ideas as to how we might be able to dynamically from our ruby/watir scripts, manage the cookies written by IE so we could control where they get written and be able to run two or more concurrent scrapes of the same website via watir without IE overwriting the cookie it uses? My company is willing to pay someone to help solve this problem. Thoughts or ideas are very welcome at this point. I have asked this same thing eslewhere but I know there are many smart folks on this list and I hope someone might have an idea to help us out. More info available if you need it.
Thanks-
-Ezra
You could try runas
(http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/runas.mspx\)
to run multiple Watir (and hence automated IE) processes as different
users, which would put the cookies in separates profiles.
Wouldn't work if you're trying to run multiple IE processes via
separate Ruby threads though.
Regards,
Sean
···
On 7/11/06, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:
Friends-
I have seen a few blog posts where folks mention that they have
gotten the ruby one click installer and watir/IE running under wine
on linux. Has anyone out there done this successfully? My company
would be willing to pay someone to help us get this up and running.
Anyone have any tips or any more information about making this kind
of thing happen?
We are using watir as a web scraper tool of sorts for scraping
crusty sites that use multiple framesets and java applets and all
kinds of other horrible stuff that defies any type of regular screen
scraping short of using watir to actually navigate the sites and
enter data into forms to get results we need from these companies
sites. And we need multiple instances of watir driving the same
companies site at once which is problematic because internet explorer
overwrites its cookies when more then one of these scrapes is run
against the same website at once. So we are trying to experiment with
running watir under wine so each scaper runs in its own env and
doesn't step on each others cookies.
Any win32 gurus have any ideas as to how we might be able to
dynamically from our ruby/watir scripts, manage the cookies written
by IE so we could control where they get written and be able to run
two or more concurrent scrapes of the same website via watir without
IE overwriting the cookie it uses? My company is willing to pay
someone to help solve this problem. Thoughts or ideas are very
welcome at this point. I have asked this same thing eslewhere but I
know there are many smart folks on this list and I hope someone might
have an idea to help us out. More info available if you need it.
Thanks-
-Ezra
Thanks Sean. We are already using runas I should have mentioned. THe reason this is failing is because some of these sites have modal dialogs and the only way to deal with these in watir is via click_no_wait. What click no wait does is uses system to start another process so the modal dialog doesn't block the entire ruby process. THen when the separate process tries to re-attach to the window there is a security error that won't let a different process then the one that started ie bind to the window. This all goes away if we can get a way to dynamically manage the cookies somehow. Or under wine each instance if wine will have its own environment that skirts the issue as well. We have been up and down the wall with this issue for the past week and have come up with no answers yet. I do appreciate the suggestion though.
Thanks
-Ezra
···
On Jul 11, 2006, at 1:21 PM, Sean O'Halpin wrote:
On 7/11/06, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:
Friends-
I have seen a few blog posts where folks mention that they have
gotten the ruby one click installer and watir/IE running under wine
on linux. Has anyone out there done this successfully? My company
would be willing to pay someone to help us get this up and running.
Anyone have any tips or any more information about making this kind
of thing happen?
We are using watir as a web scraper tool of sorts for scraping
crusty sites that use multiple framesets and java applets and all
kinds of other horrible stuff that defies any type of regular screen
scraping short of using watir to actually navigate the sites and
enter data into forms to get results we need from these companies
sites. And we need multiple instances of watir driving the same
companies site at once which is problematic because internet explorer
overwrites its cookies when more then one of these scrapes is run
against the same website at once. So we are trying to experiment with
running watir under wine so each scaper runs in its own env and
doesn't step on each others cookies.
Any win32 gurus have any ideas as to how we might be able to
dynamically from our ruby/watir scripts, manage the cookies written
by IE so we could control where they get written and be able to run
two or more concurrent scrapes of the same website via watir without
IE overwriting the cookie it uses? My company is willing to pay
someone to help solve this problem. Thoughts or ideas are very
welcome at this point. I have asked this same thing eslewhere but I
know there are many smart folks on this list and I hope someone might
have an idea to help us out. More info available if you need it.
Thanks-
-Ezra
You could try runas
(http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/runas.mspx\)
to run multiple Watir (and hence automated IE) processes as different
users, which would put the cookies in separates profiles.
Wouldn't work if you're trying to run multiple IE processes via
separate Ruby threads though.
Regards,
Sean
I've never used watir, but the best way I've found to isolate
environments is using virtualization software such as VMWare, Virtual
PC or the like. It sounds like it might be overkill for you
(especially if you want to run more than, say, 4 instances) but I
thought I'd throw it out there.
···
On 7/11/06, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:
On Jul 11, 2006, at 1:21 PM, Sean O'Halpin wrote:
Thanks Sean. We are already using runas I should have mentioned. THe
reason this is failing is because some of these sites have modal
dialogs and the only way to deal with these in watir is via
click_no_wait. What click no wait does is uses system to start
another process so the modal dialog doesn't block the entire ruby
process. THen when the separate process tries to re-attach to the
window there is a security error that won't let a different process
then the one that started ie bind to the window. This all goes away
if we can get a way to dynamically manage the cookies somehow. Or
under wine each instance if wine will have its own environment that
skirts the issue as well. We have been up and down the wall with this
issue for the past week and have come up with no answers yet. I do
appreciate the suggestion though.
Thanks
-Ezra
Ezra Zygmuntowicz wrote:
Friends-
I have seen a few blog posts where folks mention that they have
gotten the ruby one click installer and watir/IE running under wine
on linux. Has anyone out there done this successfully? My company
would be willing to pay someone to help us get this up and running.
Anyone have any tips or any more information about making this kind
of thing happen?
We are using watir as a web scraper tool of sorts for scraping
crusty sites that use multiple framesets and java applets and all
kinds of other horrible stuff that defies any type of regular screen
scraping short of using watir to actually navigate the sites and
enter data into forms to get results we need from these companies
sites. And we need multiple instances of watir driving the same
companies site at once which is problematic because internet explorer
overwrites its cookies when more then one of these scrapes is run
against the same website at once. So we are trying to experiment with
running watir under wine so each scaper runs in its own env and
doesn't step on each others cookies.
Any win32 gurus have any ideas as to how we might be able to
dynamically from our ruby/watir scripts, manage the cookies written
by IE so we could control where they get written and be able to run
two or more concurrent scrapes of the same website via watir without
IE overwriting the cookie it uses? My company is willing to pay
someone to help solve this problem. Thoughts or ideas are very
welcome at this point. I have asked this same thing eslewhere but I
know there are many smart folks on this list and I hope someone might
have an idea to help us out. More info available if you need it.
Thanks-
-Ezra
You could try runas
(http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/runas.mspx\)
to run multiple Watir (and hence automated IE) processes as different
users, which would put the cookies in separates profiles.
Wouldn't work if you're trying to run multiple IE processes via
separate Ruby threads though.
Regards,
Sean
Thanks Sean. We are already using runas I should have mentioned. THe reason this is failing is because some of these sites have modal dialogs and the only way to deal with these in watir is via click_no_wait. What click no wait does is uses system to start another process so the modal dialog doesn't block the entire ruby process. THen when the separate process tries to re-attach to the window there is a security error that won't let a different process then the one that started ie bind to the window. This all goes away if we can get a way to dynamically manage the cookies somehow. Or under wine each instance if wine will have its own environment that skirts the issue as well. We have been up and down the wall with this issue for the past week and have come up with no answers yet. I do appreciate the suggestion though.
Thanks
-Ezra
Rather than using system() you could try using Process.create in the win32-process package. I'm guessing somewhat here, but give this a shot:
require 'win32/process'
include Win32
def click_no_wait
Process.create(
:app_name => "c:\ruby\bin\ruby ...",
:creation_flags => Process::CREATE_PRESERVE_CODE_AUTHZ_LEVEL
:inherited => true
)
end
You can futz around with the various creation flags and see if any of them help you out.
HTH,
Dan
···
On Jul 11, 2006, at 1:21 PM, Sean O'Halpin wrote:
On 7/11/06, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote: