Simulating Mouse Dragging

ruby-talk
1

Is it possible to simulate a mouse drag event in a browser (or
any program in general) with Ruby? Something like:

dragmouse(x1, y1, x2, y2)

or

dragmouse("windowname", x1, y1, x2, y2)

Either Windows or Linux (Xfree86).

Why? On the 'Net, there are a lot of sites which offer maps.
You can scroll these maps by dragging the image, virtually
flying over the map. You can look at such a map as one very,
very big image, of which you see only one part at a time.
Collecting these parts gives us little overlapping maps of a
certain region. These overlapping maps can be converted to
non-overlapping maps by cropping and saving with ImageMagick.
The resulting "tiles" are zipped in a file, which can be
handled with TIV (Tiled Image Viewer).

This is no theory, it actually works. I've made TIV in Ewe [1]
(kind of Java...). It runs on Windows (natively, fast), Linux
(Java, slow) and on my iPAQ (natively, fast enough). I did the
conversion from the overlapping maps to the non-overlapping
maps with ImageMagick. Not difficult, but it takes some
processing time. Saving the screenshot can be done with
"import" (part of ImageMagick). I glued everything together
with Ruby, of course.

I once got the images by hacking the URL's of one of those
sites, but they changed it... Grabbing the images by taking
screenshots is much better, because it works for every
"vendor"... The only missing part is the programmatically
dragging of the images in the browser.

Anybody able to help? Other ideas? (Except "buying" a
product...)

gegroet,
Erik V. - http://www.erikveen.dds.nl/

[1] http://www.ewesoft.com/

2

Is it possible to simulate a mouse drag event in a browser (or
any program in general) with Ruby? Something like:

dragmouse(x1, y1, x2, y2)

or

dragmouse("windowname", x1, y1, x2, y2)

Either Windows or Linux (Xfree86).

Could recording + analyzing mouse data help? Then you can simulate it
and let X11 consume it. `gpm -R` may help.

+--- Kero ------------------------- kero@chello@nl ---+

all the meaningless and empty words I spoke |
                      Promises -- The Cranberries |

+--- M38c --- http://members.chello.nl/k.vangelder ---+

3

If you realy want to do it in a browser the simplest way would be to
capture the MOUSEMOVE events in javascript.

Another way it to build an app with Gtk+. There you have complete mouse
controll and allso got image handling. The advantage over the browser
solution is, that it would run on Linux, OS X and Windoze. Within
javascript you have to program around the pittfalls of the different DOM
models.

Cheers
detlef

···

Am Samstag, den 30.07.2005, 05:01 +0900 schrieb Erik Veenstra:

Is it possible to simulate a mouse drag event in a browser (or
any program in general) with Ruby? Something like:

dragmouse(x1, y1, x2, y2)

or

dragmouse("windowname", x1, y1, x2, y2)

Either Windows or Linux (Xfree86).

4

Erik Veenstra wrote:

Is it possible to simulate a mouse drag event in a browser (or
any program in general) with Ruby? Something like:

dragmouse(x1, y1, x2, y2)

or

dragmouse("windowname", x1, y1, x2, y2)

Either Windows or Linux (Xfree86).

I've never used it, but AutoIt for windows may be able to do what
you want: AutoIt Scripting Language - AutoIt

5

"Recording" the mouse data by reading /dev/gpmdata actually
works. Good start. Generating such a data stream is possible.
But what do we have to do to let X11 "consume" that stream?
"cat test > /dev/gpmdata" doesn't work...

gegroet,
Erik V. - http://www.erikveen.dds.nl/

···

On Fri, 29 Jul 2005 20:21:04 +0000, Kero wrote:

> Is it possible to simulate a mouse drag event in a browser
> (or any program in general) with Ruby? Something like:
>
> dragmouse(x1, y1, x2, y2)
>
> or
>
> dragmouse("windowname", x1, y1, x2, y2)
>
> Either Windows or Linux (Xfree86).

Could recording + analyzing mouse data help? Then you can
simulate it and let X11 consume it. `gpm -R` may help.

6

> Is it possible to simulate a mouse drag event in a browser
> (or any program in general) with Ruby? Something like:

Could recording + analyzing mouse data help? Then you can
simulate it and let X11 consume it. `gpm -R` may help.

"Recording" the mouse data by reading /dev/gpmdata actually
works. Good start. Generating such a data stream is possible.
But what do we have to do to let X11 "consume" that stream?
"cat test > /dev/gpmdata" doesn't work...

disclaimer: I have never tried anything like this myself.

In what sense does it not work?
(assuming you made X11 read /dev/gpmdata one way or the other).
I would guess that `cat test` is "too fast".

+--- Kero ------------------------- kero@chello@nl ---+

all the meaningless and empty words I spoke |
                      Promises -- The Cranberries |

+--- M38c --- http://members.chello.nl/k.vangelder ---+

7

> > > Is it possible to simulate a mouse drag event in a
> > > browser (or any program in general) with Ruby?
> > > Something like:
> >
> > Could recording + analyzing mouse data help? Then you can
> > simulate it and let X11 consume it. `gpm -R` may help.
>
> "Recording" the mouse data by reading /dev/gpmdata actually
> works. Good start. Generating such a data stream is
> possible. But what do we have to do to let X11 "consume"
> that stream? "cat test /dev/gpmdata" doesn't work...

disclaimer: I have never tried anything like this myself.

In what sense does it not work? (assuming you made X11 read
/dev/gpmdata one way or the other). I would guess that `cat
test` is "too fast".

I experimented a bit, yesterday night. I didn't configure X11
to read from /dev/gpmdata. But I found that one. After that,
writing to /dev/gpmdata with a properly (?) filled memory
structure, actually did move the mouse. Clicking was possible,
too. But dragging somehow still doesn't work.

The protocol is IMPS/2.

I attached my experiment.

Somebody able and willing to help?

gegroet,
Erik V. - http://www.erikveen.dds.nl/

···

----------------------------------------------------------------

# http://www.mit.edu:8001/people/gassend/protocols/intellimouse/
# http://www.microsoft.com/whdc/device/input/mcompat.mspx

class Mouse
   Yov = 0x80
   Xov = 0x40
   Y8 = 0x20
   X8 = 0x10
   BIT4 = 0x08
   MB = 0x04
   RB = 0x02
   LB = 0x01

   def initialize
     @handle = File.new("/dev/gpmdata", "wb")
     @handle.sync = true
     @button = 0
   end

   def move(dx, dy)
     @bytes =

     byte1 = (BIT4 | @button)
     byte2 = (dx.to_i)
     byte3 = (dy.to_i)
     byte4 = 0

     if dx < 0
       byte1 |= X8
       byte2 += 256
     end

     if dy < 0
       byte1 |= Y8
       byte3 += 256
     end

     @bytes << byte1
     @bytes << byte2
     @bytes << byte3
     @bytes << byte4

     send
   end

   def down(button)
     @button = button
     move(0, 0)
   end

   def up
     @button = 0
     move(0, 0)
   end

   def drag(dx, dy, button)
     down(button)
     move(dx, dy)
     up
   end

   def click(button)
     down(button)
     up
   end

   def stream
     @bytes.pack("c%d" % @bytes.length)
   end

   def inspect
     @bytes.collect{|s| s.chr}.join("").unpack("H2"*@bytes.length)
   end

   def send
     p inspect
     @handle << stream
   end
end

#system "sudo /etc/init.d/gpm stop" or puts "Oops"

mouse = Mouse.new

mouse.drag(-100, 0, Mouse::LB)
mouse.move(+100, 0)

#system "sudo /etc/init.d/gpm start" or puts "Oops"

----------------------------------------------------------------