[QUIZ] Phone Typing (#21)

The three rules of Ruby Quiz:

1. Please do not post any solutions or spoiler discussion for this quiz until
48 hours have passed from the time on this message.

2. Support Ruby Quiz by submitting ideas as often as you can:

http://www.rubyquiz.com/

3. Enjoy!

···

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

by Hans Fugal

I am amazed whenever I see or hear about the rising generation of keypad
punchers. People that can carry on IM conversations with a 12-key phone
pad without instantly going mad; it's mind-boggling. As adaptive as the
rising generation is, the "Multitap" solution is far from efficient. For
example, I learned from Wheel of Fortune that the most common letters
are RSTLNE, only one of which (T) is the first tap on one of the keys.

Your mission then, should you choose to accept it, is to develop a more
efficient algorithm for key entry. User interaction is simply the 12
keys (0-9, * and #) with the letters printed on them as usual[1], and
visual feedback of what they have typed so far. Simulating algorithms
thought up by other people (e.g. LetterWise[2]) is fair game.

[ Editor's Note:

Hans sent the following code to get people started with a simple testing
inteface for their algorithms:

  class TapEvent
    attr_reader :when, :digit
    def initialize(digit)
      @digit = digit
      @when = Time.now
    end
    def to_s
      @digit
    end
  end

  class PhonePad
    attr_reader :text, :cursor
    def set_text(text,cursor)
      @text = text
      @cursor = cursor
    end
    def register(&block)
      @observers.push block
    end
    def initialize
      @observers = []
      @text = ''
      @cursor = 0
    end
    def notify_observers(digit)
      if digit =~ /[0-9*#]/
        @observers.each { |block| block.call(TapEvent.new(digit)) }
      end
    end
  end

  class CLIPhonePad < PhonePad
    def initialize
      super
      Thread.new do
        while not $stdin.eof? do
    $stdin.readline.split('').each { |c| notify_observers(c) }
        end
      end
    end
    def set_text(text,cursor)
      super(text,cursor)
      puts to_s
    end
    def run
    end
    def to_s
      @text.dup.insert(@cursor,'_')
    end
  end

  if $0 == __FILE__
    p = CLIPhonePad.new

    p.register do |ev|
      puts "#{ev} at #{ev.when}"
      p.set_text(p.text+ev.digit, p.cursor+1)
    end

    Thread.list[0].join
  end

Hans's code is well suited to GUI adaption, if you're so inclined. (Please post
to Ruby Talk if you do build a GUI for this. That's not a spoiler.)

Here's another version by me, that doesn't require you press return after the
digits:

  begin
      require "Win32API"

      def read_char
          Win32API.new("crtdll", "_getch", [], "L").Call
      end
  rescue LoadError
      def read_char
          system "stty raw -echo"
          STDIN.getc
      ensure
          system "stty -raw echo"
      end
  end

  loop do
    char = read_char.chr
    case char
    when /^(?:\d|\*|#)$/
      ### Replace the following line with your algorithm. ###
      puts "You entered #{char}."
    when /q/i
      break
    else
      puts "'#{char}' is not a key on the keypad."
    end
  end

-JEG2 ]

This is an area of serious research for some people[2]; don't fret it if
your algorithm isn't perfect or ideal, and don't spend too long on it. :wink:
Make your goal to be better than dumb, which is to say be better
than "Multitap".

  1. http://www.yorku.ca/mack/uist01-f1.jpg
  2. http://www.yorku.ca/mack/uist01.html

Let me offer these two additions to Hans' interface: a CharPhonePad that
uses JEGII's raw character input method, and removal of the input handling
into an object (in this simple case, a class, but it could easily be an
instance of a class).

Cheers,
Dave

# include everything above "if $0 == __FILE__" from Hans' file given in the
quiz question.

class CharPhonePad < PhonePad
  def initialize
    super
    Thread.new do
      loop do
        case c = self.class.read_char
        when 3, 27 # Break on Escape and Ctrl+C
          break
        when 43 # convert '+' to '*' for ease of typing on numpad
          c = 42
        when 45, 46, 47 # convert '-', '.', '/' to '#' for ease of typing
          c = 35
        end
        notify_observers(c.chr)
      end
    end
  end
  def set_text(text, cursor)
    super(text, cursor)
    puts to_s
  end
  def to_s
    @text.dup.insert(@cursor, '_')
  end

  begin
    require "Win32API"
    def self.read_char
      c = Win32API.new("crtdll", "_getch", [], "L").Call
    end
  rescue LoadError
    def self.read_char
      system "stty raw -echo"
      STDIN.getc
    ensure
      system "stty -raw echo"
    end
  end
end

class NoOpTapHandler
  def process_event(ev)
    puts "#{ev} at #{ev.when}"
    p.set_text(p.text+ev.digit, p.cursor+1)
  end
end

if $0 == __FILE__
  p = CharPhonePad.new
  tap_handler = NoOpKeyProcessor

  p.register do |ev|
    tap_handler.process_event(ev)
  end

  Thread.list[0].join
end

Hi,

Despite getting a bit long, my answer to this week's quiz is still mostly a
monolithic file:
http://www.dave.burt.id.au/ruby/phonepad/phonepad.rb

But it needs this for its slick (and default) LetterWise tap method
(unzipped):
http://www.dave.burt.id.au/ruby/phonepad/predict3.rb.zip

So, to describe my solution. I added 2 interfaces. I posted the raw
terminal-based one earlier, which is based on JEGIII's (hopefully)
cross-platform code.
I have also added a TK interface, which plugs neatly in to Hans' platform.

And I added 3 input translators. The first is good old MultiTap, which is
standard on old dodgy phones that don't have T9 dictionary entry. The second
is one I found in a paper from MIT no less that boasts even slower text
entry than MultiTap -- I think that's quite an effort. The third is simply
an implementation of the LetterWise method from the paper linked to by the
quiz itself.

I generated letter frequency data, which is a big part of the LetterWise
method, from a smidgen over 65MB of plain-text modern novels and movie
scripts, using the following two scripts (the latter has 2 versions):
http://www.dave.burt.id.au/ruby/phonepad/1_wordfreq.zip
http://www.dave.burt.id.au/ruby/phonepad/2_predict3_rb.zip
http://www.dave.burt.id.au/ruby/phonepad/2_predict3_yaml.zip

The output of the first (unique words and respective frequencies thereof)
could also be used to create a dictionary for the T9 entry method.

Cheers,
Dave

The LetterWise algorithm seems quite natural. After reading the quiz description I wanted to implement a markov based model and it turns out thats more or less what letter wise is.

In my implementation each key has always the same letters mapped onto it. As with MultiTap multiple taps cycle through the available letters.

But with each keypress the letters mapped to a key are ordered descending by the probability that they will be typed now. Probabilities are learned from a corpus. The probabilities are defined to depend only on the n last entered letters. If the n last entered letters were not encountered in the text, a shorter prefix is used.

As training material I used all the english manpages on my system. So expect debian and linux to be quite easy to enter ;). I let the program learn probabilities for different prefix lengths, the dictionaries can be found on my website:

http://ruby.brian-schroeder.de/quiz/phonetyping/

Additionally I implemented the standard multitap algorithm, from which my algorithm is derived.

As framework I used JEG2 input routines, but added something to make left, right backspace and delete work. So I don't expect it to work on windows.

The learning breaks down to:

    prefix =
    while c = io.getc
      if k = downcase[c]
        node = (@roots[k] ||= MarkovNode.new)
        prefix.each do | p |
          node.popularity += 1
          node = (node[p] ||= MarkovNode.new)
        end
        node.popularity += 1
        prefix.pop while prefix.length >= @max_prefix
        prefix.unshift k
      end
    end

Checking for the frequency is done like this:

  def popularity(char, prefix)
    node = @roots[char[0]]
    pos = prefix.length - 1
    while pos >= 0 and node.is_a?MarkovNode and node[prefix[pos]]
      node = node[prefix[pos]]
      pos -= 1
    end
    node.to_i
  end

All the rest can be found at:

http://ruby.brian-schroeder.de/quiz/phonetyping/

as always all code browsable in full color, or available as an archive.

best regards,

Brian

···

On Sat, 26 Feb 2005 02:53:40 +0900 Ruby Quiz <james@grayproductions.net> wrote:

[Snipped Quiz Description]

Note to self: test, then post.

OK, Quizmasters, a couple of the kindsof error you can only make after 1am
have been corrected in the following bit of code. It now runs (i.e. tested
(on Windows)).

Cheers,
Dave

# include everything above "if $0 == __FILE__" from Hans' file given in the
quiz question.

class CharPhonePad < PhonePad
  def initialize
    super
    Thread.new do
      loop do
        case c = self.class.read_char
        when 3, 4, 26, 27 # Break on Ctrl+C, Ctrl+D, Ctrl+Z, Escape
          break
        when 43 # convert '+' to '*' for ease of typing on numpad
          c = 42
        when 45, 46, 47 # convert '-', '.', '/' to '#' for ease of typing
          c = 35
        end
        notify_observers(c.chr)
      end
    end
  end
  def set_text(text, cursor)
    super(text, cursor)
    puts to_s
  end
  def to_s
    @text.dup.insert(@cursor, '_')
  end

  begin
    require "Win32API"
    def self.read_char
      c = Win32API.new("crtdll", "_getch", [], "L").Call
    end
  rescue LoadError
    def self.read_char
      system "stty raw -echo"
      STDIN.getc
    ensure
      system "stty -raw echo"
    end
  end
end

class NoOpTapHandler
  def initialize(phone_pad)
    @p = phone_pad
  end
  def process_event(ev)
    puts "#{ev} at #{ev.when}"
    @p.set_text(@p.text+ev.digit, @p.cursor+1)
  end
end

if $0 == __FILE__
  p = CharPhonePad.new
  tap_handler = NoOpTapHandler.new(p)

  p.register do |ev|
    tap_handler.process_event(ev)
  end

  Thread.list[0].join
end

JEGIII

Hmm, I'm going down through the generations. :wink: Still just a II, not a III. For the curious, I shorten my name as JEG2. Use whatever you like though, I answer to anything.

And I added 3 input translators.

Wow! Nice work. I look forward to digging through these.

I generated letter frequency data, which is a big part of the LetterWise
method, from a smidgen over 65MB of plain-text modern novels and movie
scripts, using the following two scripts (the latter has 2 versions):
http://www.dave.burt.id.au/ruby/phonepad/1_wordfreq.zip
http://www.dave.burt.id.au/ruby/phonepad/2_predict3_rb.zip
http://www.dave.burt.id.au/ruby/phonepad/2_predict3_yaml.zip

Just FYI, these links don't seem to work for me.

James Edward Gray II

···

On Feb 27, 2005, at 11:59 AM, Dave Burt wrote:

> http://www.dave.burt.id.au/ruby/phonepad/1_wordfreq.zip
> http://www.dave.burt.id.au/ruby/phonepad/2_predict3_rb.zip
> http://www.dave.burt.id.au/ruby/phonepad/2_predict3_yaml.zip

Just FYI, these links don't seem to work for me.

Poking around James just drop into the directory above the files

http://www.dave.burt.id.au/ruby/phonepad/

Thought it looks like the wordfreq zip isn't there.

JEGIII

Hmm, I'm going down through the generations. :wink: Still just a II, not a
III. For the curious, I shorten my name as JEG2. Use whatever you like
though, I answer to anything.

Sorry. Fixed in code comments.

And I added 3 input translators.

Wow! Nice work. I look forward to digging through these.

Thanks. Do enjoy.

I generated letter frequency data, which is a big part of the LetterWise
method, from a smidgen over 65MB of plain-text modern novels and movie
scripts, using the following two scripts (the latter has 2 versions):
http://www.dave.burt.id.au/ruby/phonepad/1_wordfreq.zip
http://www.dave.burt.id.au/ruby/phonepad/2_predict3_rb.zip
http://www.dave.burt.id.au/ruby/phonepad/2_predict3_yaml.zip

Just FYI, these links don't seem to work for me.

My fault: these are scripts, and end in .rb, not .zip:
http://www.dave.burt.id.au/ruby/phonepad/1_wordfreq.rb
http://www.dave.burt.id.au/ruby/phonepad/2_predict3_rb.rb
http://www.dave.burt.id.au/ruby/phonepad/2_predict3_yaml.rb

An additional note on the serialization in these latter two scripts: I tried
two methods.
Hand-output YAML, which ended up about 1.8MB, takes 5-6 seconds to load on
my machine. Hand-output, because if you don't, it grows to over double the
size, and fails on a mapping with nil for a key (it omits the "~").
"p big_hash" ended up around 2.4MB (and the file has only one "\n"), and
takes a touch under 3 seconds to load.

Cheers,
Dave

···

"James Edward Gray II" <james@grayproductions.net> wrote:

On Feb 27, 2005, at 11:59 AM, Dave Burt wrote: