# \[SUMMARY\] Whiteout (#34)
**URL:** https://rubytalk.org/t/summary-whiteout-34/18871
**Category:** ruby-talk
**Created:** [9 June 2005 12:57 UTC](https://rubytalk.org/t/summary-whiteout-34/18871 "2005-06-09T12:57:33Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@James\_Edward\_Gray\_II](https://rubytalk.org/u/James_Edward_Gray_II)
#### Post date: [9 June 2005 12:57 UTC](https://rubytalk.org/t/summary-whiteout-34/18871/1 "2005-06-09T12:57:33Z")
Does this library have any practical value? Probably not. It's been suggested
in the Perl community that hacks like this are a good minor deterrent to those
trying to read source code you would rather keep hidden, but it must be stressed
that this is no form of serious security. Regardless, it's a fun little toy to
play with.
It was mentioned in the discussion that Perl, where ACME::Bleach comes from,
includes a framework for source filtering. It can be used to make modules that
modify source code much as we are doing in this quiz. Perl's [Switch.pm](http://Switch.pm) is a
good example of this, but ironically ACME::Bleach is not.
That naturally leads to the question, can you build source filters in Ruby?
Clearly we can build ACME::Bleach, but not all source filters are as simple I'm
afraid. Consider this:
#!/usr/local/bin/ruby -w
require "fix\_my\_broken\_syntax"
invalid++
Now the thought here is that fix\_my\_broken\_syntax.rb will read my source, change
it so that it does something valid, eval() it, and exit() before the invalid
code is an issue. Here's a trivial example of fix\_my\_broken\_syntax.rb:
#!/usr/local/bin/ruby -w
puts "Fixed!"
exit
Does that work? Unfortunately, no:
$ ruby invalid.rb
invalid.rb:5: syntax error
invalid++
^
Ruby never gets to loading the library, because it's not happy with the syntax
of the first file. That makes writing a source filter for anything that isn't
valid Ruby syntax complicated and if it is valid Ruby syntax, you can probably
just code it up in Ruby to begin with.
Except for whiteout.rb, our version of ACME::Bleach.
You can't build Ruby constructs out of whitespace alone, so some form of source
filtering is required. Luckily, we can get away with the approach described
above for this source filter, because a bunch of whitespace (with no code) is
valid Ruby syntax. It just doesn't do anything. Ruby will skip right over our
whitespace and load the library that restores and runs the code.
Most people took this approach. Let's examine one such example by Robin
Stocker:
#!/usr/bin/ruby
> **···**
>
> #
> # This is my solution for Ruby Quiz #34, Whiteout.
> # Author:: Robin Stocker
> #
>
> #
> # The Whiteout module includes all functionality like:
> # - whiten
> # - run
> # - encode
> # - decode
> #
> module Whiteout
>
> @@bit\_to\_code = { '0' =\> " ", '1' =\> "\t" }
> @@code\_to\_bit = @@bit\_to\_code.invert
> @@chars\_to\_ignore = ["\n", "\r"]
>
> #
> # Whitens the content of a file specified by \_filename\_.
> # It leaves the shebang intact, if there is one.
> # At the beginning of the file it inserts the require 'whiteout'.
> # See #encode for details about how the whitening works.
> #
> def Whiteout.whiten( filename )
> code = ''
> File.open( filename, 'r' ) do |file|
> file.each\_line do |line|
> if code.empty?
> # Add shebang if there is one.
> code \<\< line if line =~ /#!\s\*.+/
> code \<\< "#{$/}require 'whiteout'#{$/}"
> else
> code \<\< encode( line )
> end
> end
> end
> File.open( filename, 'w' ) do |file|
> file.write( code )
> end
> end
>
> # ...
>
> First, we can see that the module defines some module variables, which are
> really used as constants here. Their contents hint at the encoding algorithm
> we'll see later.
>
> Then we have a method for managing the transformation of the source into
> whitespace. It starts by opening the passed file and reading the code
> line-by-line. If the first line is a shebang line, it's saved in the variable
> code. Next, a "require 'whiteout'" line is added to code. Finally, all other
> lines from the file are appended to code after being passed through an encode()
> method we'll examine shortly. With the contents read and transformed, the
> method then reopens the source for writing and dumps the modifications into it.
>
> The next method is the reverse process:
>
> # ...
>
> #
> # Reads the file \_filename\_, decodes and runs it through eval.
> #
> def Whiteout.run( filename )
> text = ''
> File.open( filename, 'r' ) do |file|
> decode = false
> file.each\_line do |line|
> if not decode
> # We don't want to decode the "require 'whiteout'",
> # so start decoding not before we passed it.
> decode = true if line =~ /require 'whiteout'/
> else
> text \<\< decode( line )
> end
> end
> end
> # Run the code!
> eval text
> end
>
> # ...
>
> This method again reads the passed file. It skips over the "require 'whiteout'"
> line, then copies the rest of the file into the variable text, after passing it
> through decode() line-by-line. The final line of the method calls eval() on
> text, which should now contain the restored program.
>
> On to encode() and decode():
>
> #
> # Encodes text to "whitecode". It works like this:
> # - Chars in @@char\_to\_ignore are ignored
> # - Each byte is converted to its bit representation,
> # so that we have something like 01100001
> # - Then, it is converted to whitespace according to @@bit\_to\_code
> # - 0 results in a " " (space)
> # - 1 results in a "\t" (tab)
> #
> def Whiteout.encode( text )
> white = ''
> text.scan(/./m) do |char|
> if @@chars\_to\_ignore.include?( char )
> white \<\< char
> else
> char.unpack('B8').first.scan(/./) do |bit|
> code = @@bit\_to\_code[bit]
> white \<\< code
> end
> end
> end
> return white
> end
>
> #
> # Does the inverse of #encode, it takes "white"
> # and returns the decoded text.
> #
> def Whiteout.decode( white )
> text = ''
> char = ''
> white.scan(/./m) do |code|
> if @@chars\_to\_ignore.include?( code )
> text \<\< code
> else
> char \<\< @@code\_to\_bit[code]
> if char.length == 8
> text \<\< [char].pack("B8")
> char = ''
> end
> end
> end
> return text
> end
>
> end
>
> # ...
>
> The comments in there detail the exact process we're looking at here, so I'm not
> going to repeat them.
>
> Note that @@char\_to\_ignore contains "\n" and "\r" so they are not translated.
> The effect of that is that line-endings are untouched by this conversion. Some
> solutions used such characters in their encoding algorithm. The gotcha there is
> that any line-ending translation done to the modified source (say FTP through
> ASCII mode) will break the hidden code. Robin's solution doesn't have that
> problem.
>
> Here's the code that ties all those methods into a solution:
>
> # ...
>
> #
> # And here's the logic part of whiteout.
> # If it was run directly, whites out the files in ARGV.
> # And if it was required, decodes the whitecode and runs it.
> #
> if \_\_FILE\_\_ == $0
> ARGV.each do |filename|
> Whiteout.whiten( filename )
> end
> else
> Whiteout.run( $0 )
> end
>
> Again, the comment saves me some explaining.
>
> That was Robin's first solution to a Ruby Quiz, but I never would have known
> that from looking at the code. Thanks for sharing Robin!
>
> Obviously, a conversion of this type grossly inflates the size of the source.
> Around eight times the size, to be exact. A couple of solutions used zlib to
> control the expansion, which I thought was clever. By compressing the source
> and then encoding() (and using a base three conversion) Dominik Bathom got
> results around three times the inflation instead.
>
> Ara.T.Howard took a different approach, using whiteout.rb as a database to store
> the trimmed files. That was a very interesting process, demonstrated well in
> the submission email. The advantages to this approach would be no inflation
> penalty and the code stays readable (just not in the original location). The
> disadvantage I see is that it requires the exact same library to be present both
> at encoding and decoding, which probably makes sharing the altered code
> impractical.
>
> As always, my thanks to all who gave this little diversion an attempt. I'm sure
> we'll see tons of whitespace only code on RubyForge in the future, thanks to our
> efforts.
>
> Tomorrow begins part one of our first two-part Ruby Quiz. Stay tuned...
---
### Author:  [@Florian\_Gross2](https://rubytalk.org/u/Florian_Gross2)
#### Post date: [9 June 2005 13:59 UTC](https://rubytalk.org/t/summary-whiteout-34/18871/2 "2005-06-09T13:59:53Z")
Ruby Quiz wrote:
> That naturally leads to the question, can you build source filters in Ruby? Clearly we can build ACME::Bleach, but not all source filters are as simple I'm
> afraid. Consider this:
>
> #!/usr/local/bin/ruby -w
>
> require "fix\_my\_broken\_syntax"
>
> invalid++
> [...]
> Ruby never gets to loading the library, because it's not happy with the syntax
> of the first file. That makes writing a source filter for anything that isn't
> valid Ruby syntax complicated and if it is valid Ruby syntax, you can probably
> just code it up in Ruby to begin with.
But note that if you do #!/usr/local/bin/ruby -w -r fix\_my\_broken\_syntax you will be able to make it work.
---
### Author:  [@Ara.T.Howard6](https://rubytalk.org/u/Ara.T.Howard6)
#### Post date: [9 June 2005 14:30 UTC](https://rubytalk.org/t/summary-whiteout-34/18871/3 "2005-06-09T14:30:28Z")
a hack but:
harp:~ \> cat fix\_my\_broken\_syntax.rb
src = open($0).read
src.gsub! %r/([\_a-z][\_a-zA-Z]\*)\+\+/, '((\1+=1;\1 - 1))'
eval src
exit
harp:~ \> cat a.rb
#!/usr/local/bin/ruby -r./fix\_my\_broken\_syntax.rb
n = 41
p n++
p n
harp:~ \> ./a.rb
41
42
cheers.
-a
> **···**
>
> On Thu, 9 Jun 2005, Ruby Quiz wrote:
>
> > Does this library have any practical value? Probably not. It's been suggested
> > in the Perl community that hacks like this are a good minor deterrent to those
> > trying to read source code you would rather keep hidden, but it must be stressed
> > that this is no form of serious security. Regardless, it's a fun little toy to
> > play with.
> >
> > It was mentioned in the discussion that Perl, where ACME::Bleach comes from,
> > includes a framework for source filtering. It can be used to make modules that
> > modify source code much as we are doing in this quiz. Perl's Switch.pm is a
> > good example of this, but ironically ACME::Bleach is not.
> >
> > That naturally leads to the question, can you build source filters in Ruby?
> > Clearly we can build ACME::Bleach, but not all source filters are as simple I'm
> > afraid. Consider this:
> >
> > #!/usr/local/bin/ruby -w
> >
> > require "fix\_my\_broken\_syntax"
> >
> > invalid++
> >
> > Now the thought here is that fix\_my\_broken\_syntax.rb will read my source, change
> > it so that it does something valid, eval() it, and exit() before the invalid
> > code is an issue. Here's a trivial example of fix\_my\_broken\_syntax.rb:
> >
> > #!/usr/local/bin/ruby -w
> >
> > puts "Fixed!"
> > exit
> >
> > Does that work? Unfortunately, no:
> >
> > $ ruby invalid.rb
> > invalid.rb:5: syntax error
> > invalid++
> > ^
> >
> > Ruby never gets to loading the library, because it's not happy with the syntax
> > of the first file. That makes writing a source filter for anything that isn't
> > valid Ruby syntax complicated and if it is valid Ruby syntax, you can probably
> > just code it up in Ruby to begin with.
>
> # --
>
> > email :: ara [dot] t [dot] howard [at] noaa [dot] gov
> > phone :: 303.497.6469
> > My religion is very simple. My religion is kindness.
> > --Tenzin Gyatso
>
> ===============================================================================
---
### Author:  [@Brian\_Schroder1](https://rubytalk.org/u/Brian_Schroder1)
#### Post date: [9 June 2005 20:06 UTC](https://rubytalk.org/t/summary-whiteout-34/18871/4 "2005-06-09T20:06:46Z")
> [Snip]
> Obviously, a conversion of this type grossly inflates the size of the source.
> Around eight times the size, to be exact. A couple of solutions used zlib to
> control the expansion, which I thought was clever. By compressing the source
> and then encoding() (and using a base three conversion) Dominik Bathom got
> results around three times the inflation instead.
Using a base eight encoding plus zipping you can even reach a
deflation of source-length. See
[http://ruby.brian-schroeder.de/quiz/whiteout/](http://ruby.brian-schroeder.de/quiz/whiteout/)
regards and thanks for the summary,
Brian
> **···**
>
> --
> [http://ruby.brian-schroeder.de/](http://ruby.brian-schroeder.de/)
>
> Stringed instrument chords: [http://chordlist.brian-schroeder.de/](http://chordlist.brian-schroeder.de/)
---
### Author:  [@Klaus\_Stein1](https://rubytalk.org/u/Klaus_Stein1)
#### Post date: [10 June 2005 09:20 UTC](https://rubytalk.org/t/summary-whiteout-34/18871/5 "2005-06-10T09:20:30Z")
What about using \_\_END\_\_ for this?
Klaus
> **···**
>
> Ruby Quiz \ wrote:
>
> > #!/usr/local/bin/ruby -w
> >
> > require "fix\_my\_broken\_syntax"
> >
> > invalid++
> >
> > [Fix it]
> >
> > Does that work? Unfortunately, no:
> >
> > $ ruby invalid.rb
> > invalid.rb:5: syntax error
> > invalid++
> > ^
> >
> > Ruby never gets to loading the library, because it's not happy with the
> > syntax of the first file.
>
> --
>
> > **[Lapizistik: Willkommen](http://lapiz.istik.de/)**
> >
> > Personal website and blog of Klaus Stein. Have fun.
>
> The Answer is 42. And I am the Answer. Now I am looking for the Question.
---
### Author:  [@Logan\_Capaldo](https://rubytalk.org/u/Logan_Capaldo)
#### Post date: [10 June 2005 01:28 UTC](https://rubytalk.org/t/summary-whiteout-34/18871/6 "2005-06-10T01:28:03Z")
[snip]
> a hack but:
>
> harp:~ \> cat fix\_my\_broken\_syntax.rb
> src = open($0).read
> src.gsub! %r/([\_a-z][\_a-zA-Z]\*)\+\+/, '((\1+=1;\1 - 1))'
> eval src
> exit
>
> harp:~ \> cat a.rb
> #!/usr/local/bin/ruby -r./fix\_my\_broken\_syntax.rb
> n = 41
> p n++
> p n
>
> harp:~ \> ./a.rb
> 41
> 42
>
> cheers.
[snip]
I have some suggestions for alternate methods. I haven't actually
tried any of these yet, so take this with a grain of salt.
The more interesting one I think would be to use ParseTree, assuming
it allows (or eventually will) allow you to insert a modified
parsetree back into the interpreter. You could then traverse the tree
and look for items semantically instead of by regexps. There are
disadvantages to this of course. You couldn't add new operators and
such for instance, although I would imagine it would be good for
things like AOP (It also probably would be impossible to implement
whiteout using this method). A related option is to write a parser in
ruby for ruby that emits ParseTree sexps that can once again be
inserted into the interpreter. You could then modify this parser to
add whatever syntax constructs you like (new operators etc.) as long
as they could be mapped onto existing ruby syntax (since this is the
point of source filters usually, I see no problem with that
limitation, any more complicated and its just another language written
in ruby).
The other option to consider is a filter using pipes. Have two files,
one with the filterable source (ie written in latin or whitespace or
whatever) and another with the regexp based transformer, and wrap it
up in a script. eg:
$ cat illegible.rb
#@#@#@# -- ? : 2
dfsdasdasd
$ cat filter.rb
#!/usr/bin/env ruby
class LineNoise
def transform
....
end
end
x = LineNoise.new
IO.popen("ruby") do |rb|
File.open("illegible.rb") do |ill|
ill.each do |line|
rb.print x.transform(line)
end
end
end
$
This gets rid of the eval nastiness but adds its own nastiness (like,
where do I find illegible.rb? etc.).
Just some ideas. Of course we could all write our own languages that
are just ruby with some syntax differences 😉
> **···**
>
> On 6/9/05, Ara.T.Howard@noaa.gov \ wrote: