[ANN] FileSystem 0.1.0: Beta for me, Alpha for you

ruby-talk
1

Greetings!

Next down my non-stop pipeline of absolutely necessary libraries that only I seem to need: FileSystem, a library that will mock out File, FileUtils, Dir, and other file-dependent built-in classes. It aims to simulate an entire file-system in memory for the purposes of testing.

http://rubyforge.org/projects/filesystem/

== HOW DOES IT WORK? ==

To use it in live code, you call get methods on the FileSystem module:

   FileSystem.get_dir => Dir
   FileSystem.get_file => File
   FileSystem.get_file_utils => FileUtils
   FileSystem.get_dir.entries( '.' ) => [ '.', '..', 'file1', ... ]

(I'm thinking about adding a voodoo.rb file that actually re-assigns the constants Dir, File, and FileUtils, though I suspect this will not be that usable in many cases.)

Then, to mock out that activity for a test code, simply call FileSystem.mock= :

   FileSystem.mock = true
   FileSystem.get_dir => FileSystem::DirAdapter
   FileSystem.get_file => FileSystem::FileAdapter
   FileSystem.get_file_utils => FileSystem::FileUtilsAdapter
   FileSystem.get_dir.entries( '.' ) => [ '.', '..', ... ]

== HOW COMPLETE IS IT? ==
Oh dear, not at all. Hence the release name: "Beta for me, Alpha for you". I am now using it day-to-day, both at Rhizome[1] and my own blog-publishing software, Dauxite[2]. So I know it works for me. It currently handles basic issues of looking up paths, writing and reading files, modification times, etc. It doesn't know anything about symlinks, creation times, permissions. I suspect that if you tried to mock a Windows system with it, that it would squeal like a 2-year-old.

I'm putting out today's release in hopes of getting bug reports. Please file bug reports! Lots and lots of bug reports! Please be specific, though. If you file a bug titled "Support Windows, you l4m3r" I won't know where to start.

Thanks,
Francis "I'd mock out the air itself if my lungs were hot-swappable" Hwang
http://fhwang.net/

[1] Rhizome is the premier online resource in the field of new media arts: It gets more than 1 million page views a month and contains the world's largest online archive of new media art, at almost 1500 works.
[2] Dauxite is my own idiosyncratic, unreleased software. So it's not as heavily tested as Rhizome, but I use it at least once a week, so it's a good way to test something like FileSystem.

2

Francis Hwang ha scritto:

Greetings!

Next down my non-stop pipeline of absolutely necessary libraries that only I seem to need: FileSystem, a library that will mock out File, FileUtils, Dir, and other file-dependent built-in classes. It aims to simulate an entire file-system in memory for the purposes of testing.

http://rubyforge.org/projects/filesystem/

first: this seem incredibly cool.
second: would'nt something like "FS::Mock" or "test/filesystem" be a better name? when I first saw this on rubyforge I thought it was some kind of VFS :slight_smile:

3

Francis Hwang wrote:

Greetings!

Next down my non-stop pipeline of absolutely necessary libraries that

only I seem to need: FileSystem, a library that will mock out File,
FileUtils, Dir, and other file-dependent built-in classes. It aims to

simulate an entire file-system in memory for the purposes of testing.

Oh, dear. I believe we have our first significant name collision -
your package and Mike Hall's filesystem package, and they do different
things.

Caveat Programmor.

Regards,

Dan

4

Francis Hwang wrote:

Next down my non-stop pipeline of absolutely necessary libraries that
only I seem to need: FileSystem, a library that will mock out File,
FileUtils, Dir, and other file-dependent built-in classes. It aims to
simulate an entire file-system in memory for the purposes of testing.

Very cool! I was looking for something similar to this some time ago,
but figured it would be too much work to implement it. I'm glad you made
a different decision! :slight_smile:

Did you examine the possibility to override the methods in the standard
IO and File classes, so that it could be more transparent?

/Anders

···

--

5

Hi Francis,

I like the idea, and the idea of mocking out things like filesystems is very important. Just a few comments on the naming of things:

If the class is mainly just a test construct and/or a proxy for actual Filesystem calls, maybe "Test" or "Proxy" should be part of the class name. Secondly, method names that start with "get_" seem pretty un-rubyish. They feel more like Java getFoo() and setFoo() type functions. Maybe it would make sense to rename them just dir, file, file_utils and dir_entries?

Ben

P.S. If you ever decide to mock out a human being, let me know. This morning in the shower I was wondering how I can mock out the person who has to push a button on an embedded device I'm working on. Sure, I can easily mock out the register that pushing the button supposedly changes, but that kinda defeats the purpose of the test.

···

On Feb 16, 2005, at 09:39, Francis Hwang wrote:

To use it in live code, you call get methods on the FileSystem module:

  FileSystem.get_dir => Dir
  FileSystem.get_file => File
  FileSystem.get_file_utils => FileUtils
  FileSystem.get_dir.entries( '.' ) => [ '.', '..', 'file1', ... ]

(I'm thinking about adding a voodoo.rb file that actually re-assigns the constants Dir, File, and FileUtils, though I suspect this will not be that usable in many cases.)

Then, to mock out that activity for a test code, simply call FileSystem.mock= :

  FileSystem.mock = true
  FileSystem.get_dir => FileSystem::DirAdapter
  FileSystem.get_file => FileSystem::FileAdapter
  FileSystem.get_file_utils => FileSystem::FileUtilsAdapter
  FileSystem.get_dir.entries( '.' ) => [ '.', '..', ... ]

6

Francis Hwang ha scritto:

Greetings!
Next down my non-stop pipeline of absolutely necessary libraries that only I seem to need: FileSystem, a library that will mock out File, FileUtils, Dir, and other file-dependent built-in classes. It aims to simulate an entire file-system in memory for the purposes of testing.
http://rubyforge.org/projects/filesystem/

first: this seem incredibly cool.

Thanks! It's already been pretty helpful for me, hopefully it'll be useful to others too.

second: would'nt something like "FS::Mock" or "test/filesystem" be a better name? when I first saw this on rubyforge I thought it was some kind of VFS :slight_smile:

I think it's related to how you're supposed to access the lib in your live code. The name "FileSystem" to me seems less distracting in production code than something like "FS::Mock" ... It's only being mocked out in your tests.

I'm not 100% happy with the current form of access either: Typing "FileSystem.get_file" when you really just want "File" is sort of cumbersome. But it just seemed like the least broken thing I could go forward with while I figured out something else. When we talked about this at Ruby-NYC a few weeks ago, Patrick was saying I could write a file to change the definitions of File, FileUtils, Dir, etc. Maybe that'll be the way to go.

Anyway, I'm very very open to suggestions regarding the name and/or how to access it.

Francis Hwang

···

On Feb 16, 2005, at 9:49 AM, gabriele renzi wrote:

7

P.S. If you ever decide to mock out a human being, let me know. This morning in the shower I was wondering how I can mock out the person who has to push a button on an embedded device I'm working on. Sure, I can easily mock out the register that pushing the button supposedly changes, but that kinda defeats the purpose of the test.

Robots, man, robots. Preferably small dancing ones.

···

--
66. The regions beyond these places are either difficult of access because of their excessive winters and great cold, or else cannot be sought out because, of some divine influence of the gods.

8

Ben Giddings wrote:

P.S. If you ever decide to mock out a human being, let me know.

You smell and your momma dresses you funny. How's that?

Oh...mock "out". :wink:

Regards,

Dan

9

Aw, for Pete's sake. I guess that's what I guess for not checking RAA ...

So I guess I better rename my FileSystem to something else. Suggestions?

Francis Hwang

···

On Feb 16, 2005, at 9:59 AM, Daniel Berger wrote:

Oh, dear. I believe we have our first significant name collision -
your package and Mike Hall's filesystem package, and they do different
things.

10

Very cool! I was looking for something similar to this some time ago,
but figured it would be too much work to implement it. I'm glad you made
a different decision! :slight_smile:

I've actually done this, in half-hearted ways, three separate times in three separate apps ... So this lib is my attempt to do it once, and correct ...

Did you examine the possibility to override the methods in the standard
IO and File classes, so that it could be more transparent?

More transparency is important, yeah, but maybe I won't want to redefine methods on already existing classes & modules. Method coverage is currently really spotty, so if I redefined, say, File.mtime but not File.ctime, your test code would be mixing up its calls against the MockFileSystem with its calls to the real file system ... pretty confusing, I think.

Might be better to simply undefine and redefine those constants:

   Object.send( :remove_const, :File )
   class File
     ...
   end

Though it's quite possible that that could be really nasty, too. I'll have to poke around for a while to see.

Francis Hwang

···

On Feb 16, 2005, at 10:07 AM, Anders Bengtsson wrote:

11

If the class is mainly just a test construct and/or a proxy for actual Filesystem calls, maybe "Test" or "Proxy" should be part of the class name. Secondly, method names that start with "get_" seem pretty un-rubyish. They feel more like Java getFoo() and setFoo() type functions. Maybe it would make sense to rename them just dir, file, file_utils and dir_entries?

Yeah, good points. This may all be mooted if I decide to just replace Dir, File, FileUtils, etc with some strange magicks. We'll see.

P.S. If you ever decide to mock out a human being, let me know. This morning in the shower I was wondering how I can mock out the person who has to push a button on an embedded device I'm working on. Sure, I can easily mock out the register that pushing the button supposedly changes, but that kinda defeats the purpose of the test.

Well, this won't directly solve your problem, but for an idea of how you might solve this, you could look at EasyPrompt, which does the same for a command-line user: http://easyprompt.rubyforge.org/ In particular, check out the MockCommandLineUser.

Funny thing is, code that uses EasyPrompt (and is thus more mockable) looks a lot like the example code people use when talking about continuations in web apps ... leading me to believe that continuation-based web apps would be tons easier to test across multi-step processes. Not that I've ever used continuations, just saying.

Francis Hwang

···

On Feb 16, 2005, at 10:08 AM, Ben Giddings wrote:

12

(sorry for breaking threading)

does it already, or would it be possible, for this to act as an overlay file system?
as in, in a test, i can create a number of virtual files, any lookups on real files
succeed - thusly meaning that e.g loading of new libraries doesn't screw up :wink: - but
any access to the virtual directories / files would return pseudo objects.

whats the feasibility of this? it would vastly simplify many testcases i imagine.

Alex

13

Francis Hwang schrieb:

I'm not 100% happy with the current form of access either: Typing "FileSystem.get_file" when you really just want "File" is sort of cumbersome. But it just seemed like the least broken thing I could go forward with while I figured out something else. When we talked about this at Ruby-NYC a few weeks ago, Patrick was saying I could write a file to change the definitions of File, FileUtils, Dir, etc. Maybe that'll be the way to go.

It would be nice if you could implement it in this way. Then the code under test would be completely independent from your library. I once used this technique for a Time mock [1] and I liked it very much.

Regards,
Pit

[1] In my tests I had a lot of calls to Kernel#sleep, but I wanted them to run as fast as possible. So I implemented a Time mock where you could change the current time. In the tests, a call to sleep simply adjusted the current time accordingly.

14

MockFS / MockFiles / FileMocker / MocktasticFileCreationWizardry??

(personally i like MockFiles)

···

On Feb 16, 2005, at 5:07 PM, Francis Hwang wrote:

On Feb 16, 2005, at 9:59 AM, Daniel Berger wrote:

Oh, dear. I believe we have our first significant name collision -
your package and Mike Hall's filesystem package, and they do different
things.

Aw, for Pete's sake. I guess that's what I guess for not checking RAA ...

So I guess I better rename my FileSystem to something else. Suggestions?

Francis Hwang
http://fhwang.net/

15

Francis Hwang ha scritto:

second: would'nt something like "FS::Mock" or "test/filesystem" be a better name? when I first saw this on rubyforge I thought it was some kind of VFS :slight_smile:

I think it's related to how you're supposed to access the lib in your live code. The name "FileSystem" to me seems less distracting in production code than something like "FS::Mock" ... It's only being mocked out in your tests.

mh.. but I have the feeling that it /should/ be distracting.
Or, better put, that it should make clear that we're working with a test facility. Anyway I understand your reason and you're the author :slight_smile:

I'm not 100% happy with the current form of access either: Typing "FileSystem.get_file" when you really just want "File" is sort of cumbersome. But it just seemed like the least broken thing I could go forward with while I figured out something else. When we talked about this at Ruby-NYC a few weeks ago, Patrick was saying I could write a file to change the definitions of File, FileUtils, Dir, etc. Maybe that'll be the way to go.

Anyway, I'm very very open to suggestions regarding the name and/or how to access it.

just a suggestion, maybe you can take a look at how libvfs (on raa) does this.

16

Mock::FileSystem?

martin

···

Francis Hwang <sera@fhwang.net> wrote:

On Feb 16, 2005, at 9:59 AM, Daniel Berger wrote:

> Oh, dear. I believe we have our first significant name collision -
> your package and Mike Hall's filesystem package, and they do different
> things.

Aw, for Pete's sake. I guess that's what I guess for not checking RAA
..

So I guess I better rename my FileSystem to something else. Suggestions?

17

Francis Hwang wrote:

So I guess I better rename my FileSystem to something else. Suggestions?

FileSystem is the wrong name anyhow - since many operating systems can
have more than one type of filesystem, where the differences matter at
the file *service* layer. Consider differences in valid name characters,
path lengths, etc, for filesystems like iso9660, SMBFS, NTFS, etc.

We have had a FileService class in our C++ code for years, not only for
mocking but also for impersonation and remoting. I suggest that might
be a good name for a Ruby class also.

Clifford Heath.

18

Alexander Kellett wrote:

MockFS / MockFiles / FileMocker / MocktasticFileCreationWizardry??

(personally i like MockFiles)

I like MockFS. :slight_smile:

Dan

19

Maybe this is what you're talking about:

At Rhizome, we send out lots of customized emails, with templates. So maybe you'll have a template file in /var/www/includes/email/welcome.tmpl . But if you want to play around with the codebase on, say, another server, this is annoying, because then you'd have to build up that directory path just to put one file in it.

So instead, I save a representative copy of that template somewhere in the test code directory, say test/test_data/email. Then, if we've got an email class that uses that template:

class WelcomeEmail < RhizMail::SimpleTemplateMessage
  def initialize( user )
    template = FileSystem.get_file.open( '/var/www/includes/email/welcome.tmpl' ) { |f| f.gets nil }
    super( template )
  end
end

the setup for the test code might look like this:

class TestWelcomeEmail < Test::Unit::TestCase
  def setup
    FileSystem.mock = true
    contents = File.open( 'test/test_data/email/welcome.tmpl' ) { |f|
      f.gets nil
    }
    FileSystem.mock_file_system.fill_path( '/var/www/includes/email' )
    FileSystem.get_file.open( '/var/www/includes/email/welcome.tmpl', File::CREAT | File::WRONLY ) { |f|
      f << contents
    }
  end
end

So that when the WelcomeEmail#initialize reads the contents of /var/www/includes/email/welcome.tmpl, it's actually reading contents from the virtual file system that was set up in your test case.

Is that what you had in mind?

Francis Hwang

···

On Feb 16, 2005, at 11:25 AM, Alexander Kellett wrote:

does it already, or would it be possible, for this to act as an overlay file system?
as in, in a test, i can create a number of virtual files, any lookups on real files
succeed - thusly meaning that e.g loading of new libraries doesn't screw up :wink: - but
any access to the virtual directories / files would return pseudo objects.

whats the feasibility of this? it would vastly simplify many testcases i imagine.

20

mh.. but I have the feeling that it /should/ be distracting.
Or, better put, that it should make clear that we're working with a test facility. Anyway I understand your reason and you're the author :slight_smile:

Well, it's interesting, I think some of us would like something like this to be really distracting, and others would like it to be completely transparent. The two are pretty opposite in my mind, but I suppose there's no reason I couldn't support both, with an option to use one or the other depending on what files get included. Maybe by default we'd use, say MockFS.file and MockFS.file_utils to get the classes, but then if you wanted you could have your code simply use File and FileUtils, your test code could require 'mockfs/voodoo' or some such file to redefine those constants. I'll have to look into the details.

I'm not 100% happy with the current form of access either: Typing "FileSystem.get_file" when you really just want "File" is sort of cumbersome. But it just seemed like the least broken thing I could go forward with while I figured out something else. When we talked about this at Ruby-NYC a few weeks ago, Patrick was saying I could write a file to change the definitions of File, FileUtils, Dir, etc. Maybe that'll be the way to go.
Anyway, I'm very very open to suggestions regarding the name and/or how to access it.

just a suggestion, maybe you can take a look at how libvfs (on raa) does this.

Thanks for the tip; I'll definitely take a look.

Francis Hwang

···

On Feb 16, 2005, at 11:44 AM, gabriele renzi wrote: