Reccommended work flow for unit tests and databases in ruby

ruby-talk
1

How does one go around creating unit tests for database tables during
developement. Is there a rock solid work flow plan that everyone uses.
Im new to dbs and relatively so with unit tests. I have my project and
it usually starts with a rough idea of what fields need to be in the
table. As i code away items in the db change so it makes it difficult to
write unit tests that will withstand these changes.

I know i need a test version of a database with static items but what id
like to do is for this test version to always reflect whats happening in
the development one. So if I add a field in the development one, the
test one gets one too.

I could just go and code this and probably get something that works but
is probably ugly. I dont like to reinvent the wheel, a bad one at that
so if anyone has some tips or pointers in the right direction id really
appreciate it.

This is for small projects so it doesnt have to industrial strength
bullet proof ninja stuff, small but elegant will be good enough

···

--
Posted via http://www.ruby-forum.com/.

2

Adam Akhtar wrote:

How does one go around creating unit tests for database tables during
developement. Is there a rock solid work flow plan that everyone uses.

A myth of software development is you can plan out and invent your database schema before writing your code. This myth is so pervasive that some communities (>cough< Oracle) actively prevent programmers from changing their schemas.

Refactoring databases is a Best Practice, and all projects should include support scripts that overcome our database platform limitations.

Im new to dbs and relatively so with unit tests. I have my project and
it usually starts with a rough idea of what fields need to be in the
table. As i code away items in the db change so it makes it difficult to
write unit tests that will withstand these changes.

Investigate (and use!) the ActiveRecord system exemplified in Rails fixtures and migrations. When your program starts, only only _only_ put in the database the very few fields you are actually using. Rails's "script/generate model" will write a migration which adds that table to the db. Write tests for the model's behavior, and these will require fields in the table. Edit the model's migration file to add them. Repeat in tiny cycles, testing and integrating as often as possible. Never integrate if the tests break - revert if necessary. (This is more efficient than debugging!)

I know i need a test version of a database with static items but what id
like to do is for this test version to always reflect whats happening in
the development one. So if I add a field in the development one, the
test one gets one too.

Put sample data into the fixture files, and use it in the tests. Your scripts should then be able to migrate your test, development, and production databases in lockstep.

If you are not writing a Rails project, research how to get all these benefits without its framework. Rails can import fixture data into the development database with 'rake db:migrate'.

When you deploy to a live site, at this point stop editing your initial migration files, and only add new files that add and adjust your fields.

I could just go and code this and probably get something that works but
is probably ugly. I dont like to reinvent the wheel, a bad one at that
so if anyone has some tips or pointers in the right direction id really
appreciate it.

This is for small projects so it doesnt have to industrial strength
bullet proof ninja stuff, small but elegant will be good enough

These techniques - after you spend a little time implementing them - keep potentially big projects very small and manageable. I would use them on projects of any size!

···

--
   Phlip

3

Hi --

···

On Sun, 11 Jan 2009, Adam Akhtar wrote:

How does one go around creating unit tests for database tables during
developement. Is there a rock solid work flow plan that everyone uses.
Im new to dbs and relatively so with unit tests. I have my project and
it usually starts with a rough idea of what fields need to be in the
table. As i code away items in the db change so it makes it difficult to
write unit tests that will withstand these changes.

I know i need a test version of a database with static items but what id
like to do is for this test version to always reflect whats happening in
the development one. So if I add a field in the development one, the
test one gets one too.

I could just go and code this and probably get something that works but
is probably ugly. I dont like to reinvent the wheel, a bad one at that
so if anyone has some tips or pointers in the right direction id really
appreciate it.

This is for small projects so it doesnt have to industrial strength
bullet proof ninja stuff, small but elegant will be good enough

Rails has a bunch of Rake tasks that copy the schema from one database
to another. I don't know whether your project is a Rails application,
but even if it isn't, you might be able to integrate one of those
fairly easily, so that when you run your tests, the test database will
pick up any changes in the development database.

David

--
David A. Black / Ruby Power and Light, LLC
Ruby/Rails consulting & training: http://www.rubypal.com
Coming in 2009: The Well-Grounded Rubyist (http://manning.com/black2\)

http://www.wishsight.com => Independent, social wishlist management!

4

David A. Black wrote:

Rails has a bunch of Rake tasks that copy the schema from one database
to another. I don't know whether your project is a Rails application,
but even if it isn't, you might be able to integrate one of those
fairly easily, so that when you run your tests, the test database will
pick up any changes in the development database.

However...

Every once in a while I feel the urge to think outside those ActiveRecord scripts' box. For example, I want to write a method nab_fixtures('models.yml'), and it will reach out to a yml file and load it, on command, without clawing its way thru the uber-kewt fixtures() directive at the top of my suite.

When I feel that urge, I often have to repress it, because when I try to re-use the code inside fixtures(), it's all "refactored" into a maze of fragile and un-reusable methods. When I rewrite the system that loads a yml file into a table myself, I replace fixtures() bugs with my own!

How the heck does anyone use ActiveRecord, with fixtures and migrations, outside Rails? How, for example, can that magic luvvable plugable Merb do it??

···

--
   Phlip

5

Phlip wrote:

Investigate (and use!) the ActiveRecord system exemplified in Rails fixtures and migrations. When your program starts, only only _only_ put in the database the very few fields you are actually using. Rails's "script/generate model" will write a migration which adds that table to the db. Write tests for the model's behavior, and these will require fields in the table. Edit the model's migration file to add them. Repeat

This sounds great, but I've read in several places (sorry can't find references) that you should create a new migration to make changes to an existing table.

I've tried before to modify an existing migration but unless I totally wipe the DB, it doesn't take notice, as the schema version isn't incremented.

Is this expected behavior or am I doing something wrong?

···

--
Matt

6

Hi --

David A. Black wrote:

Rails has a bunch of Rake tasks that copy the schema from one database
to another. I don't know whether your project is a Rails application,
but even if it isn't, you might be able to integrate one of those
fairly easily, so that when you run your tests, the test database will
pick up any changes in the development database.

However...

Every once in a while I feel the urge to think outside those ActiveRecord scripts' box.

Rake will let you do that :slight_smile:

For example, I want to write a method nab_fixtures('models.yml'), and it will reach out to a yml file and load it, on command, without clawing its way thru the uber-kewt fixtures() directive at the top of my suite.

When I feel that urge, I often have to repress it, because when I try to re-use the code inside fixtures(), it's all "refactored" into a maze of fragile and un-reusable methods. When I rewrite the system that loads a yml file into a table myself, I replace fixtures() bugs with my own!

I'd have to see the code to know exactly what's going on. Is it a
problem reconciling foreign key references?

David

···

On Mon, 12 Jan 2009, Phlip wrote:

--
David A. Black / Ruby Power and Light, LLC
Ruby/Rails consulting & training: http://www.rubypal.com
Coming in 2009: The Well-Grounded Rubyist (http://manning.com/black2\)

http://www.wishsight.com => Independent, social wishlist management!

7

Matt Harrison wrote:

This sounds great, but I've read in several places (sorry can't find references) that you should create a new migration to make changes to an existing table.

If you have not yet deployed a migration to your production site, you can go back into it and tweak it. This is still within the inner TDD cycle, because your tests will cover the new versions of the tables.

I've tried before to modify an existing migration but unless I totally wipe the DB, it doesn't take notice, as the schema version isn't incremented.

Right - rake db:reset, and rake db:migrate:reset.

Tweaking existing migrations before deploying them (such as adding each field in a new table, as you TDD it) is a convenience that prevents your migrate folder from filling up with many trivial variations.

···

--
   Phlip

8

David A. Black wrote:

I'd have to see the code to know exactly what's going on. Is it a
problem reconciling foreign key references?

It's a general, structural question, so I will ask it a different way.

This thread seems to propose one can borrow ActiveRecord, plus fixtures and migrations, and use them in a Ruby project that is (shocked gasp) not Rails.

If so, then doing this, in Rails, should be easy, too!

   class MyUnitTestSuite < ActiveSupport::TestCase
     fixtures :foos

     def setup
        fixtures :bars
     end
     ...
   end

For whatever reason, we want to load some fixtures the old-fashioned way, and some fixtures optionally. Maybe in the top of a test case, or maybe conditionally in a setup. Or whatever.

If we promise the OP they can borrow ActiveRecord facilities, then borrowing them should indeed be easy. But whenever I try to call or re-use or re-wrap the methods inside fixtures, they screw up in horrible ways. This _might_ be a sign of coupling!

···

--
   Phlip

9

Phlip wrote:

Matt Harrison wrote:

This sounds great, but I've read in several places (sorry can't find references) that you should create a new migration to make changes to an existing table.

If you have not yet deployed a migration to your production site, you can go back into it and tweak it. This is still within the inner TDD cycle, because your tests will cover the new versions of the tables.

I've tried before to modify an existing migration but unless I totally wipe the DB, it doesn't take notice, as the schema version isn't incremented.

Right - rake db:reset, and rake db:migrate:reset.

Tweaking existing migrations before deploying them (such as adding each field in a new table, as you TDD it) is a convenience that prevents your migrate folder from filling up with many trivial variations.

Well I'm just getting into TDD and thats going well, but I did notice that when running rake:test, that it complains there are migrations that need to be run before the tests will be performed, even though the tests are running on a different database.

Matt

10

Well I'm just getting into TDD and thats going well, but I did notice that when running rake:test, that it complains there are migrations that need to be run before the tests will be performed, even though the tests are running on a different database.

That's because the migrations (should!) happen on the Development database, and this gets cloned into the Test database just before each test run...

11

Phlip wrote:

Well I'm just getting into TDD and thats going well, but I did notice that when running rake:test, that it complains there are migrations that need to be run before the tests will be performed, even though the tests are running on a different database.

That's because the migrations (should!) happen on the Development database, and this gets cloned into the Test database just before each test run...

So on the development database, it's safe to add the columns or whatever manually, update the migration to reflect the changes and test/code as usual. Then when the production deployment happens it uses the migrations from scratch?

Thanks

Matt

12

Matt Harrison wrote:

So on the development database, it's safe to add the columns or whatever manually, update the migration to reflect the changes and test/code as usual. Then when the production deployment happens it uses the migrations from scratch?

Not safe to add columns "manually". Add them to your migrations, rake db:migrate:reset to rebuild your databases, manually test, deploy, and lock down those migrations.

The deploy script will migrate on the server, so all your existing data don't get erased!

13

Phlip wrote:

Matt Harrison wrote:

So on the development database, it's safe to add the columns or whatever manually, update the migration to reflect the changes and test/code as usual. Then when the production deployment happens it uses the migrations from scratch?

Not safe to add columns "manually". Add them to your migrations, rake db:migrate:reset to rebuild your databases, manually test, deploy, and lock down those migrations.

The deploy script will migrate on the server, so all your existing data don't get erased!

Ok, I'll have a go at that, thanks for clearing that up.

Matt

14

hi guys, sorry this is actually a railess app. Just ruby and the
console. Ive found a few links though that show how to use active record
without rails though here including setting up migrations and fixutres.

http://rails.aizatto.com/category/activerecord/activerecord-without-rails/

and there is also this piece here

http://wiki.rubyonrails.org/rails/pages/HowToUseActiveRecordOutsideRails

does anyone know of anyother links?

From what i can tell those links dont discuss how to implement the
automatic generation of a test version of a development db that rails
does everytime you run a test. If anyone has a link etc id really
appreciate it!.

Thanks for the input so far

···

--
Posted via http://www.ruby-forum.com/.

15

Hi --

···

On Mon, 12 Jan 2009, Adam Akhtar wrote:

hi guys, sorry this is actually a railess app. Just ruby and the
console. Ive found a few links though that show how to use active record
without rails though here including setting up migrations and fixutres.

http://rails.aizatto.com/category/activerecord/activerecord-without-rails/

and there is also this piece here

http://wiki.rubyonrails.org/rails/pages/HowToUseActiveRecordOutsideRails

does anyone know of anyother links?

From what i can tell those links dont discuss how to implement the

automatic generation of a test version of a development db that rails
does everytime you run a test. If anyone has a link etc id really
appreciate it!.

I don't have a link to a complete setup but what I'd recommend is have
a look at the Rake tasks that come with Rails, and adapt them. It's
non-trivial enough that I can't offer to do it on the spot, but
reasonably straightforward and will give you some Rake practice
anyway.

David

--
David A. Black / Ruby Power and Light, LLC
Ruby/Rails consulting & training: http://www.rubypal.com
Coming in 2009: The Well-Grounded Rubyist (http://manning.com/black2\)

http://www.wishsight.com => Independent, social wishlist management!