I'm working on a program that manipulates and stores a lot of data in a database via ActiveRecord. I'm having a heck of a time trying to write unit and functional tests for this program because I don't have a reasonable way to reload the test db. I'd like to use the fixtures facility from Rails but so far I've failed at ripping it out and moving it to a non-Rails environment.
(BTW, I'm using the MockFS gem to test the filesystem portions of the code and I have to say it is working *brilliantly*. Many thanks to the author!)
Anyone have experience with this type of work? Care to share any tips on getting this to work?
Thanks very much.
cr
Hey-
I had to do the same thing to test an activerecord extension plugin outside of rails. You can feel free to steal the test harness from my plugin:
http://opensvn.csie.org/ezra/rails/ez_where_two/test/test_helper.rb
Thats the test_helper that loads activerecord with a database.yml and then defines some models and loads the fixtures as well. Have a look around the plugin for the way I made this happen.
-Ezra
···
On May 30, 2006, at 2:33 PM, cremes.devlist@mac.com wrote:
I'm working on a program that manipulates and stores a lot of data in a database via ActiveRecord. I'm having a heck of a time trying to write unit and functional tests for this program because I don't have a reasonable way to reload the test db. I'd like to use the fixtures facility from Rails but so far I've failed at ripping it out and moving it to a non-Rails environment.
(BTW, I'm using the MockFS gem to test the filesystem portions of the code and I have to say it is working *brilliantly*. Many thanks to the author!)
Anyone have experience with this type of work? Care to share any tips on getting this to work?
Thanks very much.
cr
unknown wrote:
I'm working on a program that manipulates and stores a lot of data in
a database via ActiveRecord. I'm having a heck of a time trying to
Anyone have experience with this type of work? Care to share any tips
on getting this to work?
A bit away from your original topic (and perhaps this belongs on the
Rails mailing list), but: I'd be interested to hear if you run into any
problems using AR outside of Rails. I have been writing an application
(IRC game) which uses AR with PostgreSQL, and I have had uncountable
problems due to the way AR connects to the database. I am constantly
hitting the maximum connection limit, and stray PostgreSQL child
processes are always being left around.
What DB are you using?
Pistos
···
--
Posted via http://www.ruby-forum.com/\.
You're a life saver! Thank you!
cr
Chuck Remes
cremes@mac.com
www.familyvideovault.com (not yet live!)
···
On May 30, 2006, at 8:34 PM, Ezra Zygmuntowicz wrote:
On May 30, 2006, at 2:33 PM, cremes.devlist@mac.com wrote:
I'm working on a program that manipulates and stores a lot of data in a database via ActiveRecord. I'm having a heck of a time trying to write unit and functional tests for this program because I don't have a reasonable way to reload the test db. I'd like to use the fixtures facility from Rails but so far I've failed at ripping it out and moving it to a non-Rails environment.
(BTW, I'm using the MockFS gem to test the filesystem portions of the code and I have to say it is working *brilliantly*. Many thanks to the author!)
Anyone have experience with this type of work? Care to share any tips on getting this to work?
Thanks very much.
cr
Hey-
I had to do the same thing to test an activerecord extension plugin outside of rails. You can feel free to steal the test harness from my plugin:
http://opensvn.csie.org/ezra/rails/ez_where_two/test/test_helper.rb
Thats the test_helper that loads activerecord with a database.yml and then defines some models and loads the fixtures as well. Have a look around the plugin for the way I made this happen.
I'm using mysql 5.018 (latest AR, ruby 1.8.4 all on OSX 10.4.6) and so far I haven't run into any problems like you describe. But since you've mentioned them, I'll make sure my tests include detecting and handling those types of AR exceptions.
BTW, using Ezra's sample code I've already gotten the fixtures loading in my tests. Excellent stuff. I'll do a quick write-up after I've worked out all the kinks.
cr
···
On May 31, 2006, at 9:47 AM, Pistos Christou wrote:
unknown wrote:
I'm working on a program that manipulates and stores a lot of data in
a database via ActiveRecord. I'm having a heck of a time trying to
Anyone have experience with this type of work? Care to share any tips
on getting this to work?
A bit away from your original topic (and perhaps this belongs on the
Rails mailing list), but: I'd be interested to hear if you run into any
problems using AR outside of Rails. I have been writing an application
(IRC game) which uses AR with PostgreSQL, and I have had uncountable
problems due to the way AR connects to the database. I am constantly
hitting the maximum connection limit, and stray PostgreSQL child
processes are always being left around.
What DB are you using?
Pistos Christou wrote:
unknown wrote:
I'm working on a program that manipulates and stores a lot of data in
a database via ActiveRecord. I'm having a heck of a time trying to
Anyone have experience with this type of work? Care to share any tips
on getting this to work?
A bit away from your original topic (and perhaps this belongs on the
Rails mailing list), but: I'd be interested to hear if you run into any
problems using AR outside of Rails. I have been writing an application
(IRC game) which uses AR with PostgreSQL, and I have had uncountable
problems due to the way AR connects to the database. I am constantly
hitting the maximum connection limit, and stray PostgreSQL child
processes are always being left around.
What DB are you using?
Let me guess, your IRC game is multi-threaded? AR stores connections based on the Thread.current_thread.id (or at least it does in
1.13.2, things changed slightly in 1.14.2). I ran into this problem as well. If you are using AR 1.13.2 I can give you a fix,
otherwise you'll have to do it yourself or wait about a week for me to patch 1.14.2 (if it needs it).
Here is the code:
- -------------------
# Attempt to load active-record version, but if this dies, then don't complain
# Newer versions of rails seem to include VERSION by default, however activerecord-1.13.2 doesn't.
begin ; require 'active_record/version' ; rescue LoadError; end
# This includes changes to ActiveRecord 1.13.2
class ActiveRecord::Base
# This is overridden because ActiveRecord 1.13.2 stores database connections based on the
# Thread id. The processor uses a new thread for each new set of files to process.
# We want the same connection to be used.
···
#
# Version 1.14.2 no longer has the class variable @@connection_cache, so we should
# not redefine this method if we are at or above that version.
if ActiveRecord::VERSION::STRING < '1.14.2'
def self.connection
@@connection_cache[123456][name] ||= retrieve_connection
end
end
end
- ---------------------
How this helps,
Zach
zdennis wrote:
Let me guess, your IRC game is multi-threaded? AR stores connections
I think you're onto something there... 
based on the Thread.current_thread.id (or at least it does in
1.13.2, things changed slightly in 1.14.2). I ran into this problem as
well. If you are using AR 1.13.2 I can give you a fix,
otherwise you'll have to do it yourself or wait about a week for me to
patch 1.14.2 (if it needs it).
I happen to be specifically require_gem'ing 1.13.2 for the game, solely
because stuff after 1.13.2 caused me even worse problems...
Here is the code:
How this helps,
Zach
I put the code in place, and it seems to be helping, based on
preliminary testing. It's too early to tell for sure, though, so I will
keep you posted. Thanks for this patch. If it works, it will surely be
a great boon to the game.
Pistos
···
--
Posted via http://www.ruby-forum.com/\.
Pistos Christou wrote:
zdennis wrote:
Let me guess, your IRC game is multi-threaded? AR stores connections
I think you're onto something there...
based on the Thread.current_thread.id (or at least it does in
1.13.2, things changed slightly in 1.14.2). I ran into this problem as
well. If you are using AR 1.13.2 I can give you a fix,
otherwise you'll have to do it yourself or wait about a week for me to
patch 1.14.2 (if it needs it).
I happen to be specifically require_gem'ing 1.13.2 for the game, solely
because stuff after 1.13.2 caused me even worse problems...
Here is the code:
How this helps,
Zach
I put the code in place, and it seems to be helping, based on
preliminary testing. It's too early to tell for sure, though, so I will
keep you posted. Thanks for this patch. If it works, it will surely be
a great boon to the game.
Pistos,
You can actually test this in a unit test. =)
def test_connection_count
2.times do
Thread.new{ Job.find_all }
end
assert_equal 1, ActiveRecord::Base.module_eval( "@@connection_cache" ).size, "Too many connections are being created!"
end
There is also a method on ActiveRecord::Base called 'active_connections' and it seems to always return 1, however if you actually
do the module_eval for @@connection_cache, you will see there are more connections (that are potentially active) then
'active_connections' returns.
There is another method on ActiveRecord::Base called 'threaded_connections' which returns true or false (by default true). I was
hoping that if you set it to false, it would fix the problem, but setting it to false doesn't seem to do anything for this
particular problem.
I am gone for the weekend, if you find anything else out please post so I can read when I get back, otherwise I'll dive in over
the weekend to AR and find out what I can (and to see if AR 1.14.2 has this problem)
Have a good weekend Pistos!
Zach
zdennis wrote:
You can actually test this in a unit test. =)
There is also a method on ActiveRecord::Base called 'active_connections'
There is another method on ActiveRecord::Base called
'threaded_connections' which returns true or false (by default true). I
[much snippage above]
Well, unit test or not, the evidence is here:
# ps aux | grep postgres | grep idle | grep -v grep | wc -l
22
template1=# select count(*) from pg_stat_activity where current_query =
'<IDLE>' and query_start < NOW() - '2 hours'::INTERVAL;
count
···
-------
18
The good news is, some of the processes DO seem to go away on their own.
The bad news is, some stay around for some time:
template1=# select count(*) from pg_stat_activity where current_query =
'<IDLE>' and query_start < NOW() - '20 hours'::INTERVAL;
count
-------
2
As well, I haven't yet actually gotten the "too many connections" error
from PostgreSQL, so that is somewhat encouraging.
Furthermore, we should bear in mind that some of these connections are
from the companion Rails site that works with the game (shared DB). I'm
not sure how to distinguish Rails connections from game connections.
I'm using FastCGI, and, as far as I know, that keeps processes alive.
If you ever find time, maybe we can work together on the problem via
IRC? I'm on FreeNode often with the nick Pistos.
Pistos
--
Posted via http://www.ruby-forum.com/\.