Finding duplicate records before creating using FasterCSV

Hello all,
I have a user that will be importing csv files occasionally. I have,
with a great amount of help (thanks James), set up an app. that will
read a csv and save to database.
I need to check for duplicate records before writing to database.

I'm thinking about putting a conditional to check for the duplicates via
column name.

- In controller -

Irb.transaction do
        FasterCSV.parse(file,
                        :headers => true,
                        :header_converters => lambda { |h| h.tr(" ",
"_").delete("^a-zA-Z0-9_")},
                        :converters => :all ) do |row|
                     # I'm thinking conditional looping thru the Model
here. #
                         Irb.create!( {"reconciled" => 0,
                                       "non_related" => 0
}.merge(row.to_hash))
                          rowcount += 1
                        end
      end

Thank you for any help.

JohnM

···

--
Posted via http://www.ruby-forum.com/.

Why not just add a validates_uniqueness_of() validation the the Irb model, so it will block the save of duplicate records?

James Edward Gray II

···

On Feb 4, 2010, at 1:15 PM, John Mcleod wrote:

I'm thinking about putting a conditional to check for the duplicates via
column name.

Why not just add a validates_uniqueness_of() validation the the Irb
model, so it will block the save of duplicate records?

I never thought of that.

I'll give it a try.

thanks again.

john

···

--
Posted via http://www.ruby-forum.com/\.

James Edward Gray II wrote:

I'm thinking about putting a conditional to check for the duplicates via
column name.

Why not just add a validates_uniqueness_of() validation the the Irb
model, so it will block the save of duplicate records?

And follow that up with a unique index in the DB. Validations are
insufficient by themselves.

James Edward Gray II

Best,

···

On Feb 4, 2010, at 1:15 PM, John Mcleod wrote:

--
Marnen Laibow-Koser
http://www.marnen.org
marnen@marnen.org
--
Posted via http://www.ruby-forum.com/\.

The validation works as it should but it stops the process of importing
the file. I would have to clean the csv of duplicate records before the
import.

john

···

--
Posted via http://www.ruby-forum.com/.

And follow that up with a unique index in the DB. Validations are
insufficient by themselves.

The index is very unique.
For example,

YY-XXXXabc

"YY" = year without century,
"XXXX" is an incremental integer and "abc" could be a 1,2 or 3 letter
combination.

John

···

--
Posted via http://www.ruby-forum.com/\.

Switch create!() to create(). All fixed. :wink:

James Edward Gray II

···

On Feb 4, 2010, at 1:36 PM, John Mcleod wrote:

The validation works as it should but it stops the process of importing
the file. I would have to clean the csv of duplicate records before the
import.

John Mcleod wrote:

And follow that up with a unique index in the DB. Validations are
insufficient by themselves.

The index is very unique.

Something either is or isn't unique. There's no "very". :slight_smile:

For example,

YY-XXXXabc

"YY" = year without century,
"XXXX" is an incremental integer and "abc" could be a 1,2 or 3 letter
combination.

I think you missed my point, or I missed a joke. "Unique index" is a
technical DB term -- look it up if you're not familiar with the concept.

John

Best,

···

--
Marnen Laibow-Koser
http://www.marnen.org
marnen@marnen.org
--
Posted via http://www.ruby-forum.com/\.

Great, that did it.

Thanks again, James.

John

···

--
Posted via http://www.ruby-forum.com/.

Not to move away from this discussion but what's the difference between
"create" and "create!" ?
I looked a little and couldn't find much.

John

···

--
Posted via http://www.ruby-forum.com/.

John Mcleod wrote:

Not to move away from this discussion but what's the difference between
"create" and "create!" ?
I looked a little and couldn't find much.

This is explained clearly in the AR docs.

John

Best,

···

--
Marnen Laibow-Koser
http://www.marnen.org
marnen@marnen.org
--
Posted via http://www.ruby-forum.com/\.

create() fails silently, returning the unsaved database object with errors if it could not be written to the database for some reason. create!() raises an error under the same circumstances.

James Edward Gray II

···

On Feb 4, 2010, at 2:08 PM, John Mcleod wrote:

Not to move away from this discussion but what's the difference between
"create" and "create!" ?