FasterCSV.foreach loop

ruby-talk
1

Hello community,

This is the way my CSV file looks like:

id,name,valid_time,date
1,baiki,5:00,9/10/2009
1,baiki,1:45,9/14/2009
2,lee,6:00,9/8/2009
3,fatou,9:15,8/14/2008
3,fatou,10:15,8/13/2008
2,lee,6:00,9/7/2009
2,lee,6:55,9/6/2009
2,lee,6:00,9/8/2009
1,baiki,5:00,9/11/2009
1,baiki,4:00,9/12/2009
1,baiki,2:00,9/13/2009
3,fatou,12:15,8/13/2008
3,fatou,12:15,8/18/2008

...and this is my Ruby, FasterCSV (FCSV) approach to find unique id's
out of my CSV file and afterwards read all records from those employees,
do some calculations and then generate a ODF file (using
sandros-odf-report) per employee.

But I ran into a problem - ok, I admit it: SEVERAL PROBLEMS :-).

#!/usr/bin/ruby

require 'rubygems'
require 'odf-report'
require 'fastercsv'

version = '0.01'
puts "\nGorilla CSV Convert Version
#{version}\n================================"
if ARGV[0].to_s.empty?
  puts "ERROR: No file name given. Please call me this way:\nExample: $
ruby do_payment_slips.rb july_2009.csv\n\n"
  exit
else
  puts "File to load: #{ARGV[0]}\n\n"
  input_file = ARGV[0]
end

employee_id = []
FasterCSV.foreach(input_file, :headers => true) do |row|
  employee_id.concat(row[0].to_a).uniq!
end

until employee_id.empty?
  FasterCSV.foreach(input_file, :headers => true) do |row|
    if row[0] == employee_id[0]
      puts "value: #{row}"
      next
    end
    employee_id.shift
  end
end

Inside the if-loop, I try to print the current value of row, so, it
should be:
5 records of id 1 and
4 record of id 2 and
4 records of id 3

But I get just the records of id 1, 1, 3.

What am I missing?

Thanks for any help welcome :slight_smile:

Baiki

···

--
Posted via http://www.ruby-forum.com/.

2

Hello community,

Hello.

This is the way my CSV file looks like:

id,name,valid_time,date
1,baiki,5:00,9/10/2009
1,baiki,1:45,9/14/2009
2,lee,6:00,9/8/2009
3,fatou,9:15,8/14/2008
3,fatou,10:15,8/13/2008
2,lee,6:00,9/7/2009
2,lee,6:55,9/6/2009
2,lee,6:00,9/8/2009
1,baiki,5:00,9/11/2009
1,baiki,4:00,9/12/2009
1,baiki,2:00,9/13/2009
3,fatou,12:15,8/13/2008
3,fatou,12:15,8/18/2008

Is this file large? It looks like you may eventually just want all of the data in it. If that's the case, and it's not too big to suck the whole thing into memory, it may be easier to work with that way. For example:

   data = FCSV.table(input_file)
   p data[:id].uniq # these are the uniq ID's

But I ran into a problem - ok, I admit it: SEVERAL PROBLEMS :-).

until employee_id.empty?
FasterCSV.foreach(input_file, :headers => true) do |row|
   if row[0] == employee_id[0]
     puts "value: #{row}"
     next
   end
   employee_id.shift
end
end

But I get just the records of id 1, 1, 3.

What am I missing?

You remove an ID from employee_id after reading each line of your CSV data. I think you meant to do something more like this:

employee_id.each do |id|
   FCSV.foreach(input_file, :headers => true) do |row|
     puts "value: #{row}" if row["id"] == id
   end
end

Hope that helps.

James Edward Gray II

···

On Sep 13, 2009, at 2:28 PM, Dot Baiki wrote:

3

Dear Mr. James Edward Gray II

I am sorry for late reply. I just wanted to say: it did not work out,
but because of the odf-report part.

Thanks for all the help!

Regards

···

--
Posted via http://www.ruby-forum.com/.

4

Hello again,

I know it's not your problem/software but maybe you see the mistake I
made here:

  report = ODFReport::Report.new('gorilla.odt') do |r|
    r.add_field :employees_name, employees_name
    r.add_field :employees_id, employees_records[0][:id]
    r.add_field :employees_month_of_payment, employees_month_of_payment
    i = -1
    r.add_table('table_time_details', employees_records) do |t|
      i += 1
      t.add_column(:employees_date, employees_records[i][:date])
    end

    r.add_field :employees_monthly_total, employees_monthly_total
    r.add_field :software_name, software_name
    r.add_field :version, version
  end

Seems like this line is the problem: t.add_column(:employees_date,
employees_records[i][:date])

However, thanks anyway :slight_smile:

Baiki

···

--
Posted via http://www.ruby-forum.com/.

5

James Edward Gray II wrote:

Hello.

Hi there again!

How are things going over there? Hope fine :slight_smile:

Is this file large? It looks like you may eventually just want all of
the data in it. If that's the case, and it's not too big to suck the
whole thing into memory, it may be easier to work with that way. For
example:

Well, I expect that this CSV file contain about 12-15 columns and aprox.
36k rows. I followed your suggestion.

You remove an ID from employee_id after reading each line of your CSV
data. I think you meant to do something more like this:

employee_id.each do |id|
   FCSV.foreach(input_file, :headers => true) do |row|
     puts "value: #{row}" if row["id"] == id
   end
end

Amazing. Lovely solution. Sweet :slight_smile: The only thing I had to do was to
add ".to_s"...here:

     puts "value: #{row}" if row["id"] == id.to_s

Hope that helps.

Indeed it did. Big thank you! Now I have to sort them and send them
trough odf-report and soon soon I will see something working. We'll keep
in touch.

James Edward Gray II

dot baiki

···

On Sep 13, 2009, at 2:28 PM, Dot Baiki wrote:

--
Posted via http://www.ruby-forum.com/\.

6

Sorry, I don't know what ODFReport is.

James Edward Gray II

···

On Jan 10, 2011, at 8:17 PM, Dot Baiki wrote:

Hello again,

I know it's not your problem/software but maybe you see the mistake I
made here:

report = ODFReport::Report.new('gorilla.odt') do |r|
   r.add_field :employees_name, employees_name
   r.add_field :employees_id, employees_records[0][:id]
   r.add_field :employees_month_of_payment, employees_month_of_payment
   i = -1
   r.add_table('table_time_details', employees_records) do |t|
     i += 1
     t.add_column(:employees_date, employees_records[i][:date])
   end

   r.add_field :employees_monthly_total, employees_monthly_total
   r.add_field :software_name, software_name
   r.add_field :version, version
end

Seems like this line is the problem: t.add_column(:employees_date,
employees_records[i][:date])

7

Good day

Thanks to Mr. James Edward Gray II, I understand more and more :slight_smile:

After trying to create an array of arrays, I got stuck again. It seems I
can't properly create such an array. This is my mistake:

data = FCSV.table(input_file)
employee_id = data[:id].uniq

employee_id.each do |id|
  employee_record = []
  FCSV.foreach(input_file, :headers => true) do |row|
    if row["id"].to_i == id
      employee_record << [row]
# create_odf_file(employee_record)
    end
  end
  break #just for testing once
end

I saw that FasterCSV.foreach creates an array/hash/something?
#<FasterCSV::Table mode:col_or_row row_count:14>

My array looks like this, but I can't access certain fields:
[[#<FasterCSV::Row "id":"1" "name":"baiki" "valid_time":"5:00"
"date":"9/10/2009">], [#<FasterCSV::Row "id":"1" "name":"baiki"
"valid_time":"1:45" "date":"9/14/2009">], [#<FasterCSV::Row "id":"1"
"name":"baiki" "valid_time":"5:00" "date":"9/11/2009">],
[#<FasterCSV::Row "id":"1" "name":"baiki" "valid_time":"4:00"
"date":"9/12/2009">], [#<FasterCSV::Row "id":"1" "name":"baiki"
"valid_time":"2:00" "date":"9/13/2009">]]

If I want an array which contains all records from one employee, should
I use FasterCSV's filter class method?

Or, how else can I pass this information to a class (ie. odf-report) in
order to create the ODF output?

I'll keep trying. BTW: If your once in our country, let me know :slight_smile:

Greetings
dot baiki

···

--
Posted via http://www.ruby-forum.com/.

8

Good day

Thanks to Mr. James Edward Gray II, I understand more and more :slight_smile:

I'm glad to hear it.

After trying to create an array of arrays, I got stuck again. It seems I
can't properly create such an array. This is my mistake:

data = FCSV.table(input_file)

When you execute the above line you read all of the data into memory. You would not need to do any more reading, like making calls to FCSV.foreach().

I saw that FasterCSV.foreach creates an array/hash/something?
#<FasterCSV::Table mode:col_or_row row_count:14>

The data variable above holds a FasterCSV::Table which is a container for some FasterCSV::Row objects. They represent your data.

If I want an array which contains all records from one employee, should
I use FasterCSV's filter class method?

You could grab the records for one employee using code like:

   employee1s_records = data.select { |row| row["id"] == 1 }

Hope that helps.

James Edward Gray II

···

On Sep 16, 2009, at 12:11 PM, Dot Baiki wrote:

9

Hello again

Thanks again for your advice. This is what I have now:

csv_data = FCSV.table(input_file)
employees_id = csv_data[:id].uniq

employees_id.each do |id|
  puts "ID: #{id}"
  employees_records = csv_data.select { |row| row["id"] == id }
  puts employees_records
  break # just for now
end

The thing is, no output. Empty. Right now I really don't get it. I am
using old software? That's what I use:
- ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]
- fastercsv (1.5.0, 1.2.3) ** maybe I should deinstall one??? **

This one doesn't give me any output as well.

csv_data = FCSV.table(input_file)
employees_id = csv_data[:id].uniq
employees_records = csv_data.select { |row| row["id"] == 1 }
p employees_records

Hmmm. Hope you can give me another clue!

Regards,
Baiki

···

--
Posted via http://www.ruby-forum.com/.

10

Sorry, I messed up. My fault.

Try replacing:

   row["id"]

with:

   row[:id]

I bet that fixes you up.

James Edward Gray II

···

On Sep 16, 2009, at 1:53 PM, Dot Baiki wrote:

This is what I have now:

csv_data = FCSV.table(input_file)
employees_id = csv_data[:id].uniq

employees_id.each do |id|
puts "ID: #{id}"
employees_records = csv_data.select { |row| row["id"] == id }
puts employees_records
break # just for now
end

The thing is, no output. Empty.

11

Oh, I see:

csv_data = FCSV.table(input_file)
employees_id = csv_data[:id].uniq

employees_id.each do |id|
  puts "ID: #{id}"

  employees_records = csv_data.select { |row| row[:id] == id }

  puts employees_records
  break # just for now
end

:id did the trick. Will try more.

Thanks
Baiki

···

--
Posted via http://www.ruby-forum.com/\.

12

I guess I need your help once again.

I need "employees_records" in another class (want to create ODF files
per employee). Should I try to convert this variable into a hash or
should I try to access certain fields within "employees_records" using
one of your techniques? How can I access one cell within
"employees_records" or how can I change it to a proper hash (I really do
prefer arrays)

Thank you.
Baiki

···

--
Posted via http://www.ruby-forum.com/.

13

How's this?

   employees_records = csv_data.select { |row| row[:id] == 1 }.
                                map { |row| row.to_hash }

James Edward Gray II

···

On Sep 16, 2009, at 2:31 PM, Dot Baiki wrote:

I guess I need your help once again.

I need "employees_records" in another class (want to create ODF files
per employee). Should I try to convert this variable into a hash or
should I try to access certain fields within "employees_records" using
one of your techniques? How can I access one cell within
"employees_records" or how can I change it to a proper hash (I really do
prefer arrays)

14

James Edward Gray II wrote:

How's this?

   employees_records = csv_data.select { |row| row[:id] == 1 }.
                                map { |row| row.to_hash }

Good evening,

This was just brilliant! Finally some progress (=hope) :slight_smile: Will be back
tomorrow.

Thanks so much.
Baiki

···

--
Posted via http://www.ruby-forum.com/\.

15

Good day again

I came across this article
(http://groups.google.com/group/ruby-reports/browse_thread/thread/7cba5390d1aa3798)
where was written that sorting tables or rows should not be possible? Is
that a fact? Can't imagine. How could I sort such a table. Is this a
Ruby function or from FCSV?

Regards,
Baiki

···

--
Posted via http://www.ruby-forum.com/.

16

That article just says that FasterCSV doesn't provide a method to sort a table and reorder it in memory. You can always create a second table and reorder elements as you move them over, reorder elements as you write the results out to disk, etc.

James Edward Gray II

···

On Sep 17, 2009, at 4:40 AM, Dot Baiki wrote:

I came across this article
(http://groups.google.com/group/ruby-reports/browse_thread/thread/7cba5390d1aa3798\)
where was written that sorting tables or rows should not be possible? Is
that a fact? Can't imagine. How could I sort such a table. Is this a
Ruby function or from FCSV?