# FasterCSV heavy loads?
**URL:** https://rubytalk.org/t/fastercsv-heavy-loads/45595
**Category:** ruby-talk
**Created:** [2 April 2008 08:08 UTC](https://rubytalk.org/t/fastercsv-heavy-loads/45595 "2008-04-02T08:08:47Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Jon\_Hawkins](https://rubytalk.org/u/Jon_Hawkins)
#### Post date: [2 April 2008 08:08 UTC](https://rubytalk.org/t/fastercsv-heavy-loads/45595/1 "2008-04-02T08:08:47Z")
Recently I've attempted to push a huge csv into arrays via code that
looks along the lines of this:
csvFile = FasterCSV.read('data.csv', :headers =\> true)
array = []
csvFile.each do |row|
array \<\< row['column\_name']
end
The problem arises when the csv file is someodd 2 million lines or more.
Normally I would comment about how long it took but I decided to call it
quits after 9 hours of waiting lol. Any ideas on how to handle columns
in CSV docs the same way FasterCSV does?
(And yes, theoretically I could split the 80mb csv into 20 4mb files but
whats the accomplishment in that!)
Thanks,
- Mac
> **···**
>
> --
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).
---
### Author:  [@Mike\_Woodhouse](https://rubytalk.org/u/Mike_Woodhouse)
#### Post date: [2 April 2008 12:30 UTC](https://rubytalk.org/t/fastercsv-heavy-loads/45595/2 "2008-04-02T12:30:08Z")
How many fields in a row? You're appending that many times (2 million
or more) values to an array, which I suspect is where your performance
problem lies.
You could probably check by
csvFile = FasterCSV.read('data.csv', :headers =\> true)
count = 0
csvFile.each do |row|
end
> **···**
>
> On Apr 2, 9:08 am, Michael Linfield \ wrote:
>
> > Recently I've attempted to push a huge csv into arrays via code that
> > looks along the lines of this:
> >
> > csvFile = FasterCSV.read('data.csv', :headers =\> true)
> >
> > array =
> >
> > csvFile.each do |row|
> > array \<\< row['column\_name']
> > end
> >
> > The problem arises when the csv file is someodd 2 million lines or more.
---
### Author:  [@James\_Edward\_Gray\_II](https://rubytalk.org/u/James_Edward_Gray_II)
#### Post date: [2 April 2008 13:16 UTC](https://rubytalk.org/t/fastercsv-heavy-loads/45595/3 "2008-04-02T13:16:05Z")
> Recently I've attempted to push a huge csv into arrays via code that
> looks along the lines of this:
>
> csvFile = FasterCSV.read('data.csv', :headers =\> true)
>
> array =
>
> csvFile.each do |row|
> array \<\< row['column\_name']
> end
That code is pretty inefficient, since it reads the entire file into memory only to walk over it row by row. Let's just read it row by row, instead.
column =
FCSV.foreach('data.csv', :headers =\> true) do |row|
column \<\< row['column\_name']
end
> The problem arises when the csv file is someodd 2 million lines or more.
> Normally I would comment about how long it took but I decided to call it
> quits after 9 hours of waiting lol.
FasterCSV could be choking on the CSV data, if it's not valid. It sometimes has to read to the end of the document to know that, which could take a long while with that much data.
James Edward Gray II
> **···**
>
> On Apr 2, 2008, at 3:08 AM, Michael Linfield wrote:
---
### Author:  [@Mike\_Woodhouse](https://rubytalk.org/u/Mike_Woodhouse)
#### Post date: [2 April 2008 12:30 UTC](https://rubytalk.org/t/fastercsv-heavy-loads/45595/4 "2008-04-02T12:30:28Z")
Hmph. I must have hit some unknown "send" key combination...
I meant to say, before I interrupted myself:
csvFile = FasterCSV.read('data.csv', :headers =\> true)
count = 0
csvFile.each do |row|
count += 1
end
...which replaces the array append with a lightweight operation. (I
don't know if Ruby is "smart" and likely to skip the iteration with an
empty block - probably not, but adding 1 shouldn't impose a heavy
load)
Mike
> **···**
>
> On Apr 2, 1:26 pm, Mike Woodhouse \ wrote:
>
> > On Apr 2, 9:08 am, Michael Linfield \ wrote:
> >
> > \> Recently I've attempted to push a huge csv into arrays via code that
> > \> looks along the lines of this:
> >
> > \> csvFile = FasterCSV.read('data.csv', :headers =\> true)
> >
> > \> array =
> >
> > \> csvFile.each do |row|
> > \> array \<\< row['column\_name']
> > \> end
> >
> > \> The problem arises when the csv file is someodd 2 million lines or more.
> >
> > How many fields in a row? You're appending that many times (2 million
> > or more) values to an array, which I suspect is where your performance
> > problem lies.
> >
> > You could probably check by
> >
> > csvFile = FasterCSV.read('data.csv', :headers =\> true)
> > count = 0
> > csvFile.each do |row|
> >
> > end
---
### Author:  [@Jon\_Hawkins](https://rubytalk.org/u/Jon_Hawkins)
#### Post date: [2 April 2008 17:54 UTC](https://rubytalk.org/t/fastercsv-heavy-loads/45595/5 "2008-04-02T17:54:27Z")
Mike Woodhouse wrote:
> > \> array \<\< row['column\_name']
> > csvFile = FasterCSV.read('data.csv', :headers =\> true)
> > count = 0
> > csvFile.each do |row|
> >
> > end
>
> Hmph. I must have hit some unknown "send" key combination...
>
> I meant to say, before I interrupted myself:
>
> csvFile = FasterCSV.read('data.csv', :headers =\> true)
> count = 0
> csvFile.each do |row|
> count += 1
> end
>
> ...which replaces the array append with a lightweight operation. (I
> don't know if Ruby is "smart" and likely to skip the iteration with an
> empty block - probably not, but adding 1 shouldn't impose a heavy
> load)
>
> Mike
> James
> Let's just read it row by
> row, instead.
>
> column =
> FCSV.foreach('data.csv', :headers =\> true) do |row|
> column \<\< row['column\_name']
> end
Well firstly, I did count the rows already in the csv via
file = File.readlines('data.csv')
file.length
output is about ~2,500,300
To answer your other question Mike the amount of columns is 3. All
integers.
Thanks James for that snippet, though it might be more efficient it
likely cuts the time in half (really nice) however, being that after 9
hours I gave up, I don't know what half even is! The data from each
column is being written into its own array. IE: column1Array =
column2Array = ect.
Usually if the numbers were a defined length in each column I could just
use regexp's to pull them out, however the numbers are mostly random.
I'll give your snippet a shot James and let you know how the results
turn out. Till then any additional thoughts are much appreciated.
Thanks,
Mac
> **···**
>
> > On Apr 2, 1:26 pm, Mike Woodhouse \ wrote:
>
> --
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).