Marshal.dump() taking a long time

ruby-talk
1

I have a program that parses a text file into a list of objects, then
tries to write it out using Marshal.dump(). But, it’s taking a
ridiculous amount of time.

This is how I’m using Marshal.dump:

STDERR.puts "Successfully parsed #{line_num} entries. Saving…"
print Marshal.dump(dict)
STDERR.puts “Saved.”

And this is what happens:

$ ./edict.rb < edict > edict.dat
Parsing dictionary entries…
Successfully parsed 103527 entries. Saving…
(it just sits here forever)

The text file itself consists of 103527 lines, and it took less than
10 seconds before it said “Successfully parsed”. But I waited several
minutes, and the Marshal.dump() still didn’t complete.

The object being dumped is a list of 103527 objects, each containing
two strings, and a short list of strings. This object is generated
from a 5 MB data file.

It runs fine if I feed it only a 1000-line sample of the actual data
file.

Does anyone have suggestions on how to make Marshal.dump() work
faster? I wanted to be able to load the parsed representation of the
file quickly whenever the program starts, rather than having to parse
a text file every time…

2

Hi, Philip,

From: Philip Mak [mailto:pmak@aaanime.net]
Sent: Wednesday, March 12, 2003 9:28 PM

STDERR.puts “Successfully parsed #{line_num} entries. Saving…”
print Marshal.dump(dict)

Try
anIO = STDOUT
Marshal.dump(dict, anIO)

STDERR.puts “Saved.”

String concatination while serialization might be the bottleneck.

Regards,
// NaHi

3

Thanks for the suggestion! I changed:

print Marshal.dump(dict)

to:

Marshal.dump(dict, STDOUT)

and it runs fine now.

Interesting thing I noticed before when I was doing it the slow way -
1000 entries took 1 second, but 2000 entries took 7.7 seconds. So the
time required seemed to scale up exponentially.

4

Philip Mak pmak@aaanime.net writes:

Thanks for the suggestion! I changed:

print Marshal.dump(dict)

to:

Marshal.dump(dict, STDOUT)

and it runs fine now.

Interesting thing I noticed before when I was doing it the slow way -
1000 entries took 1 second, but 2000 entries took 7.7 seconds. So the
time required seemed to scale up exponentially.

What version of ruby are you using? String concatenation is much
faster under ruby 1.8.

···

–
matt