I'm running top and when I run my app - I see the user CPU utilitization hit ~70%, but the ruby process only shows about 3%. Is this observation indicative of inefficient code? And to be complete, the user CPU utilization drops to 0% when I stop the app - so it's me.
Thanks Ruby gurus!
brian
Brian Le Roy wrote:
I'm running top and when I run my app - I see the user CPU
utilitization hit ~70%, but the ruby process only shows about 3%. Is
this observation indicative of inefficient code?
Hm, might be that the rest of the 70% are spent in the kernel (doing IO
probably). Do you do a lot IO? Or do you start another process (or
multiple processes) from your Ruby script? Did you have an eye on your
disk LED? I'm not too familiar with top but I think I remember that it
sometimes spits out odd values - might be another explanation.
And to be complete,
the user CPU utilization drops to 0% when I stop the app - so it's me.
Guilty!!! 
Kind regards
robert
I am doing some database queries using the MySQL-Ruby library, not DBI. I disabled logging, so it's not IO. Are there any differences between MySQL-Ruby and DBI?
Robert Klemme wrote:
Brian Le Roy wrote:
I'm running top and when I run my app - I see the user CPU
utilitization hit ~70%, but the ruby process only shows about 3%. Is
this observation indicative of inefficient code?Hm, might be that the rest of the 70% are spent in the kernel (doing IO
probably). Do you do a lot IO? Or do you start another process (or
multiple processes) from your Ruby script? Did you have an eye on your
disk LED? I'm not too familiar with top but I think I remember that it
sometimes spits out odd values - might be another explanation.And to be complete,
the user CPU utilization drops to 0% when I stop the app - so it's me.Guilty!!!
Kind regards
robert
Well, that's it.
MySql can take even 100% of your processing power. It happenes when you
have complex queries to run, and you run them on big tables. Are you
running complex queries ?
no, but the table has 140K records. the disk isn't crunching, so the index should be okay. i was thinking it would be passing around the database connection object causing overhead. i took the result set object and created a straight ol' array for passing. no improvement - so maybe there's nothing to do. thanks for all the responses - i'll keep messing with it and if i find something that improves it, i'll let you know!
bonefry wrote:
Well, that's it.
MySql can take even 100% of your processing power. It happenes when you
have complex queries to run, and you run them on big tables. Are you
running complex queries ?
Brian Le Roy wrote:
no, but the table has 140K records. the disk isn't crunching, so the index should be okay. i was thinking it would be passing around the database connection object causing overhead. i took the result set object and created a straight ol' array for passing. no improvement - so maybe there's nothing to do. thanks for all the responses - i'll keep messing with it and if i find something that improves it, i'll let you know!
bonefry wrote:
Well, that's it.
MySql can take even 100% of your processing power. It happenes when you
have complex queries to run, and you run them on big tables. Are you
running complex queries ?
It may may no difference, but you could try the same with postgresql, or maybe even sqlite, to see if it is mysql thats the issue.
when you have 140K records indexing all fields you are testing is
mandatory. are you doing that ?
Before he does that I'd try a different query frontend to see whether performance characteristics stay the same (i.e. time is spent in the DB, which would be my guess). The overhead is much smaller compared to installing another DBMS, transferring DDL and data to that.
Kind regards
robert
Reid Thompson <reid.thompson@ateb.com> wrote:
Brian Le Roy wrote:
no, but the table has 140K records. the disk isn't crunching, so the
index should be okay. i was thinking it would be passing around the
database connection object causing overhead. i took the result set
object and created a straight ol' array for passing. no improvement
- so maybe there's nothing to do. thanks for all the responses -
i'll keep messing with it and if i find something that improves it,
i'll let you know!bonefry wrote:
Well, that's it.
MySql can take even 100% of your processing power. It happenes when
you have complex queries to run, and you run them on big tables.
Are you running complex queries ?It may may no difference, but you could try the same with postgresql,
or maybe even sqlite, to see if it is mysql thats the issue.
Actually guys, I took out the queries - except for one that is run once. Same behaviour in top. I'm going to start marking out the app with the benchmark module and hopefully that will shed some light on it!
Robert Klemme wrote:
Reid Thompson <reid.thompson@ateb.com> wrote:
Brian Le Roy wrote:
no, but the table has 140K records. the disk isn't crunching, so the
index should be okay. i was thinking it would be passing around the
database connection object causing overhead. i took the result set
object and created a straight ol' array for passing. no improvement
- so maybe there's nothing to do. thanks for all the responses -
i'll keep messing with it and if i find something that improves it,
i'll let you know!bonefry wrote:
Well, that's it.
MySql can take even 100% of your processing power. It happenes when
you have complex queries to run, and you run them on big tables.
Are you running complex queries ?It may may no difference, but you could try the same with postgresql,
or maybe even sqlite, to see if it is mysql thats the issue.Before he does that I'd try a different query frontend to see whether performance characteristics stay the same (i.e. time is spent in the DB, which would be my guess). The overhead is much smaller compared to installing another DBMS, transferring DDL and data to that.
Kind regards
robert
Brian Le Roy wrote:
Actually guys, I took out the queries - except for one that is run
once. Same behaviour in top. I'm going to start marking out the
app with
the benchmark module and hopefully that will shed some light on it!
Note also that there is ruby -r profile. That way you don't have to
manually change your code.
robert
Robert Klemme wrote:
Reid Thompson <reid.thompson@ateb.com> wrote:
Brian Le Roy wrote:
no, but the table has 140K records. the disk isn't crunching, so
the index should be okay. i was thinking it would be passing
around the database connection object causing overhead. i took
the result set object and created a straight ol' array for
passing. no improvement - so maybe there's nothing to do. thanks
for all the responses - i'll keep messing with it and if i find
something that improves it, i'll let you know!bonefry wrote:
Well, that's it.
MySql can take even 100% of your processing power. It happenes
when you have complex queries to run, and you run them on big
tables. Are you running complex queries ?It may may no difference, but you could try the same with
postgresql, or maybe even sqlite, to see if it is mysql thats the
issue.Before he does that I'd try a different query frontend to see whether
performance characteristics stay the same (i.e. time is spent in the
DB, which would be my guess). The overhead is much smaller compared
to installing another DBMS, transferring DDL and data to that.Kind regards
robert
Thanks Robert!
I think I found the culprit.
With 20 records in the result:
total ms/call
12300.00 Mysql::Result#each
With 1 record in the result:
total ms/call
600.00 Mysql::Result#each
The query behind this result set is indexed properly. Not sure why this takes so long - shouldn't it be as fast as a regular array.each call?
Robert Klemme wrote:
Brian Le Roy wrote:
Actually guys, I took out the queries - except for one that is run
once. Same behaviour in top. I'm going to start marking out the
app with
the benchmark module and hopefully that will shed some light on it!Note also that there is ruby -r profile. That way you don't have to
manually change your code.robert
Robert Klemme wrote:
Reid Thompson <reid.thompson@ateb.com> wrote:
Brian Le Roy wrote:
no, but the table has 140K records. the disk isn't crunching, so
the index should be okay. i was thinking it would be passing
around the database connection object causing overhead. i took
the result set object and created a straight ol' array for
passing. no improvement - so maybe there's nothing to do. thanks
for all the responses - i'll keep messing with it and if i find
something that improves it, i'll let you know!bonefry wrote:
Well, that's it.
MySql can take even 100% of your processing power. It happenes
when you have complex queries to run, and you run them on big
tables. Are you running complex queries ?It may may no difference, but you could try the same with
postgresql, or maybe even sqlite, to see if it is mysql thats the
issue.Before he does that I'd try a different query frontend to see whether
performance characteristics stay the same (i.e. time is spent in the
DB, which would be my guess). The overhead is much smaller compared
to installing another DBMS, transferring DDL and data to that.Kind regards
robert
That's odd that it would take so long with only 20 records in the
result set. With 140k in the result set, that would be expected, I
suppose. But not 20.
Are you using the C mysql bindings, or the pure Ruby bindings?
On 9/9/05, Brian Le Roy <brian@le-roy.org> wrote:
Thanks Robert!
I think I found the culprit.
With 20 records in the result:
total ms/call
12300.00 Mysql::Result#eachWith 1 record in the result:
total ms/call
600.00 Mysql::Result#eachThe query behind this result set is indexed properly. Not sure why this
takes so long - shouldn't it be as fast as a regular array.each call?Robert Klemme wrote:
> Brian Le Roy wrote:
>
>>Actually guys, I took out the queries - except for one that is run
>> once. Same behaviour in top. I'm going to start marking out the
>>app with
>>the benchmark module and hopefully that will shed some light on it!
>
>
> Note also that there is ruby -r profile. That way you don't have to
> manually change your code.
>
> robert
>
>
>>Robert Klemme wrote:
>>
>>>Reid Thompson <reid.thompson@ateb.com> wrote:
>>>
>>>
>>>>Brian Le Roy wrote:
>>>>
>>>>
>>>>>no, but the table has 140K records. the disk isn't crunching, so
>>>>>the index should be okay. i was thinking it would be passing
>>>>>around the database connection object causing overhead. i took
>>>>>the result set object and created a straight ol' array for
>>>>>passing. no improvement - so maybe there's nothing to do. thanks
>>>>>for all the responses - i'll keep messing with it and if i find
>>>>>something that improves it, i'll let you know!
>>>>>
>>>>>bonefry wrote:
>>>>>
>>>>>
>>>>>>Well, that's it.
>>>>>>
>>>>>>MySql can take even 100% of your processing power. It happenes
>>>>>>when you have complex queries to run, and you run them on big
>>>>>>tables. Are you running complex queries ?
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>It may may no difference, but you could try the same with
>>>>postgresql, or maybe even sqlite, to see if it is mysql thats the
>>>>issue.
>>>
>>>
>>>Before he does that I'd try a different query frontend to see whether
>>>performance characteristics stay the same (i.e. time is spent in the
>>>DB, which would be my guess). The overhead is much smaller compared
>>>to installing another DBMS, transferring DDL and data to that.
>>>
>>>Kind regards
>>>
>>> robert
>
>
>
>
Brian Le Roy wrote:
Thanks Robert!
I think I found the culprit.
With 20 records in the result:
total ms/call
12300.00 Mysql::Result#eachWith 1 record in the result:
total ms/call
600.00 Mysql::Result#eachThe query behind this result set is indexed properly. Not sure why
this takes so long - shouldn't it be as fast as a regular array.each
call?
Unlikely, as the data has to be fetched over the network (even if it's
local). Even with prefetching etc. this is slower than iterating an array
in mem. Also, the accumulated time of each depends on the complexity of
the operation in the block - that's sometimes overlooked. Note that #each
is called only once per iteration but the block is invoked multiple times.
Kind regards
robert
I don't think it's about the number in the query set. I think it's
about the actuall query. Can you please post it in here ?
I compiled the C bindings. I'll try building arrays and pass those. Thanks for all the help!
Robert Klemme wrote:
Brian Le Roy wrote:
Thanks Robert!
I think I found the culprit.
With 20 records in the result:
total ms/call
12300.00 Mysql::Result#eachWith 1 record in the result:
total ms/call
600.00 Mysql::Result#eachThe query behind this result set is indexed properly. Not sure why
this takes so long - shouldn't it be as fast as a regular array.each
call?Unlikely, as the data has to be fetched over the network (even if it's
local). Even with prefetching etc. this is slower than iterating an array
in mem. Also, the accumulated time of each depends on the complexity of
the operation in the block - that's sometimes overlooked. Note that #each
is called only once per iteration but the block is invoked multiple times.Kind regards
robert