Long calculation & time limit

ruby-talk
1

Hello,

I'm working on a programm that is supposed to do really long
calculation. However after some time it gets killed automaticly for
taking too long. How can I disable this limitation?

Thanks =)

···

--
Posted via http://www.ruby-forum.com/.

2

You'll need to let us know what operating system you're using, and what
is killing the process: the O/S itself, or another program?
Also, what's your safety net if the program locks up while doing the
calculation? Would it be possible to break down the process into
sections which wouldn't trigger the safety?

···

--
Posted via http://www.ruby-forum.com/.

3

Thanks for the reply.

I'm using ruby & geany to code and my OS is ubuntu 12.04 (i'll check
that when i'm home).

Loops within the program are controlled with iteration caps, and i am
pretty sure that what's taking so long is the access to a very big CSV
file (3 millions and something lines) therefore it's a single CSV.open
command, which is kinda hard to break down imho.

The error (137 or 37 i'll double check)is returned by the terminal
opened by executing the script, the internetz told me it's a time
limitation thing and I don't know if it's a ruby limitation or linux's
and I don't know how to know :slight_smile:

However the program is supposed to take a long time so removing the time
limit all together would be fine by me.

Thanks for your help.

···

--
Posted via http://www.ruby-forum.com/.

4

If it's the file read that's taking a long time, maybe you should go
about that a different way. Here's an example I found after a few
seconds of googling:

There are a few good ideas and comments in there, you might want to look
into IO.foreach

···

--
Posted via http://www.ruby-forum.com/.

5

I'll look into that and from what I've seen so far it'll certainly help.
Thanks for that :slight_smile:

However, what to do when the computation is still supposed to be long?
How to disable this limitation?

···

--
Posted via http://www.ruby-forum.com/.

6

what do I look for in /var/log/messages ?
It's quite a list...

···

--
Posted via http://www.ruby-forum.com/.

7

Is it your own machine or is the scripted running on a hosted box?

Henry

···

On 5/09/2012, at 10:15 PM, toto tartemolle wrote:

Hello,

I'm working on a programm that is supposed to do really long
calculation. However after some time it gets killed automaticly for
taking too long. How can I disable this limitation?

8

grep <script-name> /var/log/{messages,syslog}

Doesn't return anything...

Is it your own machine or is the scripted running on a hosted box?

It runs on my computer

···

--
Posted via http://www.ruby-forum.com/\.

9

If you want to know exactly what part of your program is taking a long
time, then use something like this to log the intervals:

def timer
  #report time difference
  end_time = Time.now
  puts "Time elapsed #{(end_time - @beginning_time)} seconds"
  @beginning_time = Time.now
end

@beginning_time = Time.now

#do something
puts "doing something"
timer

#do something else
puts "Doing something else"
timer

···

--
Posted via http://www.ruby-forum.com/.

10

If you want to know exactly what part of your program is taking a long
time, then use something like this to log the intervals

I'm already using a timer. I mentionned earlier that the parts of the
program are supposed to take time. Time taken is not the core problem,
the limitation is.

ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 99
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Message at full

litterally: "Killed, returned error 137"

···

--
Posted via http://www.ruby-forum.com/\.

11

How can I change values in the "ulimit -a" btw?

···

--
Posted via http://www.ruby-forum.com/.

12

An exit code of 137 means the program was shut down by a KILL signal, which can happen if you run out of memory, for instance. Try looking in /var/log/messages for information about what caused your script to be terminated.

···

On 09/05/2012 02:50 PM, toto tartemolle wrote:

Thanks for the reply.

I'm using ruby & geany to code and my OS is ubuntu 12.04 (i'll check
that when i'm home).

Loops within the program are controlled with iteration caps, and i am
pretty sure that what's taking so long is the access to a very big CSV
file (3 millions and something lines) therefore it's a single CSV.open
command, which is kinda hard to break down imho.

The error (137 or 37 i'll double check)is returned by the terminal
opened by executing the script, the internetz told me it's a time
limitation thing and I don't know if it's a ruby limitation or linux's
and I don't know how to know :slight_smile:

--
Lars Haugseth

13

The name of your script would be a good start, seeing as there's not much else to go on at this point.

You should probably also look in /var/log/syslog, since later Ubuntu releases by default are no longer logging to /var/log/messages, it seems.

  $ grep <script-name> /var/log/{messages,syslog}

···

On 09/05/2012 07:58 PM, toto tartemolle wrote:

what do I look for in /var/log/messages ?
It's quite a list...

--
Lars Haugseth

14

toto tartemolle wrote in post #1074772:

The error (137 or 37 i'll double check)is returned by the terminal
opened by executing the script, the internetz told me it's a time
limitation thing and I don't know if it's a ruby limitation or linux's
and I don't know how to know :slight_smile:

It would be helpful to copy-paste the full, exact message you see.

When you are logged in as the same user who is running the ruby script,
can you also type the command "ulimit -a" at the command line, and paste
the results here.

Regards,

Brian.

···

--
Posted via http://www.ruby-forum.com/\.

15

toto tartemolle wrote in post #1075217:

ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 99
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Well it was a long shot, but I just wanted to check that cpu time was
unlimited (which it is).

Message at full

litterally: "Killed, returned error 137"

Nothing else? No ruby backtrace, nothing?

Actually that makes sense. As has been pointed out, 137 = 128 (killed)
+9 (SIGKILL). This means that some other process sent a kill signal,
e.g. using "kill -9 <pid>", which ruby cannot trap; it dies instantly.

However it is up to you to find out which other process sent the signal.
Have you or someone on this system implemented some sort of watchdog
process? This is *not* default behaviour.

Try running a simple infinite loop:

    ruby -e 'loop {}'

Does this also die?

Another suggestion: keep monitoring your process using 'top' while it
runs. Keep a note of the memory usage. See if it dies at a consistent
point.

I don't *think* that simply running out of memory will cause the process
to be killed -9; malloc should fail and ruby should be able to print
something.

···

--
Posted via http://www.ruby-forum.com/\.

16

Ubuntu (Linux) will definitely kill off processes consuming large amounts
of memory if all the system RAM has been used up and the box is swapping
heavily.

OP, check your script for memory leaks.

-Doug Seifert

···

On Sun, Sep 9, 2012 at 12:51 PM, Brian Candler <lists@ruby-forum.com> wrote:

toto tartemolle wrote in post #1075217:
>>ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 20
> file size (blocks, -f) unlimited
> pending signals (-i) 16382
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 99
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) unlimited
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited

Well it was a long shot, but I just wanted to check that cpu time was
unlimited (which it is).

>>Message at full
>
> litterally: "Killed, returned error 137"

Nothing else? No ruby backtrace, nothing?

Actually that makes sense. As has been pointed out, 137 = 128 (killed)
+9 (SIGKILL). This means that some other process sent a kill signal,
e.g. using "kill -9 <pid>", which ruby cannot trap; it dies instantly.

However it is up to you to find out which other process sent the signal.
Have you or someone on this system implemented some sort of watchdog
process? This is *not* default behaviour.

Try running a simple infinite loop:

    ruby -e 'loop {}'

Does this also die?

Another suggestion: keep monitoring your process using 'top' while it
runs. Keep a note of the memory usage. See if it dies at a consistent
point.

I don't *think* that simply running out of memory will cause the process
to be killed -9; malloc should fail and ruby should be able to print
something.

--
Posted via http://www.ruby-forum.com/\.

17

Yeah, I've used Ubuntu for years and I've never heard of a process being
killed because it took too long (how would daemons work otherwise?)
However, the kernel's OOM (Out Of Memory) killer will definitely kill
processes that are starving the system of memory. If you're loading all the
records into memory instead of processing them in a streaming or chunked
fashion you could easily be running afoul of the OOM killer.

···

On Sep 11, 2012 5:11 PM, "Douglas Seifert" <doug@dseifert.net> wrote:

On Sun, Sep 9, 2012 at 12:51 PM, Brian Candler <lists@ruby-forum.com>wrote:

toto tartemolle wrote in post #1075217:
>>ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 20
> file size (blocks, -f) unlimited
> pending signals (-i) 16382
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 99
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) unlimited
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited

Well it was a long shot, but I just wanted to check that cpu time was
unlimited (which it is).

>>Message at full
>
> litterally: "Killed, returned error 137"

Nothing else? No ruby backtrace, nothing?

Actually that makes sense. As has been pointed out, 137 = 128 (killed)
+9 (SIGKILL). This means that some other process sent a kill signal,
e.g. using "kill -9 <pid>", which ruby cannot trap; it dies instantly.

However it is up to you to find out which other process sent the signal.
Have you or someone on this system implemented some sort of watchdog
process? This is *not* default behaviour.

Try running a simple infinite loop:

    ruby -e 'loop {}'

Does this also die?

Another suggestion: keep monitoring your process using 'top' while it
runs. Keep a note of the memory usage. See if it dies at a consistent
point.

I don't *think* that simply running out of memory will cause the process
to be killed -9; malloc should fail and ruby should be able to print
something.

--
Posted via http://www.ruby-forum.com/\.

Ubuntu (Linux) will definitely kill off processes consuming large amounts
of memory if all the system RAM has been used up and the box is swapping
heavily.

OP, check your script for memory leaks.

-Doug Seifert