Ruby 2.3.3 Deadlocks from Open3

Tucker_Barbour · 2 November 2017 11:46

I've started to encounter an increase of "No live threads left. Deadlock?"
thrown from Open3 as load increase on our system. The system is a typical
competing consumer model with a single producer and multiple consumers
running across a number of VMs where multiple consumers can reside on the
same VM. The consumers are running Ruby 2.3.3 on Ubuntu 14.04 in EC2. I've
started to see an increase in errors due to potential deadlocks as load
increases, e.g. consumers are doing more work. The consumers use Open3 to
spawn a child process and read from stdout and stderr, executing business
logic based on the output. I've have seen these deadlock errors from
Open3#popen2e, Open3#capture3 and Open3#popen3. In the case of
Open3#popen2e and Open#popen3 I'm using separate threads for reading from
stdout and stderr and calling Thread#value on the wait thread in order to
determine the exit status of the child process. Open3#capture3 already
reads from stdout and stderr in separate threads so the fact that I'm see
these errors from Open3#capture3 seems to indicate the issue may not be
with our application code. Based on the stacktrace it seems the call to
Thread#value is where the Ruby runtime is detecting a potential dead lock
due to multiple threads either waiting on the same resource or all threads
have died or are sleeping. However, I'm simply reading from stdout and
stderr on separate threads and having the main thread join the wait thread
from Open3. There is no thread contention for shared resources in my
application code; at least not on purpose. Having looked at dmsg and VM
metrics it does not appear there were any abnormal memory or CPU issues
around the time of the errors. I have a work around for the issue using
Thread#join with a timeout and checking if the return value from
Thread#join is nil to determine if Thread#join timed out. In the event of a
time out I can execute some business logic and retry. However, I'd like to
determine the root cause of the issue. I'm wondering if anyone else on the
mailing list might have encountered similar issues before and has any
suggestions or guidance for debugging.

Here is a distilled version of what I'm doing, nothing complicated, and a
potential solution.

gist.github.com

https://gist.github.com/ctbarbour/91b12a65feb6c024cf4aafb996ca108b

loop.sh

#!/bin/bash

for i in {0..10000}; do
    echo $i
done

popen2e_deadlock_test.rb

#!/usr/bin/env ruby

require 'open3'

def do_stuff(&block)
  Open3.popen2e('./loop.sh') do |stdin, stdout_and_err, wait_thr|
    stdin.close
    while line = stdout_and_err.gets("\n")
      yield line
    end

This file has been truncated. show original

popen3_deadlock_test.rb

require 'open3'

cmdout = ''
cmderr = ''

result = Open3.popen3('./loop.sh') do |stdin, stdout, stderr, wait_thr|
  out_thr = Thread.new { stdout.read(nil, cmdout) }
  err_thr = Thread.new { stderr.read(nil, cmderr) }
  threads = [out_thr, err_thr, wait_thr].map { |thr| thr.join(300) }
  threads.any?(&:nil?) ? :error : wait_thr.value

This file has been truncated. show original

There are more than three files. show original

- Tucker

···

--
Tucker Barbour
tucker.barbour@gmail.com

Topic		Replies	Views
Thread deadlock problem ruby-talk	5	64	11 September 2007
Why does this code deadlock? ruby-talk	10	107	19 April 2007
Threaded IO trouble ruby-talk	12	82	15 August 2008
Weird deadlock message ruby-talk	4	53	18 July 2008
Threading problem ruby-talk	3	63	10 February 2003

Ruby 2.3.3 Deadlocks from Open3

Related Topics