System calls with ` in parameters

ruby-talk
1

Hi All,

I am trying to add functionality to an interesting script. The part
that is tripping me up is that I have to make a system call to echo, and
feed several parameters. One of the parameters occasionally includes
the ` character.

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `. This causes the
command to fail because it executes up to the tick ` in someVar and not
to the end as intended.

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

But this fails when there is a tick ` in #{someVar} as well.

Any help would be appreciated.

Thanks,
Gerad

···

--
Posted via http://www.ruby-forum.com/.

2

Shellwords should do it:

ruby -rshellwords -e 'p Shellwords.escape "`"'

···

On Feb 9, 2011, at 3:48 PM, Gerad S. wrote:

Hi All,

I am trying to add functionality to an interesting script. The part
that is tripping me up is that I have to make a system call to echo, and
feed several parameters. One of the parameters occasionally includes
the ` character.

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `. This causes the
command to fail because it executes up to the tick ` in someVar and not
to the end as intended.

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

But this fails when there is a tick ` in #{someVar} as well.

Any help would be appreciated.

3

Hi All,

I am trying to add functionality to an interesting script. The part
that is tripping me up is that I have to make a system call to echo, and
feed several parameters. One of the parameters occasionally includes
the ` character.

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `. This causes the
command to fail because it executes up to the tick ` in someVar and not
to the end as intended.

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

Why do you escape the string interpolation? Don't you want those file
names inserted there? It seems you rather want

system("echo '#{someVar}' | ./some-script.py >'#{outputFile}'")

But this fails when there is a tick ` in #{someVar} as well.

Here's a pure Ruby solution:

File.open outputFile, "w" do |out|
  IO.popen "./some-script.py", "rw" do |io|
    t = Thread.new do
      IO.copy_stream(io, out, 1024)
      # pre 1.9:
      # while ( b = io.read(1024) )
      # out.write(b)
      # end
    end

    io.puts(someVar)
    io.close_write

    t.join
  end
end

Of course, it would be even better if you rewrote the Python script in Ruby. :wink:

Cheers

robert

···

On Thu, Feb 10, 2011 at 12:48 AM, Gerad S. <geradstemke@gmail.com> wrote:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

4

If you use Ruby 1.9.2, Open3.pipline() provides a way to run a pipline
without shell.

% ruby -ropen3 -e 'Open3.pipeline(["echo", "foo`bar`baz"], ["head",
"-6c", :out => "/tmp/qux"])'
% od -c /tmp/qux
0000000 f o o ` b a
0000006

···

2011/2/10 Gerad S. <geradstemke@gmail.com>:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

--
Tanaka Akira

5

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `. This causes the
command to fail because it executes up to the tick ` in someVar and not
to the end as intended.

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

But this fails when there is a tick ` in #{someVar} as well.

Could you show us a working example of the problem?

s = "foo `date` baz"

=> "foo `date` baz"

system "echo #{s}"

foo Thu Feb 10 07:18:23 EST 2011 baz
=> true

system "echo \"#{s}\""

foo Thu Feb 10 07:18:32 EST 2011 baz
=> true

system "echo '#{s}'"

foo `date` baz
=> true

require 'shellwords'

=> true

`echo #{Shellwords.escape(s)}`

=> "foo `date` baz\n"

···

On Wed, Feb 9, 2011 at 6:48 PM, Gerad S. <geradstemke@gmail.com> wrote:

6

Gerad S. wrote in post #980690:

Hi All,

I am trying to add functionality to an interesting script. The part
that is tripping me up is that I have to make a system call to echo, and
feed several parameters. One of the parameters occasionally includes
the ` character.

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `.

Does this test case represent your problem?

someVar="foo`bar"

=> "foo`bar"

`echo "#{someVar}"`

sh: Syntax error: EOF in backquote substitution
=> ""

Quick workaround: use single quotes instead (but see below).

`echo '#{someVar}'`

=> "foo`bar\n"

The same occurs with system():

system("echo \"#{someVar}\"")

sh: Syntax error: EOF in backquote substitution
=> false

system("echo '#{someVar}'")

foo`bar
=> true

It's an issue with the shell, not with ruby; the same happens at the
shell prompt too.

$ echo "foo`bar"

bash: unexpected EOF while looking for matching ``'

bash: syntax error: unexpected end of file

The issue is that in the shell, double-quoted strings can *contain*
backtick expansions, for example

$ echo "foo`hostname`bar"
foox100bar

Backticks inside single quotes are not treated specially. However,
single quotes inside single quotes are still a problem:

someVar="foo'bar"

=> "foo'bar"

`echo '#{someVar}'`

sh: Syntax error: Unterminated quoted string
=> ""

So using the Shellwords escaping mechanism, as already shown, is
probably what you want.

···

--
Posted via http://www.ruby-forum.com/\.

7

Cool! Thanks for sharing.

Is there also a version which would return an IO or accept a block
with IO parameter so we can feed something into the head of the
pipeline and read from the tail?

Kind regards

robert

···

On Thu, Feb 10, 2011 at 11:21 AM, Tanaka Akira <akr@fsij.org> wrote:

2011/2/10 Gerad S. <geradstemke@gmail.com>:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

If you use Ruby 1.9.2, Open3.pipline() provides a way to run a pipline
without shell.

% ruby -ropen3 -e 'Open3.pipeline(["echo", "foo`bar`baz"], ["head",
"-6c", :out => "/tmp/qux"])'
% od -c /tmp/qux
0000000 f o o ` b a
0000006

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

8

Robert Klemme wrote in post #980746:

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

Why do you escape the string interpolation? Don't you want those file
names inserted there? It seems you rather want

system("echo '#{someVar}' | ./some-script.py >'#{outputFile}'")

Hi Robert,

Thanks for the help. I used your suggestion of single quotes and now my
script is working like a charm.

Cheers,
Gerad

···

--
Posted via http://www.ruby-forum.com/\.

9

Brian Candler wrote in post #980806:

Does this test case represent your problem?

someVar="foo`bar"

=> "foo`bar"

`echo "#{someVar}"`

sh: Syntax error: EOF in backquote substitution
=> ""

This indeed is a good representation of the problem.

I will have to research if my #{someVar} could possibly contain a '
character. Perhaps Shellwords would be a safer bet.

In case you are still interested, an example of the command with actual
data is below (notice how the sequence @@@@@@S` in the string before " |
collab-table.py" is causing the issue):

`echo "Facs Rep|21|@@@@@@Ir||FACS Version 2.1.36.0|2010-09-27
10:57:04.7108-07|21|2|Under Review|||||f||||||f|2010-09-27
11:14:42.686225-07|09/27/2010||||message|t|Facs
Rep>09/27/2010|closed|f||18| ||||
Facs Rep|21|@@@@@@Lz||FACS Version 3.1.0.0|2010-10-27
10:02:43.778138-07|21|2|Under Review|||||f||||||f|2010-10-27
10:51:17.216686-07|10/27/2010||||message|t|Facs
Rep>10/27/2010|closed|f||19| ||||
Facs Rep|21|@@@@@@N]||FACS Version 3.2.0.0|2010-11-09
16:56:17.214108-08|21|1|Under Review|||||f||||||f|2010-11-09
16:56:17.214108-08|11/09/2010||||message|t|Facs
Rep>11/09/2010|open|f||20| ||||
Facs Rep|21|@@@@@@QT||FACS Version 3.3.0.0|2010-12-09
10:19:06.322005-08|21|1|Under Review|||||f||||||f|2010-12-09
10:19:06.322005-08|12/09/2010||||message|t|Facs
Rep>12/09/2010|open|f||21| ||||
Facs Rep|21|@@@@@@S`||FACS Version 3.4.0.0|2011-01-07
10:17:25.142238-08|21|1|Under Review|||||f||||||f|2011-01-07
10:17:25.142238-08|01/07/2011||||message|t|Facs
Rep>01/07/2011|open|f||22| ||||" | ./collab-table.py "message" "D260888"
"hub" "ps_nysdot" "New York State Department of Transportation"
"Alexander Hamilton Bridge & Highbridge Interchange Ramps Rehab" "Facs
Rep" "21" "2011-02-10 11:58:12-0800" "20" "4" "4">
"cor_6/2011-02-10/D260888/correspondence/CollabmessageList_4.html"`

···

--
Posted via http://www.ruby-forum.com/\.

10

I keep finding out about nifty 1.9 features like this piecemeal. Is
there an up-to-date guide to 1.8-1.9 library changes somewhere?

···

On Thu, Feb 10, 2011 at 3:33 AM, Robert Klemme <shortcutter@googlemail.com> wrote:

 IO\.copy\_stream\(io, out, 1024\)

--
Avdi Grimm

11

% ruby -ropen3 -e '
Open3.pipeline_w(%w[head -6c], %w[cat -n]) {|w|
  w.write "ab\ncd\nef\n"
}'
     1 ab
     2 cd

% ruby -ropen3 -e '
Open3.pipeline_r(["echo", "foo`bar`baz"], ["head","-6c"]) {|r|
  p r.read
}'
"foo`ba"

···

2011/2/10 Robert Klemme <shortcutter@googlemail.com>:

Is there also a version which would return an IO or accept a block
with IO parameter so we can feed something into the head of the
pipeline and read from the tail?

--
Tanaka Akira

12

Is there also a version which would return an IO or accept a block
with IO parameter so we can feed something into the head of the
pipeline and read from the tail?

% ruby -ropen3 -e '
Open3.pipeline_w(%w[head -6c], %w[cat -n]) {|w|
w.write "ab\ncd\nef\n"
}'
1 ab
2 cd

% ruby -ropen3 -e '
Open3.pipeline_r(["echo", "foo`bar`baz"], ["head","-6c"]) {|r|
p r.read
}'
"foo`ba"

Very nice! I tried

13:09:25 Temp$ ./pl.rb
< 1 000>
< 2 001>
< 3 002>
< 4 003>
< 5 004>
13:09:31 Temp$ cat pl.rb
#!/bin/env ruby19

require 'open3'

Open3.pipeline_rw(["cat", "-n"], ["sed", "-e", 's/^.*$/<&>/']) do

sin, sout, threads|

  th = Thread.new(sout) do |io|
    io.each_line do |l|
      puts l
    end
  end

  5.times do |i|
    sin.printf "%03d\n", i
  end

  sin.close
  th.join
end
13:09:58 Temp$

Cheers

robert

···

On Mon, Feb 14, 2011 at 4:52 AM, Tanaka Akira <akr@fsij.org> wrote:

2011/2/10 Robert Klemme <shortcutter@googlemail.com>:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/