[ANN] socket_sendfile

ruby-talk
1

= socket_sendfile

Rubyforge Project:

http://rubyforge.org/projects/rctools/

Documentation:

http://dev.robotcoop.com/Libraries/socket_sendfile/

== About

Socket#sendfile implements sendfile(2) for sending files without copying data
to the process. See the sendfile(2) manual page for more details.

Note that your system must support the sendfile(2) system call the same way
FreeBSD does for Socket#sendfile to work.

== Installing socket_sendfile

First you need an OS that supports sendfile() the way FreeBSD does.

Then install the gem:

   $ sudo gem install socket_sendfile

== Using socket_sendfile

   require 'rubygems'
   require 'socket'
   require 'socket_sendfile'

   socket = TCPSocket.open host, port

   File.open 'myfile' do |fp|
     socket.sendfile fp
   end

···

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

2

[partially answering my own question ...]

The reason that builder can't create xml elements with "-" chars in the element is because strings with "-" chars are not legal Ruby symbols. Builder uses method_missing to create the xml element markup and so it seems Ruby's parser chokes when it tries to convert the string into a symbol.

I've got a workaround below but it is more general than I want (it replaces all "_" chars with "-"). I left some code commented out that I was trying too get working but didn't. It's a bit confusing to debug because the inheritance chain (XmlMarkup < XmlBase < BlankSlate) end up hiding all the normal methods -- I'm a bit surprised that my code works at all.

Anyway, I'm just learning Ruby and figuring out just what is going on here is fun. If anybody has suggestions I'd appreciate it.

Thanks

#!/usr/bin/env ruby
# file: jnlpmarkup.rb

# extends Builder::XmlMarkup for creating XML markup for jnlp documents.
# Some jnlp elements have a "-" char in the element name. Builder doesn't
# accept this character. So this class postprocesses the result and replaces
# proxy jnlp element names such as "<application_desc>" with "<application-desc>"

···

#
# because I can't get what I intended to work right now this just
# replaces all "_" chars with '-' chars

require 'builder/xmlmarkup'

module Builder

   class JnlpMarkup < XmlMarkup

     @@jnlp_in = %w{application_desc applet_desc component_desc installer_desc offline_allowed related_content all_permissions j2ee_application_client_permissions ext_download}
     @@jnlp_out = %w{application-desc applet-desc component-desc installer-desc offline-allowed related-content all-permissions j2ee-application-client-permissions ext-download}

     def method_missing(sym, *args, &block)
       super.gsub(/_/, '-')

# various things that didn't work below:
# x = super
# x = x.gsub(/application_desc/, 'application-desc')
# x = x.gsub(/applet_desc/, 'applet_desc')
# x = x.gsub(/#{@@jnlp_in[i]}/, @@jnlp_out[i])
# @@jnlp_in.each_index do |i| x = x.gsub(/#{@@jnlp_in[i]}/, @@jnlp_out[i]) end
     end
   end
end

--
- Stephen Bannasch
   Concord Consortium, http://www.concord.org

3

i'm just about to write a server which sends 1-3gb images as the response...
this could be quite useful.

one gripe. it doesn't work:

   /home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group readable by
default. any way to adjust this? an env var perhaps? seems like this should
just warn.

ok. one more issue:

     fortytwo :~/tmp > ruby ../a.rb
     /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:392: warning: Insecure world writable dir /usr/local, mode 040777
     /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb: In function `bsock_sendfile':
     /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:46: warning: implicit declaration of function `sendfil
     e'
     /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:396:in `build': error executing gcc -shared -Wall -W -Wpointer-arith -
     Wcast-qual -Wcast-align -Wwrite-strings -Wmissing-noreturn -Werror -g -O2 -I /usr/local/ruby-1.8.4/lib/ruby/1.8/i686-linux -o /home/ahoward/.rub
     y_inline/Inline_BasicSocket_0a2b.so /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c : 256 (CompilationError)
     Renamed /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c to /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c.bad from /usr/local/
     ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:591:in `inline'
             from /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:6
             from /usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
             from ../a.rb:3

my system definitely has sendfile

     fortytwo :~/tmp > PAGER=cat man sendfile|head -12
     SENDFILE(2) Linux Programmer's Manual SENDFILE(2)

     NAME
            sendfile - transfer data between file descriptors

     SYNOPSIS
            #include <sys/sendfile.h>

            ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

guess the signature won't cut it though eh? maybe i'll just have to use dl?

regards.

-a

···

On Fri, 24 Mar 2006, Eric Hodel wrote:

= socket_sendfile

Rubyforge Project:

http://rubyforge.org/projects/rctools/

Documentation:

http://dev.robotcoop.com/Libraries/socket_sendfile/

== About

Socket#sendfile implements sendfile(2) for sending files without copying data
to the process. See the sendfile(2) manual page for more details.

Note that your system must support the sendfile(2) system call the same way
FreeBSD does for Socket#sendfile to work.

== Installing socket_sendfile

First you need an OS that supports sendfile() the way FreeBSD does.

Then install the gem:

$ sudo gem install socket_sendfile

== Using socket_sendfile

require 'rubygems'
require 'socket'
require 'socket_sendfile'

socket = TCPSocket.open host, port

File.open 'myfile' do |fp|
   socket.sendfile fp
end

--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

4

Eric Hodel wrote:

= socket_sendfile

Rubyforge Project:

http://rubyforge.org/projects/rctools/

Documentation:

http://dev.robotcoop.com/Libraries/socket_sendfile/

Was there something wrong with my existing ruby-sendfile library?

http://rubyforge.org/projects/ruby-sendfile

···

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/\.

5

== Using socket_sendfile

require 'rubygems'
require 'socket'
require 'socket_sendfile'

socket = TCPSocket.open host, port

File.open 'myfile' do |fp|
   socket.sendfile fp
end

i'm just about to write a server which sends 1-3gb images as the response...
this could be quite useful.

one gripe. it doesn't work:

  /home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group readable by
default. any way to adjust this? an env var perhaps? seems like this should
just warn.

That's RubyInline complaining. Let me check with Ryan Davis to see what can be done.

ok. one more issue:

    fortytwo :~/tmp > ruby ../a.rb
    /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:392: warning: Insecure world writable dir /usr/local, mode 040777
    /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb: In function `bsock_sendfile':
    /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:46: warning: implicit declaration of function `sendfil
    e'
    /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:396:in `build': error executing gcc -shared -Wall -W -Wpointer-arith -
    Wcast-qual -Wcast-align -Wwrite-strings -Wmissing-noreturn -Werror -g -O2 -I /usr/local/ruby-1.8.4/lib/ruby/1.8/i686-linux -o /home/ahoward/.rub
    y_inline/Inline_BasicSocket_0a2b.so /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c : 256 (CompilationError)
    Renamed /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c to /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c.bad from /usr/local/
    ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:591:in `inline'
            from /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:6
            from /usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
            from ../a.rb:3

my system definitely has sendfile

           #include <sys/sendfile.h>

           ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

guess the signature won't cut it though eh? maybe i'll just have to use dl?

On FreeBSD it looks like this:

      #include <sys/types.h>
      #include <sys/socket.h>
      #include <sys/uio.h>

      int
      sendfile(int fd, int s, off_t offset, size_t nbytes,
          struct sf_hdtr *hdtr, off_t *sbytes, int flags);

And has this to say about non-blocking sockets:

      When using a socket marked for non-blocking I/O, sendfile() may send
      fewer bytes than requested. In this case, the number of bytes success-
      fully written is returned in *sbytes (if specified), and the error EAGAIN
      is returned.

Try this patch:

socket_sendfile.rb.linux.patch (1.56 KB)

···

On Mar 23, 2006, at 7:24 PM, ara.t.howard@noaa.gov wrote:

On Fri, 24 Mar 2006, Eric Hodel wrote:

6

i'm just about to write a server which sends 1-3gb images as the response...
this could be quite useful.

one gripe. it doesn't work:

  /home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group readable by
default.

Your home dir is group and world writable. A 755 directory works, the message is broken. (Well, just fixed in perforce.)

any way to adjust this? an env var perhaps?

The INLINEDIR environment variable controls this, so set it to a directory that's got the right permissions (at least 755).

seems like this should just warn.

Other people may be able to inject code into ~/.ruby_inline if the directory is writeable by other people than you.

That would be Really Bad.

A warning is insufficient. You would be royally screwed by the time you read it.

···

On Mar 23, 2006, at 7:24 PM, ara.t.howard@noaa.gov wrote:

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

7

Your home dir is world and group writable, not just group readable. There is a big big big difference between those two. There is no way I'm going to let it just warn in that case. However, you do point out that the error message poor. I've changed the error message to:

   "#{path} is insecure (#{'%o' % mode}). It may not be group or world writable. Exiting."

···

On Mar 23, 2006, at 7:24 PM, ara.t.howard@noaa.gov wrote:

i'm just about to write a server which sends 1-3gb images as the response...
this could be quite useful.

one gripe. it doesn't work:

  /home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group readable by
default. any way to adjust this? an env var perhaps? seems like this should
just warn.

--
_why: zenspider's most intense moments of solice are immediately following the slaughter [...]
_why: that topknot's the only thing keeping a lid on the righteous anger
bricolage: yeah, that and his flagrant obsession with dvorak

8

ahhhhhh.

:wink:

-a

···

On Sat, 25 Mar 2006, Toby DiPasquale wrote:

Eric Hodel wrote:

= socket_sendfile

Rubyforge Project:

http://rubyforge.org/projects/rctools/

Documentation:

http://dev.robotcoop.com/Libraries/socket_sendfile/

Was there something wrong with my existing ruby-sendfile library?

http://rubyforge.org/projects/ruby-sendfile

--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

9

I didn't know it existed, but:

It isn't as easy to install as a gem is.

It doesn't work well with non-blocking sockets on FreeBSD. Handling it in Ruby is too annoying to be worthwhile.

Mine is 1.0.

Mine has tests.

···

On Mar 24, 2006, at 8:39 AM, Toby DiPasquale wrote:

Eric Hodel wrote:

= socket_sendfile

Rubyforge Project:

http://rubyforge.org/projects/rctools/

Documentation:

dev.robotcoop.com

Was there something wrong with my existing ruby-sendfile library?

http://rubyforge.org/projects/ruby-sendfile

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

10

i totally understand what you are saying. however i dont' see why a ruby lib
should be more secure that ruby itself:

   fortytwo :/tmp > ls -ltar /|grep tmp
   drwxrwxrwt 21 root root 815104 Mar 24 07:41 tmp

   fortytwo :/tmp > echo 'puts 42' > a.rb
   fortytwo :/tmp > chmod 777 a.rb

   fortytwo :/tmp > ruby -W2 -e' $SAFE = 0; load "a.rb"'
   42

it sure seems fine loading world writable files from world writable
directories... unless you set $SAFE above the default:

   fortytwo :/tmp > ruby -W2 -e' $SAFE = 1; load "a.rb"'
   -e:1: warning: Insecure world writable dir /usr/local, mode 040777
   -e:1:in `load': loading from unsafe path /usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8:/usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8/i686-linux:/usr/local/ruby-1.8.4/lib/ruby/site_ruby:/usr/local/ruby-1.8.4/lib/ruby/1.8:/usr/local/ruby-1.8.4/lib/ruby/1.8/i686-linux:. (SecurityError)
           from -e:1

this seems sufficient and allows stupid people like me to shoot their own foot
off if they choose. does RubyInline really need more that this : do nothing
unless not $SAFE.nil? and $SAFE > 0?

regards.

-a

···

On Fri, 24 Mar 2006, Ryan Davis wrote:

On Mar 23, 2006, at 7:24 PM, ara.t.howard@noaa.gov wrote:

i'm just about to write a server which sends 1-3gb images as the response...
this could be quite useful.

one gripe. it doesn't work:

  /home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group readable by
default. any way to adjust this? an env var perhaps? seems like this should
just warn.

Your home dir is world and group writable, not just group readable. There is a big big big difference between those two. There is no way I'm going to let it just warn in that case. However, you do point out that the error message poor. I've changed the error message to:

"#{path} is insecure (#{'%o' % mode}). It may not be group or world writable. Exiting."

--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

11

Your home dir is group and world writable. A 755 directory works, the
message is broken. (Well, just fixed in perforce.)

thanks figured that out later... unfortunately it needs to be as some webdav
stuff lands there and www has it's own group which i cannot affect...

any way to adjust this? an env var perhaps?

The INLINEDIR environment variable controls this, so set it to a directory
that's got the right permissions (at least 755).

this works - thanks!

seems like this should just warn.

Other people may be able to inject code into ~/.ruby_inline if the directory
is writeable by other people than you.

That would be Really Bad.

A warning is insufficient. You would be royally screwed by the time you
read it.

all good but not at $SAFE==0 or $SAFE==nil. after all, $SAFE==nil really
ought to mean $VERY_DANGEROUS_PLEASE==true! :wink:

regards.

-a

···

On Fri, 24 Mar 2006, Eric Hodel wrote:
--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

12

unknown wrote:

···

On Sat, 25 Mar 2006, Toby DiPasquale wrote:

Was there something wrong with my existing ruby-sendfile library?

http://rubyforge.org/projects/ruby-sendfile

ahhhhhh.

:wink:

Does that work for you, ara? It should work on Linux, Solaris and
FreeBSD out of the box. I didn't put it in a gem because I wasn't sure
how to get the GemSpec to correctly identify only the platforms on which
it would build. Let me know.

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/\.

13

Eric Hodel wrote:

It isn't as easy to install as a gem is.

Given.

It doesn't work well with non-blocking sockets on FreeBSD. Handling
it in Ruby is too annoying to be worthwhile.

Thanks for the first suggestions for improvements to ruby-sendfile. I
will fix it over the weekend and make a gem out of it. That way Linux
and Solaris users won't be left out, as I had originally intended they
not be.

P.S. In your test suite for socket_sendfile, you can use the
IO#nonblock= call by requiring 'io/nonblock' instead of having to
explicitly call fcntl. Makes the code more readable, IMO.

···

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/\.

14

it sure as heck does!

     harp:~/ruby-sendfile > cat a.rb
     require 'sendfile'

     open(__FILE__){|f| STDOUT.sendfile f}

     harp:~/ruby-sendfile > ruby a.rb
     require 'sendfile'

     open(__FILE__){|f| STDOUT.sendfile f}

this is fantastic. basically i'll be responding to certain web requests with
something like

   send_mime_header
   sendfile huge_massive_file

and was thinking i'd have to fork a cat to make it fast enough - so this is
fantastic.

thanks alot.

-a

···

On Sat, 25 Mar 2006, Toby DiPasquale wrote:

unknown wrote:

On Sat, 25 Mar 2006, Toby DiPasquale wrote:

Was there something wrong with my existing ruby-sendfile library?

http://rubyforge.org/projects/ruby-sendfile

ahhhhhh.

:wink:

Does that work for you, ara? It should work on Linux, Solaris and
FreeBSD out of the box. I didn't put it in a gem because I wasn't sure
how to get the GemSpec to correctly identify only the platforms on which
it would build. Let me know.

--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

15

I didn't even bother to mark which platforms it would/wouldn't work on. Failing might get people interested enough to send you patches.

···

On Mar 24, 2006, at 9:19 AM, Toby DiPasquale wrote:

unknown wrote:

On Sat, 25 Mar 2006, Toby DiPasquale wrote:

Was there something wrong with my existing ruby-sendfile library?

http://rubyforge.org/projects/ruby-sendfile

ahhhhhh.

:wink:

Does that work for you, ara? It should work on Linux, Solaris and
FreeBSD out of the box. I didn't put it in a gem because I wasn't sure
how to get the GemSpec to correctly identify only the platforms on which
it would build. Let me know.

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

16

Thanks, I was wondering where that wandered off to.

···

On Mar 24, 2006, at 11:25 AM, Toby DiPasquale wrote:

P.S. In your test suite for socket_sendfile, you can use the
IO#nonblock= call by requiring 'io/nonblock' instead of having to
explicitly call fcntl. Makes the code more readable, IMO.

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com