Rsync functionality in Ruby?

Hello,

I wrote a ruby script to allow me to automatically update a repository working copy from an unversioned hierarchy. I use rsync to synchronise the directories prior to svn add/delete and commit. Worked great on my test system (linux). However, because of security issues, on the system where this script will be used there is no rsync utility available.

So, I was wondering, is there an existing rsync-like ruby solution? I checked this newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync functionality (lots of utilities that use/manage rsync though.)

Note that I don't really need the "r" portion of rsync (the sync occurs on the same filesystem), just the ability to synchronise two hierarchies with "--exclude <files>" capability.

I thought I'd ask before embarking on a wheel-reinvention project.

cheers,

paulv

p.s. I'm pushing my luck here, but guidance on an algorithm or recipe for sync functionality would also be appreciated. :o)

Can't you compile it yourself as a user? If not, could you compile "cpdup"?
I'm not sure it runs on Linux, but it's perfect on *BSDs.

Just compile it with ./configure --prefix=$HOME/usr. That usually works for any
program. I do it regularily to install Ruby as a non-root user.

I don't think rsync installed as user is in any way more dangerous than Ruby.

Hope this helps.

Regards,

   Michael

···

On Fri, 31 Aug 2007 18:50:05 +0200, Paul van Delst <Paul.vanDelst@noaa.gov> wrote:

Hello,

I wrote a ruby script to allow me to automatically update a repository working copy from an unversioned hierarchy. I use rsync to synchronise the directories prior to svn add/delete and commit. Worked great on my test system (linux). However, because of security issues, on the system where this script will be used there is no rsync utility available.

I haven't found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I'd be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :slight_smile:

Dusty Doris

···

On Aug 31, 12:49 pm, Paul van Delst <Paul.vanDe...@noaa.gov> wrote:

Hello,

I wrote a ruby script to allow me to automatically update a repository working copy from
an unversioned hierarchy. I use rsync to synchronise the directories prior to svn
add/delete and commit. Worked great on my test system (linux). However, because of
security issues, on the system where this script will be used there is no rsync utility
available.

So, I was wondering, is there an existing rsync-like ruby solution? I checked this
newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync
functionality (lots of utilities that use/manage rsync though.)

Note that I don't really need the "r" portion of rsync (the sync occurs on the same
filesystem), just the ability to synchronise two hierarchies with "--exclude <files>"
capability.

I thought I'd ask before embarking on a wheel-reinvention project.

cheers,

paulv

p.s. I'm pushing my luck here, but guidance on an algorithm or recipe for sync
functionality would also be appreciated. :o)

Paul van Delst wrote:

Hello,

Hi.

I wrote a ruby script to allow me to automatically update a repository working copy from an unversioned hierarchy.

Is there anyway to use SVN's import command to get you 80% of
the way there? See, for example,

  http://subversion.tigris.org/faq.html#in-place-import

Or, some combination of `cp` and `svn diff`? E.g.,

   svn co working_copy
   cp -r unversioned working_copy
   svn diff > patch
   [process the patch results]

or perhaps some sort of inverse, e.g.,

  U_DIRS=`find unversioned -type d`
  U_FILES=`find unversioned -type f`
  svn co working_copy
  cd working_copy
  S_DIRS=`find . \( -name .svn -prune \) -o -type d -print`
  S_FILES=`find . \( -name .svn -prune \) -o -type f -print`
  svn rm --force [ S_DIRS - U_DIRS ]
  svn rm --force [ S_FILES - U_FILES ]
  cp -r unversioned/* .
  svn ci -m'To sync with unversioned.'

Regards,

···

--
Bil Kleb
http://nasarb.rubyforge.org

dusty wrote:

···

On Aug 31, 12:49 pm, Paul van Delst <Paul.vanDe...@noaa.gov> wrote:

Hello,

I wrote a ruby script to allow me to automatically update a repository working copy from
an unversioned hierarchy. I use rsync to synchronise the directories prior to svn
add/delete and commit. Worked great on my test system (linux). However, because of
security issues, on the system where this script will be used there is no rsync utility
available.

So, I was wondering, is there an existing rsync-like ruby solution? I checked this
newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync
functionality (lots of utilities that use/manage rsync though.)

I haven't found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I'd be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :slight_smile:

Hi,

Thanks for the reply. But, if I understand your reply, I can't use anything that uses unix rsync under the covers, because there is no rsync available on the system in question.

cheers,

paulv

I haven't found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I'd be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :slight_smile:

Is this rsync+ssh? If so, what did you handle passwords when making the connection (I wound up going though disgusting conniptions using expect)? I know that you can avoid login by setting up .ssh/known_hosts but that is frowned upon in some circles.

Cheers,
Bob

···

On 31-Aug-07, at 4:15 PM, dusty wrote:

Dusty Doris

----
Bob Hutchison -- tumblelog at http://www.recursive.ca/so/
Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch
http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/

You should look at FileUtils::cp_r with the :preserve option. It's not the same as rsync by any stretch, but it should be fairly similar assuming you didn't need --delete or --include/--exclude patterns. If you do need those, then the source of FileUtils is probably a pretty good place to start rolling your own since it properly does symlinks, permissions, times, and other metadata.

Corey

···

On Aug 31, 12:49 pm, Paul van Delst <Paul.vanDe...@noaa.gov> wrote:

Hello,

I wrote a ruby script to allow me to automatically update a repository working copy from
an unversioned hierarchy. I use rsync to synchronise the directories prior to svn
add/delete and commit. Worked great on my test system (linux). However, because of
security issues, on the system where this script will be used there is no rsync utility
available.

So, I was wondering, is there an existing rsync-like ruby solution? I checked this
newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync
functionality (lots of utilities that use/manage rsync though.)

If you're dicking around with expect you have the
password in plain text somewhere along the line, so you'd be lucky to
avoid frowns even then :slight_smile:

;Use RSA authentication - a blank passphrase on the key (for cronjobs), or
use ssh-agent (for everything else - unit testing etc).

···

On 01/09/07, Bob Hutchison <hutch@recursive.ca> wrote:

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

--
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

Sorry I took so long to reply. Forgot about this thread. I am using
straight rsync. Running an rsync daemon on the servers I want. Then
I setup modules for the access. The reason I do this is to avoid
having ssh keys with blank passwords on my servers. Also, this is
only available on the internal network, so I'll also use things like
firewall rules and access-list in rsync to make sure only the machines
I want to get rsync access can.

Here's an example of a class that calls rsync to get a file listing in
a module.

···

On Sep 1, 11:05 am, Bob Hutchison <hu...@recursive.ca> wrote:

On 31-Aug-07, at 4:15 PM, dusty wrote:

> I haven't found any either, but I do have a nagios plugin using rsync
> to check the timestamps on remote files over rsync. New to posting
> here, whats the best way to post code? Include it inline? I'd be
> happy to share if you want it. I basically use open4 to call rsync
> and then parse the results. It might be helpful in your wheel
> project. :slight_smile:

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

Cheers,
Bob

> Dusty Doris

----
Bob Hutchison -- tumblelog at http://www.recursive.ca/so/
Recursive Design Inc. -- weblog athttp://www.recursive.ca/
hutchhttp://www.recursive.ca/ -- works onhttp://www.raconteur.info/
cms-for-static-content/home/

----

require 'rubygems'
require 'open4'

class RsyncUtils

  attr_reader :rsync

  def initialize(path='')
    paths = ['/bin',
             '/usr/bin',
             '/usr/local/bin',
             '/opt/local/bin'
             ]
    paths << path unless path.empty?
    paths.each() do |path|
      if(test(?x, "#{path}/rsync"))
        @rsync = "#{path}/rsync"
      end
    end
    raise "Cannot find rsync, please specify the path" unless @rsync
  end

  def list(hostname,path,recurs=false)
    recursive = "--recursive" if recurs
    r = run_command("#{@rsync} #{recursive} #{hostname}::#{path}")
    if r.exitstatus == 0
      r.results = RsyncFile.parse_list(r.stdout)
    end
    r
  end

  def list_recursive(hostname,path)
    list(hostname,path,true)
  end

  private
  def run_command(command)
    pid, stdin, stdout, stderr = Open4.popen4(command)
    ignored, status = Process::waitpid2 pid
    RsyncResult.new(stdout.readlines,
                    stderr.readlines,
                    status.exitstatus)
  end

end

Here is a class that hands back my results
----------

class RsyncResult

  attr_reader :stdout, :stderr, :exitstatus
  attr_accessor :results

  def initialize(stdout,stderr,exitstatus)
    @stdout = stdout
    @stderr = stderr
    @exitstatus = exitstatus
  end

end

Here is the class that parses the filelisting to tell me about the
files

------

class RsyncFile

  attr_reader :name, :type, :time, :size

  FILE_TYPE = 0
  DIRECTORY_TYPE = 1
  UNKNOWN_TYPE = 2

  def initialize(name,type,time,size)
    @name = name
    @time = time
    @size = size
    @type = type
  end

  def is_file?
    @type == FILE_TYPE
  end

  def is_dir?
    @type == DIRECTORY_TYPE
  end

  def self.parse_list(files)
    results =
    files.each do |file|
      parts = file.split
      case parts[0]
      when /^[rwx-]{10}$/
        type = FILE_TYPE
      when /^d[rwx-]{9}$/
        type = DIRECTORY_TYPE
      else
        type = UNKNOWN_TYPE
      end
      time = Time.local(*ParseDate::parsedate("#{parts[2]}
#{parts[3]}"))
      size = parts[1].to_i
      name = parts[4]
      results << RsyncFile.new(name,type,time,size)
    end
    results
  end
end

Here is how I'd use it in a script.

  rsync = RsyncUtils.new
  check = rsync.list(@options["--hostname"],@options["--
path"],@options["--recursive"])
  if check.exitstatus > 0
    raise "#{check.stderr}"
  end

#This would list all the files - the rsync output
  if @options["--verbose"]
    puts check.stdout
    exit(0)
  end

Now assume I have another class that is do a comparison of the results
for me. That is where I'm calling crit.compare. That's not as
important, as you can see I'm simply iterating through the results and
checking the Time on the current server vs. the time on the remove
file.

  check.results.each do |r|
    if r.type == RsyncFile::FILE_TYPE
      diff = ((Time.now - r.time) - @options["--drift"].to_i).to_i
      critical_files << r.name unless crit.compare(diff)
      warning_files << r.name unless warn.compare(diff)
    end
  end

So, basically I would call the script

./rsync_file_age.rb --hostname somehost --path somedir/anotherdir/ --
warning 0:10 --critical 10:20

That would compile a command to RsyncUtils such as

rsync somehost::somedir/anotherdir/

It would then use open4 to run that command and would parse the
results.

Hope that is helpful.

BTW an rsyncd.conf file might look like this and would provide the
somedir module.

pid file = /var/run/rsyncd.pid
hosts allow = 10.0.0.10

uid = nobody
gid = nogroup
use chroot = no
max connections = 4
syslog facility = local5

[somedir]
  path = /var/somedir
  read only = true

Other options for unidirectional update on system with GNU utils are:

- use cp -au
- maybe you can also facilitate tar

You can build synchronization out of these by invoking then twice, i.e. once for each direction.

Other than that it should not be too difficult to implement something on your own using methods in class File.

Kind regards

  robert

···

On 01.09.2007 01:36, Corey Jewett wrote:

On Aug 31, 12:49 pm, Paul van Delst <Paul.vanDe...@noaa.gov> wrote:

Hello,

I wrote a ruby script to allow me to automatically update a repository working copy from
an unversioned hierarchy. I use rsync to synchronise the directories prior to svn
add/delete and commit. Worked great on my test system (linux). However, because of
security issues, on the system where this script will be used there is no rsync utility
available.

So, I was wondering, is there an existing rsync-like ruby solution? I checked this
newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync
functionality (lots of utilities that use/manage rsync though.)

You should look at FileUtils::cp_r with the :preserve option. It's not the same as rsync by any stretch, but it should be fairly similar assuming you didn't need --delete or --include/--exclude patterns. If you do need those, then the source of FileUtils is probably a pretty good place to start rolling your own since it properly does symlinks, permissions, times, and other metadata.

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

If you're dicking around with expect you have the
password in plain text somewhere along the line, so you'd be lucky to
avoid frowns even then :slight_smile:

I know, funny isn't it :slight_smile:

;Use RSA authentication - a blank passphrase on the key (for cronjobs), or
use ssh-agent (for everything else - unit testing etc).

I think I'm going to push a bit on the RSA authentication, see where I get.

Cheers,
Bob

···

On 2-Sep-07, at 3:44 AM, Dick Davies wrote:

On 01/09/07, Bob Hutchison <hutch@recursive.ca> wrote:

--
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

----
Bob Hutchison -- tumblelog at http://www.recursive.ca/so/
Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch
http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/