# Rsync functionality in Ruby? **URL:** \ **Category:** ruby-talk\ **Created:** [31 August 2007 16:50 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297 "2007-08-31T16:50:05Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Paul\_Van\_Delst](https://avatars.discourse-cdn.com/v4/letter/p/7cd45c/32.png) [@Paul\_Van\_Delst](https://rubytalk.org/u/Paul_Van_Delst)\ **Post date:** [31 August 2007 16:50 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/1 "2007-08-31T16:50:05Z")
Hello, I wrote a ruby script to allow me to automatically update a repository working copy from an unversioned hierarchy. I use rsync to synchronise the directories prior to svn add/delete and commit. Worked great on my test system (linux). However, because of security issues, on the system where this script will be used there is no rsync utility available. So, I was wondering, is there an existing rsync-like ruby solution? I checked this newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync functionality (lots of utilities that use/manage rsync though.) Note that I don't really need the "r" portion of rsync (the sync occurs on the same filesystem), just the ability to synchronise two hierarchies with "--exclude \" capability. I thought I'd ask before embarking on a wheel-reinvention project. cheers, paulv p.s. I'm pushing my luck here, but guidance on an algorithm or recipe for sync functionality would also be appreciated. :o) ---
**Author:** ![George5](https://avatars.discourse-cdn.com/v4/letter/g/7c8e57/32.png) [@George5](https://rubytalk.org/u/George5)\ **Post date:** [31 August 2007 20:02 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/2 "2007-08-31T20:02:30Z")
Can't you compile it yourself as a user? If not, could you compile "cpdup"? I'm not sure it runs on Linux, but it's perfect on \*BSDs. Just compile it with ./configure --prefix=$HOME/usr. That usually works for any program. I do it regularily to install Ruby as a non-root user. I don't think rsync installed as user is in any way more dangerous than Ruby. Hope this helps. Regards,    Michael > **路路路** > > On Fri, 31 Aug 2007 18:50:05 +0200, Paul van Delst \ wrote: > > > Hello, > > > > I wrote a ruby script to allow me to automatically update a repository working copy from an unversioned hierarchy. I use rsync to synchronise the directories prior to svn add/delete and commit. Worked great on my test system (linux). However, because of security issues, on the system where this script will be used there is no rsync utility available. ---
**Author:** ![dusty](https://avatars.discourse-cdn.com/v4/letter/d/a183cd/32.png) [@dusty](https://rubytalk.org/u/dusty)\ **Post date:** [31 August 2007 20:15 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/3 "2007-08-31T20:15:09Z")
I haven't found any either, but I do have a nagios plugin using rsync to check the timestamps on remote files over rsync. New to posting here, whats the best way to post code? Include it inline? I'd be happy to share if you want it. I basically use open4 to call rsync and then parse the results. It might be helpful in your wheel project. 馃檪 Dusty Doris > **路路路** > > On Aug 31, 12:49 pm, Paul van Delst \ wrote: > > > Hello, > > > > I wrote a ruby script to allow me to automatically update a repository working copy from > > an unversioned hierarchy. I use rsync to synchronise the directories prior to svn > > add/delete and commit. Worked great on my test system (linux). However, because of > > security issues, on the system where this script will be used there is no rsync utility > > available. > > > > So, I was wondering, is there an existing rsync-like ruby solution? I checked this > > newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync > > functionality (lots of utilities that use/manage rsync though.) > > > > Note that I don't really need the "r" portion of rsync (the sync occurs on the same > > filesystem), just the ability to synchronise two hierarchies with "--exclude \" > > capability. > > > > I thought I'd ask before embarking on a wheel-reinvention project. > > > > cheers, > > > > paulv > > > > p.s. I'm pushing my luck here, but guidance on an algorithm or recipe for sync > > functionality would also be appreciated. :o) ---
**Author:** ![BIl\_Kleb1](https://avatars.discourse-cdn.com/v4/letter/b/a587f6/32.png) [@BIl\_Kleb1](https://rubytalk.org/u/BIl_Kleb1)\ **Post date:** [4 September 2007 08:20 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/4 "2007-09-04T08:20:08Z")
Paul van Delst wrote: > Hello, Hi. > I wrote a ruby script to allow me to automatically update a repository working copy from an unversioned hierarchy. Is there anyway to use SVN's import command to get you 80% of the way there? See, for example,   [http://subversion.tigris.org/faq.html#in-place-import](http://subversion.tigris.org/faq.html#in-place-import) Or, some combination of `cp` and `svn diff`? E.g.,    svn co working\_copy    cp -r unversioned working\_copy    svn diff \> patch    [process the patch results] or perhaps some sort of inverse, e.g.,   U\_DIRS=`find unversioned -type d`   U\_FILES=`find unversioned -type f`   svn co working\_copy   cd working\_copy   S\_DIRS=`find . \( -name .svn -prune \) -o -type d -print`   S\_FILES=`find . \( -name .svn -prune \) -o -type f -print`   svn rm --force [S\_DIRS - U\_DIRS]   svn rm --force [S\_FILES - U\_FILES]   cp -r unversioned/\* .   svn ci -m'To sync with unversioned.' Regards, > **路路路** > > -- > Bil Kleb > [http://nasarb.rubyforge.org](http://nasarb.rubyforge.org) ---
**Author:** ![Paul\_Van\_Delst](https://avatars.discourse-cdn.com/v4/letter/p/7cd45c/32.png) [@Paul\_Van\_Delst](https://rubytalk.org/u/Paul_Van_Delst)\ **Post date:** [31 August 2007 21:30 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/5 "2007-08-31T21:30:06Z")
dusty wrote: > **路路路** > > > On Aug 31, 12:49 pm, Paul van Delst \ wrote: > > > > > Hello, > > > > > > I wrote a ruby script to allow me to automatically update a repository working copy from > > > an unversioned hierarchy. I use rsync to synchronise the directories prior to svn > > > add/delete and commit. Worked great on my test system (linux). However, because of > > > security issues, on the system where this script will be used there is no rsync utility > > > available. > > > > > > So, I was wondering, is there an existing rsync-like ruby solution? I checked this > > > newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync > > > functionality (lots of utilities that use/manage rsync though.) > > > > I haven't found any either, but I do have a nagios plugin using rsync > > to check the timestamps on remote files over rsync. New to posting > > here, whats the best way to post code? Include it inline? I'd be > > happy to share if you want it. I basically use open4 to call rsync > > and then parse the results. It might be helpful in your wheel > > project. 馃檪 > > Hi, > > Thanks for the reply. But, if I understand your reply, I can't use anything that uses unix rsync under the covers, because there is no rsync available on the system in question. > > cheers, > > paulv ---
**Author:** ![Bob\_Hutchison1](https://avatars.discourse-cdn.com/v4/letter/b/9e8a1a/32.png) [@Bob\_Hutchison1](https://rubytalk.org/u/Bob_Hutchison1)\ **Post date:** [1 September 2007 15:05 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/6 "2007-09-01T15:05:32Z")
> I haven't found any either, but I do have a nagios plugin using rsync > to check the timestamps on remote files over rsync. New to posting > here, whats the best way to post code? Include it inline? I'd be > happy to share if you want it. I basically use open4 to call rsync > and then parse the results. It might be helpful in your wheel > project. 馃檪 Is this rsync+ssh? If so, what did you handle passwords when making the connection (I wound up going though disgusting conniptions using expect)? I know that you can avoid login by setting up .ssh/known\_hosts but that is frowned upon in some circles. Cheers, Bob > **路路路** > > On 31-Aug-07, at 4:15 PM, dusty wrote: > > > Dusty Doris > > ---- > Bob Hutchison -- tumblelog at [http://www.recursive.ca/so/](http://www.recursive.ca/so/) > Recursive Design Inc. -- weblog at [http://www.recursive.ca/hutch](http://www.recursive.ca/hutch) > [http://www.recursive.ca/](http://www.recursive.ca/) -- works on [http://www.raconteur.info/cms-for-static-content/home/](http://www.raconteur.info/cms-for-static-content/home/) ---
**Author:** ![Corey\_Jewett1](https://avatars.discourse-cdn.com/v4/letter/c/df705f/32.png) [@Corey\_Jewett1](https://rubytalk.org/u/Corey_Jewett1)\ **Post date:** [31 August 2007 23:36 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/7 "2007-08-31T23:36:19Z")
You should look at FileUtils::cp\_r with the :preserve option. It's not the same as rsync by any stretch, but it should be fairly similar assuming you didn't need --delete or --include/--exclude patterns. If you do need those, then the source of FileUtils is probably a pretty good place to start rolling your own since it properly does symlinks, permissions, times, and other metadata. Corey > **路路路** > > > > On Aug 31, 12:49 pm, Paul van Delst \ wrote: > > > > > > > Hello, > > > > > > > > I wrote a ruby script to allow me to automatically update a repository working copy from > > > > an unversioned hierarchy. I use rsync to synchronise the directories prior to svn > > > > add/delete and commit. Worked great on my test system (linux). However, because of > > > > security issues, on the system where this script will be used there is no rsync utility > > > > available. > > > > > > > > So, I was wondering, is there an existing rsync-like ruby solution? I checked this > > > > newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync > > > > functionality (lots of utilities that use/manage rsync though.) ---
**Author:** ![Dick\_Davies](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@Dick\_Davies](https://rubytalk.org/u/Dick_Davies)\ **Post date:** [2 September 2007 07:44 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/8 "2007-09-02T07:44:45Z")
If you're dicking around with expect you have the password in plain text somewhere along the line, so you'd be lucky to avoid frowns even then 馃檪 ;Use RSA authentication - a blank passphrase on the key (for cronjobs), or use ssh-agent (for everything else - unit testing etc). > **路路路** > > On 01/09/07, Bob Hutchison \ wrote: > > > Is this rsync+ssh? If so, what did you handle passwords when making > > the connection (I wound up going though disgusting conniptions using > > expect)? I know that you can avoid login by setting up .ssh/ > > known\_hosts but that is frowned upon in some circles. > > -- > Rasputin :: Jack of All Trades - Master of Nuns > [http://number9.hellooperator.net/](http://number9.hellooperator.net/) ---
**Author:** ![dusty](https://avatars.discourse-cdn.com/v4/letter/d/a183cd/32.png) [@dusty](https://rubytalk.org/u/dusty)\ **Post date:** [22 September 2007 15:45 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/9 "2007-09-22T15:45:03Z")
Sorry I took so long to reply. Forgot about this thread. I am using straight rsync. Running an rsync daemon on the servers I want. Then I setup modules for the access. The reason I do this is to avoid having ssh keys with blank passwords on my servers. Also, this is only available on the internal network, so I'll also use things like firewall rules and access-list in rsync to make sure only the machines I want to get rsync access can. Here's an example of a class that calls rsync to get a file listing in a module. > **路路路** > > On Sep 1, 11:05 am, Bob Hutchison \ wrote: > > > On 31-Aug-07, at 4:15 PM, dusty wrote: > > > > \> I haven't found any either, but I do have a nagios plugin using rsync > > \> to check the timestamps on remote files over rsync. New to posting > > \> here, whats the best way to post code? Include it inline? I'd be > > \> happy to share if you want it. I basically use open4 to call rsync > > \> and then parse the results. It might be helpful in your wheel > > \> project. 馃檪 > > > > Is this rsync+ssh? If so, what did you handle passwords when making > > the connection (I wound up going though disgusting conniptions using > > expect)? I know that you can avoid login by setting up .ssh/ > > known\_hosts but that is frowned upon in some circles. > > > > Cheers, > > Bob > > > > \> Dusty Doris > > > > ---- > > Bob Hutchison -- tumblelog at [http://www.recursive.ca/so/](http://www.recursive.ca/so/) > > Recursive Design Inc. -- weblog athttp://www.recursive.ca/ > > hutchhttp://www.recursive.ca/ -- works onhttp://www.raconteur.info/ > > cms-for-static-content/home/ > > ---- > > require 'rubygems' > require 'open4' > > class RsyncUtils > >   attr\_reader :rsync > >   def initialize(path='') >     paths = ['/bin', >              '/usr/bin', >              '/usr/local/bin', >              '/opt/local/bin' >              ] >     paths \<\< path unless path.empty? >     paths.each() do |path| >       if(test(?x, "#{path}/rsync")) >         @rsync = "#{path}/rsync" >       end >     end >     raise "Cannot find rsync, please specify the path" unless @rsync >   end > >   def list(hostname,path,recurs=false) >     recursive = "--recursive" if recurs >     r = run\_command("#{@rsync} #{recursive} #{hostname}::#{path}") >     if r.exitstatus == 0 >       r.results = RsyncFile.parse\_list(r.stdout) >     end >     r >   end > >   def list\_recursive(hostname,path) >     list(hostname,path,true) >   end > >   private >   def run\_command(command) >     pid, stdin, stdout, stderr = Open4.popen4(command) >     ignored, status = Process::waitpid2 pid >     RsyncResult.new(stdout.readlines, >                     stderr.readlines, >                     status.exitstatus) >   end > > end > > Here is a class that hands back my results > ---------- > > class RsyncResult > >   attr\_reader :stdout, :stderr, :exitstatus >   attr\_accessor :results > >   def initialize(stdout,stderr,exitstatus) >     @stdout = stdout >     @stderr = stderr >     @exitstatus = exitstatus >   end > > end > > Here is the class that parses the filelisting to tell me about the > files > > ------ > > class RsyncFile > >   attr\_reader :name, :type, :time, :size > >   FILE\_TYPE = 0 >   DIRECTORY\_TYPE = 1 >   UNKNOWN\_TYPE = 2 > >   def initialize(name,type,time,size) >     @name = name >     @time = time >     @size = size >     @type = type >   end > >   def is\_file? >     @type == FILE\_TYPE >   end > >   def is\_dir? >     @type == DIRECTORY\_TYPE >   end > >   def self.parse\_list(files) >     results = >     files.each do |file| >       parts = file.split >       case parts[0] >       when /^[rwx-]{10}$/ >         type = FILE\_TYPE >       when /^d[rwx-]{9}$/ >         type = DIRECTORY\_TYPE >       else >         type = UNKNOWN\_TYPE >       end >       time = Time.local(\*ParseDate::parsedate("#{parts[2]} > #{parts[3]}")) >       size = parts[1].to\_i >       name = parts[4] >       results \<\< RsyncFile.new(name,type,time,size) >     end >     results >   end > end > > Here is how I'd use it in a script. > >   rsync = RsyncUtils.new >   check = rsync.list(@options["--hostname"],@options["-- > path"],@options["--recursive"]) >   if check.exitstatus \> 0 >     raise "#{check.stderr}" >   end > > #This would list all the files - the rsync output >   if @options["--verbose"] >     puts check.stdout >     exit(0) >   end > > Now assume I have another class that is do a comparison of the results > for me. That is where I'm calling crit.compare. That's not as > important, as you can see I'm simply iterating through the results and > checking the Time on the current server vs. the time on the remove > file. > >   check.results.each do |r| >     if r.type == RsyncFile::FILE\_TYPE >       diff = ((Time.now - r.time) - @options["--drift"].to\_i).to\_i >       critical\_files \<\< r.name unless crit.compare(diff) >       warning\_files \<\< r.name unless warn.compare(diff) >     end >   end > > So, basically I would call the script > > ./rsync\_file\_age.rb --hostname somehost --path somedir/anotherdir/ -- > warning 0:10 --critical 10:20 > > That would compile a command to RsyncUtils such as > > rsync somehost::somedir/anotherdir/ > > It would then use open4 to run that command and would parse the > results. > > Hope that is helpful. > > BTW an rsyncd.conf file might look like this and would provide the > somedir module. > > pid file = /var/run/rsyncd.pid > hosts allow = 10.0.0.10 > > uid = nobody > gid = nogroup > use chroot = no > max connections = 4 > syslog facility = local5 > > [somedir] >   path = /var/somedir >   read only = true ---
**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\ **Post date:** [1 September 2007 09:30 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/10 "2007-09-01T09:30:04Z")
Other options for unidirectional update on system with GNU utils are: - use cp -au - maybe you can also facilitate tar You can build synchronization out of these by invoking then twice, i.e. once for each direction. Other than that it should not be too difficult to implement something on your own using methods in class File. Kind regards   robert > **路路路** > > On 01.09.2007 01:36, Corey Jewett wrote: > > > > > On Aug 31, 12:49 pm, Paul van Delst \ wrote: > > > > > > > > > Hello, > > > > > > > > > > I wrote a ruby script to allow me to automatically update a repository working copy from > > > > > an unversioned hierarchy. I use rsync to synchronise the directories prior to svn > > > > > add/delete and commit. Worked great on my test system (linux). However, because of > > > > > security issues, on the system where this script will be used there is no rsync utility > > > > > available. > > > > > > > > > > So, I was wondering, is there an existing rsync-like ruby solution? I checked this > > > > > newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync > > > > > functionality (lots of utilities that use/manage rsync though.) > > > > You should look at FileUtils::cp\_r with the :preserve option. It's not the same as rsync by any stretch, but it should be fairly similar assuming you didn't need --delete or --include/--exclude patterns. If you do need those, then the source of FileUtils is probably a pretty good place to start rolling your own since it properly does symlinks, permissions, times, and other metadata. ---
**Author:** ![Bob\_Hutchison1](https://avatars.discourse-cdn.com/v4/letter/b/9e8a1a/32.png) [@Bob\_Hutchison1](https://rubytalk.org/u/Bob_Hutchison1)\ **Post date:** [4 September 2007 13:11 UTC](https://rubytalk.org/t/rsync-functionality-in-ruby/40297/11 "2007-09-04T13:11:41Z")
> > Is this rsync+ssh? If so, what did you handle passwords when making > > the connection (I wound up going though disgusting conniptions using > > expect)? I know that you can avoid login by setting up .ssh/ > > known\_hosts but that is frowned upon in some circles. > > If you're dicking around with expect you have the > password in plain text somewhere along the line, so you'd be lucky to > avoid frowns even then 馃檪 I know, funny isn't it 馃檪 > ;Use RSA authentication - a blank passphrase on the key (for cronjobs), or > use ssh-agent (for everything else - unit testing etc). I think I'm going to push a bit on the RSA authentication, see where I get. Cheers, Bob > **路路路** > > On 2-Sep-07, at 3:44 AM, Dick Davies wrote: > > > On 01/09/07, Bob Hutchison \ wrote: > > > -- > > Rasputin :: Jack of All Trades - Master of Nuns > > [http://number9.hellooperator.net/](http://number9.hellooperator.net/) > > ---- > Bob Hutchison -- tumblelog at [http://www.recursive.ca/so/](http://www.recursive.ca/so/) > Recursive Design Inc. -- weblog at [http://www.recursive.ca/hutch](http://www.recursive.ca/hutch) > [http://www.recursive.ca/](http://www.recursive.ca/) -- works on [http://www.raconteur.info/cms-for-static-content/home/](http://www.raconteur.info/cms-for-static-content/home/)