Gem for segmented SFTP file downloads?


(Ryan Lue) #1

I'm trying to build a utility that will fetch large files or directories
off of a seedbox (or any SSH server, for that matter).
Individual connections are throttled,
so I have traditionally relied on a CLI utility called LFTP
to split large files into smaller segments to be downloaded in parallel.

I get the sense that nothing really compares to `Net::SFTP`,
but it doesn't look like it natively supports segmented downloads.
Does anyone know of a gem that does?

Alternately, I'm considering writing my own gem
to extend `Net::SFTP` to add support for segmented downloads.
But in order for this to work, I need an option like curl's `--range`
to specify a byte range to fetch for a given file,
and it doesn't look like `Net::SFTP` supports this either.
Have I missed something by any chance?
Is there another way to achieve the same thing?

—Ryan


(Marvin Gülker) #2

Hi,

I get the sense that nothing really compares to `Net::SFTP`,
but it doesn't look like it natively supports segmented downloads.
Does anyone know of a gem that does?

Sadly, I don't.

Alternately, I'm considering writing my own gem
[...] Is there another way to achieve the same thing?

There is libssh, a C library that might be able to do that:

  https://www.libssh.org/

You'd have to build a C extension for Ruby or use fiddle from the
stdlib to interact with it, though. You might want to check if somebody
provides bindings to libssh already.

Marvin

···

Am 20. Februar 2019 um 13:02 Uhr +0800 schrieb Ryan Lue:

--
Blog: https://mg.guelker.eu


(Ryan Lue) #3

There is libssh, a C library that might be able to do that:

  https://www.libssh.org/

You'd have to build a C extension for Ruby or use fiddle from the
stdlib to interact with it, though. You might want to check if somebody
provides bindings to libssh already.

Marvin

Ooh, great thinking. It looks like there are existing gems for libssh
_and_ libssh2. I’ll check them out.

Thanks a million!

—Ryan


(Gerard Fowley) #4

Hi Ryan,

Net::SFTP::Operations::File is part of net-sftp. It provides an IO-like
interface for files over SFTP. I have used it to perform low-level file
operations over SFTP. I think you'll find what you need there...

http://net-ssh.github.io/net-sftp/classes/Net/SFTP/Operations/File.html

···

On Sun, Feb 24, 2019 at 9:16 PM Ryan Lue <lists@ryanlue.com> wrote:

> There is libssh, a C library that might be able to do that:
>
> https://www.libssh.org/
>
> You'd have to build a C extension for Ruby or use fiddle from the
> stdlib to interact with it, though. You might want to check if somebody
> provides bindings to libssh already.
>
> Marvin

Ooh, great thinking. It looks like there are existing gems for libssh
_and_ libssh2. I’ll check them out.

Thanks a million!

—Ryan

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>

--

-gf-