Tuesday, October 30, 2012

Uploading a whole directory to a remote server with LFTP

As any user of a restricted VPS or PaaS product (like WPEngine) might know, there are sometimes restrictions that stop you from using ssh or scp.

In my most recent case I really needed to move about 30GB of tiny thumbnail files to WPEngine. I considered downloading them all to my machine with Cyberduck, then uploading them over, but that would have been way too slow. Cyberduck goes into a 'preparing...' cycle that seems to never end, and caps my CPU at 100%.

And in any case, SFTP doesn't support recursive put. What?!

Thankfully, there's LFTP. LFTP is available in just about any sane package manger and works in this fashion:

lftp sftp://user@mysub.domain.com

It uses the same syntax as sftp, plus a few awesome additions.
 
In my case, I wanted to move an entire wp-content/uploads folder over. This is really easy. Simply cd into your existing uploads directory, fire up lftp, and...

lftp user@site.wpengine.com:/wp-content/uploads/2012/10> mirror -R --parallel=20
This is the money shot. mirror -R  means "mirror in reverse", that is, move all local files to remote. And the parallel directive is extremely useful when moving tons of small files. I was seeing incredible (>6MB/sec) transfer rates of these tiny files between sites. And when uploading on my small comcast connection, I was able to saturate the pipe with parallelism this high.

Simply put, there exists no GUI tool that can pull something off this elegantly and quickly. Use it.

4 comments:

  1. Damn man, I don't think this works anymore. Commands to the server just end up timing out. They caught onto us!

    ReplyDelete
  2. I think that this was not a complete and balanced decision that was not taken at the time and could lead to disastrous consequences.

    ReplyDelete
  3. The functionality of this structure allows you to load or recursively update the directory tree. You can speed up the operation by enabling the parallel loading.

    ReplyDelete