In my most recent case I really needed to move about 30GB of tiny thumbnail files to WPEngine. I considered downloading them all to my machine with Cyberduck, then uploading them over, but that would have been way too slow. Cyberduck goes into a 'preparing...' cycle that seems to never end, and caps my CPU at 100%.
And in any case, SFTP doesn't support recursive put. What?!
Thankfully, there's LFTP. LFTP is available in just about any sane package manger and works in this fashion:
lftp sftp://user@mysub.domain.com
It uses the same syntax as sftp, plus a few awesome additions.
In my case, I wanted to move an entire wp-content/uploads folder over. This is really easy. Simply cd into your existing uploads directory, fire up lftp, and...
lftp user@site.wpengine.com:/wp-content/uploads/2012/10> mirror -R --parallel=20This is the money shot. mirror -R means "mirror in reverse", that is, move all local files to remote. And the parallel directive is extremely useful when moving tons of small files. I was seeing incredible (>6MB/sec) transfer rates of these tiny files between sites. And when uploading on my small comcast connection, I was able to saturate the pipe with parallelism this high.
Simply put, there exists no GUI tool that can pull something off this elegantly and quickly. Use it.
Damn man, I don't think this works anymore. Commands to the server just end up timing out. They caught onto us!
ReplyDeleteVery nice demo animation!
ReplyDeleteI think that this was not a complete and balanced decision that was not taken at the time and could lead to disastrous consequences.
ReplyDeleteThe functionality of this structure allows you to load or recursively update the directory tree. You can speed up the operation by enabling the parallel loading.
ReplyDelete