It’s been about half a decade since I’ve been called upon to play with SSH and RSYNC. Back in the day, I was the administrator of a dedicated server (owned by Verio) running Red Hat Enterprise Linux 4 (Yes, 4, that is not a typo. LOL) located somewhere in North America which I managed remotely from the comfort of my apartment. Before there was WHM/Cpanel, there was SSH. But even with WHM installed later on, it still wasn’t as robust as it is today. So some tasks were better done via SSH.
Site mirror and backup via Rsync was one of this.
So what is SSH?
SSH is a network protocol that allows a client application (there are both Linux and Windows versions) to securely communicate with a remote server which grants a user access to the command-line interface of the remote server. Essentially, this allows for remote command execution, as well as access to other secure network services between the two machines. For more on SSH -> http://en.wikipedia.org/wiki/Secure_Shell
And what is Rsync?
Rsync is a utility application and essentially a network protocol as well for Unix-like system. Although there already are ported version for Microsoft Windows. What this application does is allow for efficiently synchronizing files and directories from one location to another (this can be a location local to a machine or remotely between two differently located machines). What separates this from the rest is the efficiency with which it transfers files by minimizing transfer sizes using so-called delta encoding, or data differencing, where appropriate as well as in the speed-optimized Rsync file transfer algorithm. Plus it has the added bonus of allowing encrypted transfers via SSH. For more on rsync -> http://en.wikipedia.org/wiki/Rsync
So yes, I managed a server remotely via command line. Sounds geeky cool, but not entirely cool. Passing long parameter values to a command can be a real pain in the ass. No, the hacker movies lied to you. There are no fancy user interfaces and cool images. Not even the cascading barrage of characters as in the Matrix movie. Anyway, I digress.
Back to the present.
So I’ve been tasked to replicate a production site to a staging server. Unfortunately, the site administrators are a little too picky with their server security. They wouldn’t want people accessing their server any other way other than via SSH. And prior to that, they asked for an SSH public key for which they can white list. Whoa! I’ve long done away with tinkering with a machine via CLI when I finally matured. LOL. Graphical interfaces weren’t made just for pure eye candy. They made manipulating machines easier. No need to type all those forgettable flags and parameters. What took you minutes to type and enter, now just took a fraction of a second.
Anyhow, that explains why I needed to get back to refreshing my memory on how to use these tools. Which, as it turns out, isn’t such a bad thing actually. It reminded me of how handy these tools are, and that they are better many ways than the tools I’ve currently been working with.
The current tool I have for this job is Mozilla’s Filezilla. Don’t get me wrong, Filezilla is a great tool in itself. But I don’t think it cuts it when it comes to replicating large sites and moving a sizeable amount of files. It just takes forever.
Sorry, I blabbed.
The entire point of this post is actually to have an online note on basic SSH and Rsync usage in one easily accessible place. For easy recall, as with most posts here. So, without further ado, the SSH and Rsync Cheat Sheet!
To generate an ssh key pair, run the following command in the console:
ssh-keygen -t rsa
Leave the passphrase blank if you do not wish to “unlock” your key each time you use it. By default, the 2 files will be saved in the .ssh directory within your personal directory.
Connecting via SSH with SSH keys is the preferred connection method as opposed to password authentication which is more vulnerable to brute force attacks. To make this connection possible, the SSH server must first register your key. To do this on a server running WHM/Cpanel, login via the web interface and go to the Security Center to access the Manage SSH Keys Page. Click on import key and in the form that comes up, copy-paste either your public key or your private key in the appropriate text field. You do not need to enter both keys. One of either will suffice.
To connect to an SSH server which recognizes your key, you enter the command like so:
ssh -vv user@hostnameorip
The -vv flags will allow for verbose feedback, allowing you to easily troubleshoot the problem if and when you are unable to connect. By default, the SSH app will look for the SSH keys in the .ssh directory residing within your personal directory. So I’d say, it’s best not to move that around. If you are not prompted for a password and you see a typical command-line prompt, then that means your key has been recognized. Voila! You’re in.
SSH listens on port 22 by default. If the listening port has been altered (ex. 2223), you’ll have to connect by specifying the new port like so:
ssh -vv -p 2223 user@hostnameorip
To perform an Rsync from a remote server to a local machine (what is also called as a pull rsync) via CLI SSH:
rsync -avz -e ssh user@hostnameorip:/remote/dir1 /local/dir2/
Omitting the trailing slash (/) from the remote dir allows for a full recursive sync of the remote dir into the local dir. Such that the final output would be /local/dir2/dir1.
To perform a push rsync (as opposed to pull), just call the ssh connection on the target destination:
rsync -avz /local/dir2/ -e ssh user@hostnameorip:/remote/dir1
For servers with running SSH on non-default ports, you enclose the call to ssh in quotes and the required parameters like so:
rsync -avz -e "ssh -p 2222" user@hostnameorip:/remote/dir1 /local/dir2/
For more rsync tips, visit: https://calomel.org/rsync_tips.html
References:
http://en.wikipedia.org/wiki/Secure_Shell
http://en.wikipedia.org/wiki/Rsync
http://askubuntu.com/questions/61557/how-do-i-set-up-ssh-authentication-keys
Edits:
I came across this problem after I performed an rsync on one server to another. I got an “internal server error”. Turns out this was caused by file and directory permissions – the suexec on the source server was messing up with the permissions on the target files.
You could add a chmod directive to your rsync like so:
rsync --chmod=Du=rwx,go=rx,Fu=rw,og=r ...
But personally, I’d rather not. I hate it when command calls become lengthy. So my alternative solution to this is to perform a recursive chmod on the target system afterwards like so:
find . -type d -exec chmod 755 {} \;
find . -type f -exec chmod 644 {} \;
The first command call finds all objects of type d (directory) on the current working directory and subdirectories and then perform a chmod on them with parameters 755. The second command call does the same except that it looks for objects of type f (file) and performs a 644 chmod.
References:
http://rainsoftletters.wordpress.com/2008/07/22/recursively-chmodi-only-directories-or-files/
http://www.askapache.com/security/chmod-stat.html#chmod-0-to-7777
Add this:
TAR.GZ
This format is my weapon of choice for most compression. It gives very good compression while not utilizing too much of the CPU while it is compressing the data. To compress a directory use the following syntax:
# tar -zcvf archive_name.tar.gz directory_to_compress
To decompress an archive use the following syntax:
# tar -zxvf archive_name.tar.gz
This will extract the files in the archive_name.tar.gz archive in the current directory. Like with the tar format you can optionally extract the files to a different directory:
# tar -zxvf archive_name.tar.gz -C /tmp/extract_here/
Snippet from:
http://www.simplehelp.net/2008/12/15/how-to-create-and-extract-zip-tar-targz-and-tarbz2-files-in-linux/
If you wish to exclude certain sub-directories from your source directory, just add the –exclude flag like so:
rsync -a –exclude cache/ source/ target/
Multiple files:
$ tar cvf some.tar file1 file2 file3
https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/