jump to navigation

The basics of rsync on Linux November 11, 2010

Posted by Tournas Dimitrios in Linux.

Rysnc helps you transfer data from one location to another in an efficient manner. It is one of those tools that you learn to use and wonder how you lived without it. Rsync is the de facto standard in backup solutions because of its flexibility and power .Rsync checks each file and transfers only what has changed. What does this mean exactly? Rsync will actually look and see what in the file has changed and upload only the part of the file that has changed. Unlike ftp and other transfer solutions rsync doesn’t simply re-upload the entire file.The difference in the files are then compressed (an optionally encrypted through ssh) then sent so the transfer uses the minimal amount of bandwidth. Rsync is often used by Amazon S3 users as they must pay for bandwidth. When you are paying bandwidth bit by bit you can’t afford anything but rsync.
Important features of rsync :

  • Speed: First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
  • Security: rsync allows encryption of data using ssh protocol during transfer.
  • Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.
  • Privileges: No special privileges are required to install and execute rsync
  • Rsync is powerful but unforgiving
  • Rsync follows the unix methodology, do one thing and do it well. Thus it doesn’t provide encryption only efficient file transferring. Run rsync through SSH if you need encryption.
  • Rsync will not delete files that have been removed unless you supply the –delete flag.
  • Windows doesn’t keep file modification times to better than two seconds. Use the –modify-window=2 option to get around this when syncing to Windows file shares.

Syncing vs Full Backup :
Before geting into the details I think it is worth explaining the difference between full backups and incremental backups. With incremental backups I like to think of them as syncing. You are making the two sets of data match. For example, if one data set contains one extra file the incremental backup will only add that one file to the backup. As opposed to re-copying the other files. This is useful for maintaining frequent backups without the added bandwidth or processing overhead.

After reading the basics of rsync follow the 8 examples , because as we know , Linux is 10% reading and 90% practice!!!

The basic syndax is :   rsync  options  source  destination Source and destination could be either local or remote. In case of remote, specify the login name, remote server name and location.
Example 1. Synchronize Two Directories in a Local Server :To sync two directories in a local computer, use the following rsync -zvr command.

  • -z is to enable compression
  • -v verbose
  • -r indicates recursive

$ rsync -zvr /var/opt/installation/inventory/ /root/temp
building file list … done
sent 26385 bytes received 1098 bytes 54966.00 bytes/sec
total size is 44867 speedup is 1.63

Now let us see the timestamp on one of the files that was copied from source to destination. As you see below, rsync didn’t preserve timestamps during sync.
$ ls -l /var/opt/installation/inventory/test.xml /root/temp/test.xml
-r–r–r– 1 bin bin 949 Jun 22 2010 /var/opt/installation/inventory/test.xml
-r–r–r– 1 root bin 949 Nov 10 2010 /root/temp/test.xml
Example 2. Preserve timestamps during Sync using rsync -a :
rsync option -a indicates archive mode. -a option does the following,

  • Recursive mode
  • Preserves symbolic links
  • Preserves permissions
  • Preserves timestamp
  • Preserves owner and group

Now, executing the same command provided in example 1 (But with the rsync option -a) as shown below , rsync preserved timestamps during sync.
$ rsync -azv /var/opt/installation/inventory/ /root/temp/
building file list … done
sent 26499 bytes received 1104 bytes 55206.00 bytes/sec
total size is 44867 speedup is 1.63

$ ls -l /var/opt/installation/inventory/tta.xml /root/temp/tta.xml
-r–r–r– 1 root bin 949 Jun 18 2009 /var/opt/installation/inventory/tta.xml
-r–r–r– 1 root bin 949 Jun 18 2009 /root/temp/tta.xml

Example 3. Synchronize Only One File :To copy only one file, specify the file name to rsync command, as shown below.
$ rsync -v /var/lib/rpm/Pubkeys /root/temp/

sent 42 bytes received 12380 bytes 3549.14 bytes/sec
total size is 12288 speedup is 0.99

Example 4. Synchronize Files From Local to Remote : rsync allows you to synchronize files/directories between the local and remote system.While doing synchronization with the remote server, you need to specify username and ip-address of the remote server. You should also specify the destination directory on the remote server. The format is username@machinename:path
$ rsync -avz /root/temp/ username@

It asks for password while doing rsync from local to remote server.  Sometimes you don’t want to enter the password while backing up files from local to remote server. For example, If you have a backup shell script, that copies files from local to remote server using rsync, you need the ability to rsync without having to enter the password.To do that, setup ssh public key based authentication .
Example 5. Synchronize Files From Remote to Local : When you want to synchronize files from remote to local, specify remote path in source and local path in target as shown below.
$ rsync -avz username@ /root/temp
receiving file list … done
sent 406 bytes received 15810230 bytes 2432405.54 bytes/sec
total size is 45305958 speedup is 2.87
Example 6. Remote shell for Synchronization : rsync allows you to specify the remote shell which you want to use. You can use rsync ssh to enable the secured remote connection.Use rsync -e ssh to specify which remote shell to use. In this case, rsync will use ssh.
$ rsync -avz -e ssh username@ /root/temp
receiving file list … done

sent 406 bytes received 15810230 bytes 2432405.54 bytes/sec
total size is 45305958 speedup is 2.87
Example 7. View the rsync Progress during Transfer :When you use rsync for backup, you might want to know the progress of the backup. i.e how many files are copies, at what rate it is copying the file, etc. rsync –progress option displays detailed progress of rsync execution .
rsync -avz –progress thegeekstuff@ /root/temp/
Example 8 .Backup up your files in another server using encripted SSH :If you have two servers, could be a good idea to make cross backups of the important data, so if any of them fails you can always restore from the other.Here we will use rsync and ssh to make this possible, rsync is used to sync files between folders on the same machine or between machines, ssh will open an encrypted tunnel so the data could be secure on the transfer.We will have the option to make this automatic using a cron job . Of course you have to enable the ssh login with no password first .
rsync -e ‘ssh -ax’ -avz –delete –delete-excluded backed.server:/source/directory $HOME/backup/

You can also use rsnapshot utility (that uses rsync) to backup local linux server, or backup remote linux server.


No comments yet — be the first.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s