This Blog is intended to collect information of my various Intrests,pen my opinion on the information gathered and not intended to educate any one of the information posted,but are most welcome to share there view on them
Wednesday, June 15, 2011
How to take Linux backups powered by Rsync
RSync backups data and does it very clean and well. Rsync only transfers those data that have been modified and changed so that the destination host has an exact replica from the source host. Rysnc is a command line backup tool that handles data transfers in an effective and secure manner like any other known commercial backup softwares around. Rync blends in and integrates flawlessly with linux shell commands combined linux I/O redirections.
Rsync uses a reliable algorithm to bring remote and host files into sync very quickly. Rsync is fast because it just sends the differences in the files over the network instead of sending the complete files. Rsync is often used as a very powerful mirroring process or just as a more capable replacement for the rcp command. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm.
Rsync installation is installed by default System Tool group installation. If rsync is not available from command line and from rpm database, you can install from yum repo using yum like so
# yum -y install rsync
To verify that rsync has been installed successfully
# rpm -qa rsync
There are two different approach on establishing rsync communication between two hosts.
First is by using a remote-shell program such as ssh or rsh. If you want to use this approach, it is required that openssh server or an active ssh connection is present from both sending and receiving host. Openssh installation and configuration would not be covered from this entry. This entry would assume that ssh daemon service is currently configured and listening properly to assigned host port.
Secondly, is by using rsync listening INETD service directly.
This entry hope to cover both of them two worlds.
Rsync usage from command line terminal
Transfer and/or update files from local host into a remote host using rsync
Basic rsync argument is to specify file glob, a source and a destination folder. Destination can be a local host or a remote host. A basic example to transfer from local to remote would be
The above would transfer, using rsync, all *.mp3 files from current local directory into remote_destination_folder of remotehost using remoteusername for authentication. A destination host can be locally or remote host and can be specified as hostname or IP address.
-t preserve time of selected files
*.mp3 selected files to be transferred
remoteusername bash enabled user account from destination host
remotehost destination host where remoteusername is allowed to have access
remote_destination_folder destination folder from destination host owned and accessible
For recursive traversing of directory folders for transferring data from current host to another host using rsync, this would be like so
# rsync -avrzt /var/www myuser@server1:/var/backup/
-a archived mode enabled
-v verbosed mode enabled
-t preserve time stamp
-z compression transfer enabled
-r resursive mode enabled
/var/www selected folder or files glob source location
myuser user account from destination host
server1 destination host
/var/backup destination folder
The above command would issues rsync with verbose mode enabled, compression enabled, arhived mode of rsync data transfer/update. Rsync transfers files and folders from /var/www (including the www folder) into /var/backup folder of server1 host using myuser as login credentials. The /var/backup from destination host is owned or writeable by myuser . All rsync files are created with file ownership and permission owned by myuser having 600 file mode. The transfer would be done preserving symbolic links, devices, attributes, permissions and ownerships of files.
Linux command line tools when combined with each other creates another set of power tools. The above command searches all *.conf files from /etc folder and transfer them to /usr/folder of remotehost via rsync and using user the user login name. This can be handy if you like to backup specific pattern of files from your system or user accounts individual address books, files like that.
To transfer multiple files from multiple directory with some file excemptions using rsync and grep from local to remote would be like so:
If both location contains abc.txt, the latest abc.txt would be transferred to remote host. If /home1/www exist and /home2/www exist, the files from both source kicatuib would be merged into /destination/www .
Fire up two new terminal windows, and from the first window, establish ssh connection from local to remote rsync destination. Then from the second windows try to issue these rsync command. You will notice that rsync never ask any password any more since there is an existing ssh connection with local host to remote host.
More rsync arguments
-W transfer the whole files without considering any update changes from existing destination file or folder
–progress show progress bar and/or percentage
-4 prefers IPv4
-6 prefers IPv6
–bwlimit=KBPS execute rsync with bandwidth limit rate in KBPS
-h display a more human readable screen output
–log-file=FILE dumps file activity into a file
–ignore-existing ignores already existing copy from destination host
–max-size tells rsync to avoid transferring files larger than specified size like
–max-size=10m avoid file transfer with 10MB in filesize
–port tells rsync to connect to rsync server with a different port, default is 873
–stats tells more file transfer info and rsync algorithm stats on the fly
At this point, be reminded to kick the black ninja box to refresh possible monitor related eye strains.
Using Rsync in Daemon Xinetd Mode
Edit /etc/xinetd.d/rsync and modify
disable = yes
disable = no
To run rsync in deamon mode via xinetd, make sure you have similar lines from your /etc/xinetd.d/rsync file like shown below:
disable = no
socket_type = stream
wait = no
user = root
server = /usr/bin/rsync
server_args = –daemon –config=/etc/rsync/rsyncd.conf -v
log_on_failure += USERID
Create /etc/rsync/rsyncd.conf as a default conf file for rsync in daemon xinetd mode.
#rsync shared folder
#make your UID/GUID above owns the below folder and files
#all files and folder would be seen by rsync client
comment = Linux Rsync Server
exclude = *.mp3
# cat /etc/rsync/rsyncd.motd
Welcome to my Rsync Server!
Restart xinetd service for the changes to take effect like so
# service xinetd restart
Verify that rsync is running as xinetd daemon service mode using one tool like ss
# ss -a | grep rsync
LISTEN 0 0 *:rsync *:*
Rsync uses port 873 as its default port for xinetd service. Make sure it is also open from your current firewall settings. A sample firewall rule for opeing rsync from your firewall would be like so
-A RH-Firewall-1-INPUT -m state –state NEW -m tcp -p tcp –dport 873 -j ACCEPT
# service iptables restart
If you wish to have a passwordless rsync, you need to refer to passwordless/passphraseless ssh from one of last month’s entry.
Rsync in daemon mode via command line
An alternative to run rsync in daemon mode is by specifying it from command line. Like so
–daemon enables rsync to be run as daemon mode from terminal
–port specifies port number to use
–config specified rsync conf file
host-IP-address IP address where to bind rsync from
-v enables verbose mode
Listing out files and folder from rsync server
To list out the files from rsync server, simply point your rsync client to the host running rsync in daemon mode or server mode. To test your rsync server from a client linux host would be like so
The above command downloads only files from folder1 to current folder location
Rsync Log Monitoring
Monitoring rsync server messages for any errors or system messages would be like so
# tail -f /var/log/rsync.log
Rsync can also be used for source code control and management for delivering a centralized and distributed sync source codes from rsync server to group of programmers or source developers among departments, more like a CVS approach.
Using rsync linux command provides many abilities and benefits. To name a few, mirror an entire harddisk or partitions, folders and files, entire domain websites, mail spools for replica servers, user’s home folder, a mirror FTP site and much more.