Syncing data between servers with rsync, ubuntu 11.04

Categories: ubuntu
Tags: No Tags
Comments: No Comments
Published on: August 2, 2011

Part 1 – Setting up SSH key authentication

If you have important data or you need to replicate data between 2 or more computers running Ubuntu (or any other Linux), you can easaly do this with rsync and public keys for ssh. The script will replicate the synchronized on all hosts so all host contains the sum of all sync-directory.

fist install ssh server on the system to be accessed (named here SOURCE) by running

user@SOURCE$sudo apt-get install openssh-server

and on all clients (named here DESTINATION) the ssh client

user@DESTINATION$sudo apt-get install openssh-client

Now you can start to generate private and public keys for all DESTINATION hosts

user@DESTINATION$ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/syncuser/.ssh/id_rsa):
Created directory '/home/syncuser/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/syncuser/.ssh/id_rsa.
Your public key has been saved in /home/syncuser/.ssh/id_rsa.pub.
The key fingerprint is:
0a:9e:e7:7d:f6:7d:1a:8e:07:2e:d8:a4:81:04:2f:39 syncuser@ubuntu11server

then copy the public key (id_rsa.pub) to SOURCE host

user@DESTINATION$scp .ssh/id_rsa.pub user@SOURCE:

on the SOURCE host do

user@SOURCE$cat id_rsa.pub >>.ssh/authorized_keys

for each DESTINATION host, check that it’s working by issue

user@DESTINATION$ssh user@SOURCE
Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-10-server x86_64)
user@SOURCE$

if everything worked ok you should not get any password question when logging in.

Part 2 – Setting up what to synchronize

Now it’s time to setup the directory to synchronize between the servers, place it anywhere in the file system, but make sure that the sync user has write access to the directory.

user@DESTINATION$sudo mkdir /syncdir
user@DESTINATION$sudo chown user /syncdir

 

Now create a script to synchronize the directories with rsync, use your favorite editor to create it

user@DESTINATION$vi ~/syncdir.sh
#!/bin/bash
#
# Syncdir.sh
# Syncronize script to sync 2 a directory on one or more machines
#
# (C) 2011 Kjell Havneskold nimmis@arctics.se (http://nimmis.arctics.se)
#
#

#the path to directory to sync from the SOURCE host end with /
SPATH=/syncdir/
#the path to directory to sync to the DESTINATION host end with /
DPATH=/syncdir/

#hostname of SOURCE
HOST=SOURCESERVERNAME
#user name to login with on SOURCE host
USER=user

#file in DESTINATION host to log result from sync script
LOGFILE=~/sync.log

echo "---------------------------------------------------------------------------------" >> $LOGFILE
# sync from SOURCE, can't use -auz as we don't run this as root so noch chown/chgrp
rsync -rlptuz -rsh=ssh $USER@$HOST:$SPATH $DPATH  2>&1 >> $LOGFILE
# sync to source
rsync -rlptuz -rsh=ssh $DPATH $USER@$HOST:$SPATH   2>&1 >> $LOGFILE
echo "Sync completed at: `/bin/date`" >> $LOGFILE

 

or download it http://nimmis.arctics.se/wp-content/uploads/2011/08/syncdir.sh_.txt and rename it to syncdir.sh

user@DESTINATION$wget  http://nimmis.arctics.se/wp-content/uploads/2011/08/syncdir.sh_.txt
user@DESTINATION$mv  syncdir.sh_.txt syncdir.sh

 

then make it executable by issue the command

user@DESTINATION$chmod +x syncdir.sh

 

Test i manually by running

user@DESTINATION$./syncdir.sh

 

No error should appear and looking at the log-file should look something like this

user@DESTINATION$tail sync.log
---------------------------------------------------------------------------------
Sync completed at: Tue Aug  2 21:03:13 CEST 2011

Your are finished with the synchronize script, if you have more DESTINATION hosts copy the script to whose also and modify if sync-directory is different.

Part 3 – Setting up automatic sync with cron

Now when we know that the script is working on the DESTINATION host, it’s time to setup that the script is run automatically, in this case once every hour.

On the DESTINATION host edit the cron jobs for the user by issue

user@DESTINATION$cronjob -e

if it’s the first time you use it you will be asked to choose which editor to use

user@DESTINATION$cronjob -e
no crontab for user - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/ed
  2. /bin/nano        <---- easiest
  3. /usr/bin/vim.tiny

Choose 1-3 [2]:

if you choose wrong or don’t get the question you can always select editor with the EDITOR variable (to get out of vi do :q!)

user@DESTINATION$EDITOR=nano cronjob -e

now we can enter the command to run it automatic at the begining of each hour

user@DESTINATION$cronjob -e

# Run my rsync script once every hour xx:00
0 * * * * /path/to/syncdir.sh

checkl that it looks ok by listning the cron list

user@DESTINATION$crontab -l

It’s finished, look in the loggfile something after passing next hour to check that the script realy was executed.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Welcome , today is Thursday, September 21, 2017