Bash function and logging for an rsync backup script


To synchronize multiple folders via rsync, the individual folders for the sync can be written into a bash file as follows, so far so simple:

#!/bin/bash
rsync -av --delete /folder1/ /backup/folder1
rsync -av --delete /folder2/ /backup/folder2
rsync -av --delete /folder3/ /backup/folder3
rsync -av --delete /folder4/ /backup/folder4

To allow the status of the operation to be logged in a text file, I have adapted the lines as follows:

#!/bin/bash
printf "`date '+%Y%m%d %H%M'` /folder1 `rsync -av --delete /folder1/ /backup/folder1  --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished `date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
printf "`date '+%Y%m%d %H%M'` /folder2 `rsync -av --delete /folder2/ /backup/folder2  --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished `date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
printf "`date '+%Y%m%d %H%M'` /folder3 `rsync -av --delete /folder3/ /backup/folder3  --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished `date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
printf "`date '+%Y%m%d %H%M'` /folder4 `rsync -av --delete /folder4/ /backup/folder4  --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished `date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log

The --stats parameter allows to write a summary of the sync operation to a log file;
head -n 2000 limits the logfile to 2000 lines and writes new entries to the beginning of the logfile via ">".

So that I don't have to change the folder names in each line several times, these can be replaced with variables:

#!/bin/bash
FOLDER=/folder1
printf "`date '+%Y%m%d %H%M'` $FOLDER  `rsync -av --delete $FOLDER/ /backup$FOLDER --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished \n`date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
FOLDER=/folder2
printf "`date '+%Y%m%d %H%M'` $FOLDER  `rsync -av --delete $FOLDER/ /backup$FOLDER --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished \n`date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
FOLDER=/folder3
printf "`date '+%Y%m%d %H%M'` $FOLDER  `rsync -av --delete $FOLDER/ /backup$FOLDER --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished \n`date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
FOLDER=/folder4
printf "`date '+%Y%m%d %H%M'` $FOLDER  `rsync -av --delete $FOLDER/ /backup$FOLDER --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished \n`date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log

After I also copy the command line multiple times with this variant, it can be swapped out into a function. On this occasion I split the one-liner with "\" at the end of the line into several lines:

#!/bin/bash
backup () {
	printf "`date '+%d.%m.%Y %H:%M'` | $1 \n---------------------------------------\n`\
	rsync -av --delete $1/ /backup$1 --exclude .snapshots* --exclude .Trash* --stats | \
	sed '0,/^$/d'`\nfinished: `date '+%d.%m.%Y %H:%M'` \n---------------------------------------\n\n`\
	head -n 2000 /var/log/rsync.txt`" > /var/log/rsync.txt
}
backup /folder1
backup /folder2
backup /folder3
backup /folder4

So far so clear, but to make the log output more readable, I split the log lines into variables. This way you can see a summary of the backup process at the beginning of the logfile, including the runtime of the backup folders and the update of the summary after the script is finished:

#!/bin/bash
LOGFILE="/var/log/rsync.txt"
STARTTIME="`date '+%d.%m.%y %H:%M'`"
START=`date +%s`
THISLOGHEAD="______________________________________\n\
|\n\
|  Script:   $0\n\
|  Hostname: `hostname`\n\
|  Started:  $STARTTIME\n"
LOGCONTENT="`head -n 2000 $LOGFILE`"
THISLOG=""
SUMMARY=""

backup () {
	local FSTARTTIME="`date '+%H:%M'`"
	local FSTART=`date +%s`
	printf "$THISLOGHEAD|\n\n\
---------------------------------------\n\
| Summary:\n\
---------------------------------------\n\
$SUMMARY\n\
`date '+%H:%M'`-now: $1 -> $2\n\n\
---------------------------------------\n\
| Details:\
\n---------------------------------------\n\
$THISLOG\n$LOGCONTENT" > $LOGFILE
	THISLOG="$THISLOG\n`date '+%d.%m.%y %H:%M'` | $1 \n->$2\n---------------------------------------\n`\
	rsync -av --delete $1 $2 --exclude .snapshots* --exclude .Trash* --stats | \
	sed '0,/^$/d'`\nfinished: `date '+%d.%m.%y %H:%M'` \n---------------------------------------\n\n"
	printf "$THISLOGHEAD\n$THISLOG\n$LOGCONTENT" > $LOGFILE
	local FSECONDS="$((`date +%s`-$FSTART))"
	SUMMARY="$SUMMARY \n$FSTARTTIME-`date '+%H:%M'` (`date -d@$FSECONDS -u +%H:%M:%S`): $1 -> $2"
}

backup /folder1 /backup
backup /folder2 /backup
backup /folder3 /backup
backup /folder4 /backup

#Finish Script: Runtime-Information and Final-Summary:
SECONDS="$((`date +%s`-$START))"
printf "$THISLOGHEAD\
|  Finished: `date '+%d.%m.%y %H:%M'`\n\
|  Runtime:  `date -d@$SECONDS -u +%H:%M:%S`\n|\n\n\
---------------------------------------\n\
| Summary:\n\
---------------------------------------\n\
$SUMMARY\n\n\
---------------------------------------\n\
| Details:\n\
---------------------------------------\n\
$THISLOG\n\
|______________________________________\n\
$LOGCONTENT" > $LOGFILE

The variable $THISLOG collects the log entries during the runtime of the script, $SUMMARY is used for the summary and printed with "printf ... > $LOGFILE" the logfile is rewritten at every update.
For the header of the script the variable $THISLOGHEAD is used and the original log content is appended by the variable $LOGCONTENT. The destination can be specified as the second parameter in this example. The output to the text file looks like this with my backup folders:

______________________________________
|
|  Script:   ./rsync-backup.sh
|  Hostname: sox
|  Started:  06.08.21 09:33
|

---------------------------------------
| Summary:
---------------------------------------
 
09:33-09:33 (00:00:01): /scripts -> /media/dreambox/backup/scripts-nas 
09:33-09:41 (00:07:57): /daten/Dokumente -> /media/dreambox/backup/Dokumente 
09:41-09:42 (00:00:26): /daten/PhotoVideo -> /media/dreambox/backup/PhotoVideo 
09:42-09:47 (00:05:47): /docker -> /media/dreambox/backup/docker 
...

---------------------------------------
| Details:
---------------------------------------

06.08.21 09:33 | /scripts 
->/media/dreambox/backup/scripts-nas
---------------------------------------
Number of files: 23 (reg: 20, dir: 3)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 4
Total file size: 26,129 bytes
Total transferred file size: 3,347 bytes
Literal data: 3,347 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 4,377
Total bytes received: 109

sent 4,377 bytes  received 109 bytes  2,990.67 bytes/sec
total size is 26,129  speedup is 5.82
finished: 06.08.21 09:33 
---------------------------------------


06.08.21 09:33 | /daten/Dokumente 
->/media/dreambox/backup/Dokumente
---------------------------------------
Number of files: 353,503 (reg: 324,561, dir: 28,891, link: 51)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 83
Total file size: 204,805,207,475 bytes
Total transferred file size: 449,566,337 bytes
Literal data: 449,566,337 bytes
Matched data: 0 bytes
File list size: 3,143,254
File list generation time: 0.038 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 459,277,206
Total bytes received: 31,776

sent 459,277,206 bytes  received 31,776 bytes  961,903.63 bytes/sec
total size is 204,805,207,475  speedup is 445.90
finished: 06.08.21 09:41 
---------------------------------------

...
positive Bewertung({{pro_count}})
Rate Post:
{{percentage}} % positive
negative Bewertung({{con_count}})

THANK YOU for your review!

Publication: 2022-11-30 from Bernhard 🔔


Questions / Comments