General security

Data Backups with Bacula: Notifications

Introduction

When the backup solution is already working, we can also take a look at notifications more closely. We can send a notification email for every backup, which is run during the night, to tell us whether the backup has been run successfully or not. We can decide the type of messages we would like to be sent to our email address, whereas other messages should only be logged to a local file.

Bacula uses the bsmtp Bacula's SMTP client for mail submission. If we open the bacula-dir.conf configuration file, we can see that the following bsmtp command is used to send an email. The placeholders of the mail command are replaced as follows:

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Download Now

• %% : %
• %c : client's name
• %d : director's name
• %e : job exit code
• %h : client address
• %i : job ID
• %j : unique job name
• %l : job level
• %n : job name
• %r : recipients
• %s : since time
• %t : job type like INFO, WARNING, ERROR, FATAL, etc
• %v : volume name

The bacula-dir.conf configuration file contains the following mailcommand directive responsible for sending emails to administrator.

[plain]

mailcommand = "/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: %t %e of %c %l" %r”

[/plain]

The command line options for bsmtp are shown below. Note that there are no options to permit authentication to an SMTP server. Normally that's not needed in internal networks, but if we would like to use an external SMTP server, we need to use a different SMTP client like sendmail. From the bsmtp command we can see that it will try to connect to the SMTP server – we need to change the 'localhost' to an actual domain of our internal SMTP server.

To specify where to send the emails in Message resources we have to use the destination, which supports two syntaxes:

• destination = message-type1, message-type2, …
• destination = address = message-type1, message-type2, ...

The first syntax only has one equal sign in its command and the destination can be one of: stdout, stderr or console. Usually, the console destination is used, which sends messages to the Bacula Console, which can be displayed in bconsole program. The second syntax has two equal signs in it and supports the following destinations:

Destination Description

director Messages are sent to the current director that started the job.

file Messages are sent to a filename specified in the address field, where the file is overwritten if already exists.

append Messages are sent to a filename specified in the address field, where the messages are appended to the file if already exists.

syslog Send the messages to syslog; the address field is ignored.

mail Messages are sent to the comma-separated email addresses specified by the address field. An email is sent on every job run, so a lot of messages can be generated during the night.

mail on error Same as mail destination, except that messages are only sent when the job terminates with an error condition.

mail on success Same as mail destination, except that messages are only sent when the job terminates normally.

operator Same as mail destination, except that each messages is sent as received, generating one email per message.

console Messages are sent to the Bacula console.

stdout Messages are sent to stdout.

stderr Messages are sent to stderr.

catalog Messages are sent to catalog database, which are written to the Log table with a timestamp of message generation, which can be used to log reporting programs.

The message-type fields can be one of the following:

Message-Type Description

info Information messages.

warning Warning messages.

error Error messages, which don't cause the Job to terminate.

fatal Fatal error messages, which cause the Job to terminate.

terminate Messages generated when the daemon shuts down.

notsaved Messages generated when the files are not saved, usually because they cannot be accessed.

skipped Messages generated when the files are skipped, usually because of an incremental backup or file exclusion list.

mount Messages generated on volume mounts from storage daemon.

restored Messages generated for each restored file.

all All messages.

security Security related messages usually an unauthorized connection attempts.

alert Alert messages.

volmgmt Volume management messages.

The Message resource taken from bacula-dir.conf is presented below, where five message directives are used starting with: mail (send all emails except skipped emails to admin@company.com), operator (send all mount messages to admin@company.com), console (send all messages except skipped/saved messages to bacula console), append (save all messages except skipped messages to /var/log/bacula/bacula.log) and catalog (send all messages to catalog).

[plain]

Messages {

Name = Standard

mailcommand = "/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: %t %e of %c %l" %r"

operatorcommand = "/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: Intervention needed for %j" %r"

mail = admin@company.com = all, !skipped

operator = admin@company.com = mount

console = all, !skipped, !saved

append = "/var/log/bacula/bacula.log" = all, !skipped

catalog = all

}

[/plain]

To test whether the mail sending is working as it should, we need to pipe the email message to the bsmtp mail client, which accepts the message from stdin. We should also specify the SMTP server localhost with -h option, 'From' header with -f option, 'Subject' header with -s option and specify the actual address we wish to send to at the end. Note that the -d 10 option enables debugging messages, where the actual SMTP commands are shown and can help us debug the unexpected problems.

[plain]

# echo "This is test message." | /usr/sbin/bsmtp -h localhost -f backup@domain.com -s "Test" user@domain.com -d 50

[/plain]

When messaging doesn't work at your end, you should first add the "-d 50" to your mailcommand directive in the bacula-dir.conf configuration file. You can use a configuration line as presented below, which should display debug configuration messages.

[plain]

mailcommand = "/usr/sbin/bsmtp -h mail -f mail@domain.com -s "Bacula: %t %e of %c %l" %r -d 50"

[/plain]

Afterwards, we have to stop the current instance of bacula-dir and start bacula-dir in debug mode by passing it the "-d 200" configuration option. Other configuration options are self-explanatory, but for completeness let's present them here: -u is used to specify the user under which Bacula will run, -g is similar except for the group and -f is used to run the daemon in foreground (as contrary to the background).

[plain]

# /etc/init.d/bacula-director stop

# /usr/sbin/bacula-dir -c /etc/bacula/bacula-dir.conf -u bacula -g bacula -d 200 -f

[/plain]

Next, you can run any defined backup job in bconsole and observe the messages being printed in the debug window. The outputted messages clearly state the whole communication with the mail server located at mail.domain.com. The commands sent to the mail server are presented in bold and are marked with '-->' arrows. First the HELO command is sent to identify the current system, MAIL FROM specifies mail sender, RCPT TO specifies mail recipient, DATA declares the message body and QUIT terminates the connection.

[plain]

edit_job_codes: /usr/sbin/bsmtp -h mail -f mail@domain.com -s "Bacula: %t %e of %c %l" %r -d 50

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:302-0 Debug level = 50

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:381-0 My hostname is: bacula.domain.com

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:405-0 From addr=mail@domain.com

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:445-0 Connected

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:133-0 Calling fgets on read socket rfp.

Message delivery ERROR: Mail prog: mail <-- 220 mail.domain.com ESMTP

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:143-0 mail --> 220 mail.domain.com ESMTP

Message delivery ERROR: Mail prog: bacula.domain.com --> HELO bacula.domain.com

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:133-0 Calling fgets on read socket rfp.

Message delivery ERROR: Mail prog: mail <-- 250 mail.domain.com

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:143-0 mail --> 250 mail.domain.com

Message delivery ERROR: Mail prog: bacula.domain.com --> MAIL FROM:<mail@domain.com>

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:133-0 Calling fgets on read socket rfp.

Message delivery ERROR: Mail prog: mail <-- 250 2.1.0 Ok

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:143-0 mail --> 250 2.1.0 Ok

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:489-0 rcpt to: admin@domain.com

Message delivery ERROR: Mail prog: bacula.domain.com --> RCPT TO:<admin@domain.com>

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:133-0 Calling fgets on read socket rfp.

Message delivery ERROR: Mail prog: mail <-- 250 2.1.5 Ok

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:143-0 mail --> 250 2.1.5 Ok

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:496-0 Data

Message delivery ERROR: Mail prog: bacula.domain.com --> DATA

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:133-0 Calling fgets on read socket rfp.

Message delivery ERROR: Mail prog: mail <-- 354 End data with <CR><LF>.<CR><LF>

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:143-0 mail --> 354 End data with <CR><LF>.<CR><LF>

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:503-0 From: mail@domain.com

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:506-0 Subject: Bacula: Backup OK of myclient-fd Incremental

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:534-0 Sender: bacula@bacula.domain.com

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:539-0 To: admin@domain.combsmtp: bsmtp.c:546-0

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:559-0 Date: Fri, 22 Aug 2014 13:37:11 +0200 (CEST)

Message delivery ERROR: Mail prog: bacula.domain.com --> .

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:133-0 Calling fgets on read socket rfp.

Message delivery ERROR: Mail prog: mail <-- 250 2.0.0 Ok: queued as AE7576109D

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:143-0 mail --> 250 2.0.0 Ok: queued as AE7576109D

Message delivery ERROR: Mail prog: bacula.domain.com --> QUIT

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:133-0 Calling fgets on read socket rfp.

Message delivery ERROR: Mail prog: mail <-- 221 2.0.0 Bye

Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:143-0 mail --> 221 2.0.0 Bye

[/plain]

You've seen that the bsmtp doesn't authenticate users against the mail server and doesn't support such functionality. If mail server requires authentication, we have to use a different mail client like sendmail. At this point, we should check our inbox for messages sent by Bacula, which can be a great help when identifying whether backup jobs were run successfully or not.

At the end of debugging, we need to remove the "-d 50" parameter from the bacula-dir.conf and restart Bacula normally by using the init.d script.

[plain]

# /etc/init.d/bacula-director start

[/plain]

Conclusion

In this article we've presented the Bacula notification system and how it can be configured to fit our needs. We should configure the Bacula notifications to be sent to our email address, where we can easily determine whether a job has been run successfully or not. We can only send error messages generated when the backup job has failed for one reason or another, which limits the number of messages received to our inbox. My suggestion is that you send all messages when starting to use Bacula to see whether everything works as expected, and later on change the configuration file to only send failed backup jobs that need to be investigated further.

When receiving a failed backup job email, we should take proper steps to determine why the backup job has failed. We need to connect to the director and identify the failing client and the reason for failure. So far, my backup jobs failed because the file daemon on the clients was not accessible, because I forgot to configure the bacula-fd to be run during operating system boot. This is especially important for mobile devices, which will be restarted more frequently than regular server devices. Because I had a proper messaging mechanism in place, such errors were quickly discovered and mitigated, but most importantly appropriate rc.d links were created to start bacula-fd at boot time, which prevented further failures.

When in doubt, configure Bacula to send you all messages to email and gradually reconfigure the director to disable the messages of no particular importance.

References

[1] Solid-state drive, https://en.wikipedia.org/wiki/Solid-state_drive.

[2] Tape drive https://en.wikipedia.org/wiki/Tape_drive.

[3] List of backup software https://en.wikipedia.org/wiki/List_of_backup_software.

[4] Bacula-Web, http://www.bacula-web.org/.

[5] The Bootstrap File, http://www.bacula.org/5.2.x-manuals/en/main/main/Bootstrap_File.html.

[6] ESXi 5.1: Using Raw Device Mappings (RDM) on an HP Microserver,
http://forza-it.co.uk/esxi-5-1-using-raw-device-mappings-rdm-on-an-hp-microserver/.

[7] Bacula Installation and Configuration Guide, https://access.redhat.com/site/sites/default/files/attachments/install_1.pdf.

[8] Overview on modifying the Synology Server, bootstrap, ipkg etc, http://forum.synology.com/wiki/index.php/Overview_on_modifying_the_Synology_Server,_bootstrap,_ipkg_etc.

[9] Data Encryption, http://www.bacula.org/5.2.x-manuals/en/main/main/Data_Encryption.html.

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Download Now

[10] Messages Resource, http://www.bacula.org/5.2.x-manuals/en/main/main/Messages_Resource.html.