General security

Data Backups with Bacula: Client Data Backup and Restoration

Dejan Lukan
August 29, 2014 by
Dejan Lukan

Introduction

When configuring the Bacula client, we don't need the full-blown Bacula installation, but just the bacula-client package that we can simply install by using the apt-get command. The client configuration consists of editing the /etc/bacula/bacula-fd.conf configuration file and ensuring that the name and the password in the Director definition block are the same as defined in the bacula-dir.conf on the Bacula server.

[plain]

# apt-get install bacula-client

[/plain]

FREE role-guided training plans

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Data Backup

When choosing to backup a new client, we should add a new client definition in the bacula-dir.conf. The Client definitions should specify the name, the address and port number of a file daemon. Special care should be taken that proper password from the bacula-fd.conf configuration from from the client endpoint is copied to the Bacula director configuration file.

[plain]

Client {

Name = client-fd

Address = client

FDPort = 9102

Catalog = MyCatalog

Password = "ffAC33z3JNGCYfzj/EPW7Aisuw0JWi7SmjtnGP+"

File Retention = 30 days

Job Retention = 6 months

AutoPrune = yes

}

[/plain]

Next, we can connect to the Bacula server by using bconsole, while a new backup can be issued by using the run command in bconsole as shown below.

[plain]

*run

A job name must be specified.

The defined Job resources are:

1: BackupClient

2: Catalog

Select Job resource (1-4): 2

Run Backup job

JobName: BackupClient

Level: Incremental

Client: computer-fd

FileSet: Full Set

Pool: File (From Job resource)

Storage: File (From Job resource)

When: 2014-08-09 13:38:27

Priority: 10

OK to run? (yes/mod/no): yes

Job queued. JobId=8

You have messages.

[/plain]

The run command printed some messages, which we can see with the messages command presented below – the run command was not run successfully, because there was some problem with permissions.

[plain]

09-Aug 13:35 baculaserver-dir JobId 7: No prior Full backup Job record found.

09-Aug 13:35 baculaserver-dir JobId 7: No prior or suitable Full backup found in catalog. Doing FULL backup.

09-Aug 13:35 baculaserver-dir JobId 7: Start Backup JobId 7, Job=BackupClient.2014-09-09_13.35.16_04

09-Aug 13:35 baculaserver-sd JobId 7: Error: dev.c:120 Unable to stat device /backup/: ERR=Permission denied

[/plain]

The permissions of the /backup/ directory is root:root as presented below. This is clearly the problem, since Bacula storage runs under bacula:bacula.

[plain]

# ls -ld /backup/

drwxr-xr-x 3 root root 4096 Aug 5 20:46 /backup/

[/plain]

To enable the bacula-sd daemon to write the files to the disk, we have to change the directory permissions:

[plain]

# chown bacula:bacula /backup -R

[/plain]

Once we rerun the backup, we can again take a look at the messages created by Bacula in order to see whether everything worked. Below we can see that Bacula informed us that we have to create a new volume with the label command. A volume is a single file/tape where Bacula will write backup data, while a pool groups multiple volumes together to overcome the limitation of being restricted to the length of a single volume. Bacula stores backup data in a volume and keeps the information about which file is stored in each volume in a catalog. We can create volumes manually by using the label command or we can use the 'Label Format' directive inside Pool definitions, which will automatically create a volume with a specified name.

[plain]

*messages

09-Aug 13:59 backupserver-sd JobId 10: Job BackupClient.2014-08-09_13.57.49_07 is waiting. Cannot find any appendable volumes.

Please use the "label" command to create a new Volume for:

Storage: "FileStorage" (/backup/)

Pool: File

Media type: File

[/plain]

Next, we can again rerun the backup job, but this time let's first enable the autodisplay, which will automatically display all new messages without us requiring to run messages command manually. Below we can see the output of a successful backup where a full backup was done since one wasn't found before. The backup job was done on a BackupClient with an ID 1 and a new volume "Clients-0001" was created: this means that a file with the same name is also created in the /backup/ directory, which will hold the contents of that volume. There is also some statistical data presenting the fileset, pool, catalog, storage, priority and schedule of the backup. The important line is "Termination: Backup OK", which states that the backup job was successfully run.

[plain]

*autodisplay ON

09-Aug 18:04 baculaserver-dir JobId 1: No prior Full backup Job record found.

09-Aug 18:04 baculaserver-dir JobId 1: No prior or suitable Full backup found in catalog. Doing FULL backup.

09-Aug 18:04 baculaserver-dir JobId 1: Start Backup JobId 1, Job=BackupClient .2014-08-09_18.04.12_04

09-Aug 18:04 baculaserver-dir JobId 1: Created new Volume "Clients-0001" in catalog.

09-Aug 18:04 baculaserver-dir JobId 1: Using Device "FileStorage"

09-Aug 18:04 baculaserver-sd JobId 1: Labeled new Volume "Clients-0001" on device "FileStorage" (/backup/).

09-Aug 18:04 baculaserver-sd JobId 1: Wrote label to prelabeled Volume "Clients-0001" on device "FileStorage" (/backup/)

09-Aug 18:04 baculaserver-sd JobId 1: Job write elapsed time = 00:00:01, Transfer rate = 133.0 M Bytes/second

09-Aug 18:04 baculaserver-dir JobId 1: Bacula baculaserver-dir 5.2.6 (21Feb12):

Build OS: x86_64-pc-linux-gnu debian 7.0

JobId: 1

Job: BackupClient.2014-08-09_18.04.12_04

Backup Level: Full (upgraded from Incremental)

Client: "computer-fd" 5.2.6 (21Feb12) x86_64-pc-linux-gnu,debian,7.0

FileSet: "Full Set" 2014-08-09 18:04:12

Pool: "File" (From Job resource)

Catalog: "MyCatalog" (From Client resource)

Storage: "File" (From Job resource)

Scheduled time: 09-Aug-2014 18:04:09

Start time: 09-Aug-2014 18:04:14

End time: 09-Aug-2014 18:04:15

Elapsed time: 1 sec

Priority: 10

FD Files Written: 2

SD Files Written: 2

FD Bytes Written: 133,089,280 (133.0 MB)

SD Bytes Written: 133,089,490 (133.0 MB)

Rate: 133089.3 KB/s

Software Compression: None

VSS: no

Encryption: no

Accurate: no

Volume name(s): Clients-0001

Volume Session Id: 1

Volume Session Time: 1407513760

Last Volume Bytes: 133,188,801 (133.1 MB)

Non-fatal FD errors: 0

SD Errors: 0

FD termination status: OK

SD termination status: OK

Termination: Backup OK

09-Aug 18:04 baculaserver-dir JobId 1: Begin pruning Jobs older than 6 months .

09-Aug 18:04 baculaserver-dir JobId 1: No Jobs found to prune.

09-Aug 18:04 baculaserver-dir JobId 1: Begin pruning Files.

09-Aug 18:04 baculaserver-dir JobId 1: No Files found to prune.

09-Aug 18:04 baculaserver-dir JobId 1: End auto prune.

[/plain]

Once the backup has been completed, we can execute the "status storage" command in bconsole to find out what is stored on the storage.

[plain]

*status storage

Running Jobs:

No Jobs running.

====

Jobs waiting to reserve a drive:
====

Terminated Jobs:

JobId Level Files Bytes Status Finished Name

===================================================================

1 Full 2 133.1 M OK 10-Aug-14 13:03 BackupClient

====

[/plain]

Data Restoration

Restoring the data is equally important as backing up the data: the backup of data is done for the sole reason to be restored one day in case of an emergency. If we don't have a clear process how to restore the data when needed, then we might as well not have a backup solution in place. Whenever introducing a backup solution to the network, we also have to provide detailed instructions for restoring the data.

To restore the data in Bacula, we can enter bconsole and issue the restore command, which will give us a list of options to choose from as presented below. Most often we'll use option 5 to restore the most recent backup for a client, but we might restore just a specific file from a backup or select a backup from a specific time, etc.

[plain]
*restore

First you select one or more JobIds that contain files

to be restored. You will be presented several methods

of specifying the JobIds. Then you will be allowed to

select which files from those JobIds are to be restored.

To select the JobIds, you have the following choices:

1: List last 20 Jobs run

2: List Jobs where a given File is saved

3: Enter list of comma separated JobIds to select

4: Enter SQL list command

5: Select the most recent backup for a client

6: Select backup for a client before a specified time

7: Enter a list of files to restore

8: Enter a list of files to restore before a specified time

9: Find the JobIds of the most recent backup for a client

10: Find the JobIds for a backup for a client before a specified time

11: Enter a list of directories to restore for found JobIds

12: Select full restore to a specified Job date

13: Cancel

Select item: (1-13):

[/plain]

After we choose option 5 to select the most recent backup for a client, Bacula will ask us to choose a client which we would like to work on. Then it will build a directory tree for the JobIds that were used to backup the data. In the file selection mode, we have to select the files we would like to restore by using the add/remove commands. The commands below present a way to restore the files from the Bacula volume. Upon entering the file selection mode, we executed the pwd command to print the name of the current working directory. Then we issued the ls command to print the files in the current directory, upon which we changed the working directory to /backup. After that we selected the file mysqldump-31_07_2014.sql for restoration by using the add command and finished the restoration.

[plain]

$ pwd

cwd is: /

$ ls

*backup

$ cd backup

$ ls

*mysqldump-31_07_2014.sql

$ add mysqldump-31_07_2014.sql

1 file marked.

$ done

[/plain]

The done command will print the information about the restore job: the name of the fileset, the backup/restore client, the priority and other options will be given. Also the location of the restored files will be saved to a directory specified with the Where option.

[plain]

$ done

Bootstrap records written to /var/lib/bacula/backup-dir.restore.1.bsr

The job will require the following

Volume(s) Storage(s) SD Device(s)

===========================================================================

Clients-0001 File FileStorage

Volumes marked with "*" are online.

1 file selected to be restored.

Run Restore job

JobName: RestoreFiles

Bootstrap: /var/lib/bacula/backup-dir.restore.1.bsr

Where: /restore/

Replace: always

FileSet: “Full Set”

Backup Client: computer-fd

Restore Client: computer-fd

Storage: File

When: 2014-08-10 13:29:50

Catalog: MyCatalog

Priority: 10

Plugin Options: *None*

OK to run? (yes/mod/no): yes

[/plain]

The Where option directly corresponds with the Job definition in bacula-dir.conf as presented below.

[plain]

Job {

Name = "RestoreFiles"

Type = Restore

Client=computer-fd

FileSet="Full Set"

Storage = File

Pool = Default

Messages = Standard

Where = /restore

}

[/plain]

The successful restoration will result in selected files being written into the /restore/ directory, which we can further use to restore the backup. In the example above, we've seen how a .sql database was restored, which still needs to be applied to the database by using the appropriate command. Often we're restoring important files, like files uploaded to an internal document system, a web application platform, etc. In such cases, the files are already there and don't need to be imported into an application to make them accessible from it, but merely moved to the right directory as configured.

Database Backups

Often there are many applications running in internal network which require some kind of database, including Bacula itself. Bacula keeps catalog metadata in the Sqlite database by default, but it's often the case that Mysql/Postgresql/Oracle database is used. With Sqlite databases, we only have to backup a single file and thus no further actions are needed, but it's usually the case that databases are contained on a remote central location to which applications connect. This makes them easy to maintain and simple to backup. In this tutorial we'll concentrate on the Mysql database and configure Bacula to automatically backup all databases contained on a separate database server. Usually, a crontab entry is created to backup the database via mysqldump tool at specified intervals, like once a week.

I've been using the following script, which I've placed under /etc/cron.weekly/ to dump the webapp database to a file on a weekly basis.

[plain]
#!/bin/bash

/usr/bin/mysqldump -u root -ppassword webapp 2>/var/log/mysqldump.log > /backup/webapp-$(date +"%d_%m_%Y").sql
[/plain]

Despite the crontab being a good solution, it still requires a log-in to the remote machine, creation of the backup script, and configuring the crontab in order to dump the database to a file. Above we only dumped one database to a file, but we can copy-paste the mysqldump command to dump as many databases as we want. To let Bacula run the backup, we have to add the ClientRunBeforeJob into the Job – in bacula-dir.conf configuration file. An example of this Job can be seen below, where Bacula will run /root/mysqlbackup script prior to backing up the data.

[plain]

Job {

Name = "MysqlClient"

JobDefs = "DefaultJob"

Client = mysqlclient-fd

ClientRunBeforeJob= "/root/mysqlbackup"

}

[/plain]

By letting Bacula run the backup script, we're effectively not relying on crontab to run the jobs for us. We're also running the backup job all at the same time: we don't have to think about running the crontab job before Bacula or calculating anything. Let's imagine that we setup crontab to run every day at a specific time and Bacula is configured to run every week; what would happen to 7 versions of backup files (one for each day in the week)? It depends on the backup script – if the backup script overwrites the previous version of the backup file, we would only backup one of the files in a whole week. This can happen when changing the Bacula scheduling mechanism and forgetting about crontabs in Linux distributions. The best way to avoid the confusion is to let Bacula run the script by itself prior to backing up the data. By using such methodology, we're eliminating the risk of mistakes and keeping Bacula in control of every aspect of the backup process, which indicates good design.

It's also imperative to understand that ClientRunBeforeJob runs the script and waits for it to finish before backing up the data. Therefore, we must differentiate between a synchronous and asynchronous script. When an asynchronous script is run, the script will be executed in the background and control will be immediately returned to us. Contrary to the asynchronous script, a synchronous script runs in the foreground, thus returning control to the user once the script has finished processing. This is the script we must use, since ClientRunBeforeJob will wait for the script to give control back to Bacula before proceeding. If asynchronous script is used, the control will be given back to Bacula immediately and Bacula will backup an old copy of the backed up files, which is undesirable. To circumvent such behavior, we should use a synchronous script, which waits for the script to finish before giving control back to Bacula.

References

[1] Solid-state drive, https://en.wikipedia.org/wiki/Solid-state_drive.

[2] Tape drive https://en.wikipedia.org/wiki/Tape_drive.

[3] List of backup software https://en.wikipedia.org/wiki/List_of_backup_software.

[4] Bacula-Web, http://www.bacula-web.org/.

[5] The Bootstrap File, http://www.bacula.org/5.2.x-manuals/en/main/main/Bootstrap_File.html.

[6] ESXi 5.1: Using Raw Device Mappings (RDM) on an HP Microserver,
http://forza-it.co.uk/esxi-5-1-using-raw-device-mappings-rdm-on-an-hp-microserver/.

[7] Bacula Installation and Configuration Guide, https://access.redhat.com/site/sites/default/files/attachments/install_1.pdf.

[8] Overview on modifying the Synology Server, bootstrap, ipkg etc, .

[9] Data Encryption, http://www.bacula.org/5.2.x-manuals/en/main/main/Data_Encryption.html.

FREE role-guided training plans

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

[10] Messages Resource, http://www.bacula.org/5.2.x-manuals/en/main/main/Messages_Resource.html.

Dejan Lukan
Dejan Lukan

Dejan Lukan is a security researcher for InfoSec Institute and penetration tester from Slovenia. He is very interested in finding new bugs in real world software products with source code analysis, fuzzing and reverse engineering. He also has a great passion for developing his own simple scripts for security related problems and learning about new hacking techniques. He knows a great deal about programming languages, as he can write in couple of dozen of them. His passion is also Antivirus bypassing techniques, malware research and operating systems, mainly Linux, Windows and BSD. He also has his own blog available here: http://www.proteansec.com/.