Moving Backups to Hardened Linux Repositories

It's not enough to have a backup of your data. You need to make sure that you will be able to recover from that backup when the time comes. And one of the best ways to make sure you can do it, is to make protect your backups from being modified intentionally or unintentionally. 

In Veeam Backup & Replication, a hardened repository is using a Linux server to provide immutability for your backups. The feature was first released in version 11. Let's see what makes the hardened repository special, how it protects your backups from changes and how easy is to actually start using it 

Immutable file attribute

Linux file system allows setting special attributes to its files. One of these attributes is immutable attribute. As long as it is set on a file, that file cannot be modified by any user, not even root. More, root user is the only user that can actually set and unset the immutable attribute on a specific file. You can do it using lsattr and chattr commands in Linux as seen in the below screenshot:

Veeam hardened repo uses exactly the same mechanism of making backup files immutable.  

Isolate Linux processes 

To run a successful repository, Veeam needs several functionalities: to receive data from proxies, to open and close firewall ports, to set and unset immutability as per retention policy. In order to harden the repository, Veeam implements these functionalities as separate Linux processes.

The processes that sets and unsets immutable attribute on the backup files is called veeamimmureposvc and it needs to run with root privileges, as root user is the only user that can modify immutable attribute.

veeamtransport --run-service is the Data Mover service performing data receiving, processing and storing.  Because it is a service exposed on the network, it is running under a standard Linux user. In case of a  breach, the service will give access only to a standard user with limited privileges. The Linux user under which this service runs must not be allowed to elevate its privileges. 

A third process takes care of dynamically opening and closing  firewall ports: veeamtransport --run-environmentsvc and this one is also running with elevated privileges. 

The following screen shot shows the three main services that are part of a hardened repository. 

Single use credentials

Another layer of protection is added through the way the credentials are handled within the backup server.

To add the Linux repo to the backup server you need to specify Linux credentials. These credentials are only used during the initial configuration process and they are never stored in backup server's credential manager. Temporary privilege elevation may be needed during the repository configuration  for deployment and installation of Veeam processes. After the configuration process finishes, all elevated privileges must be revoked from the user. 

 

Additional repository features - fast clone

This one is not a security related feature, but it comes in as a great add on to the hardened repository.

In case you formatted your file system with XFS file system and you have a supported Linux distribution (see this user guide page for more details), Veeam will use fast clone to reduce used disk space on the repository and increase the speed of synthetic backups and transformations. Fast clone works by referencing existing data blocks on the repository instead of copying the data blocks between files. 

Using the hardened repository

For new backup jobs, just point them to your hardened repository. In case you have existing backups then you need to migrate those to your new repo. With v12 comes a new feature that allows to move any backup from an existing repository to another one. Simply select your backup, right click it and you will see that now you can "move backup"

Let's look at moving backups from a Windows NTFS repository to our hardened Linux repo. We start with an empty repository configured with a service account called veeambackup

The first backup chain is for an unencrypted backup job. The backup job is configured to use a standard Windows repository. There are 2 full restore points in the backup chain. Each restore point is 960 MB and the total size on disk is 1.87 GB. We use "move backup" to send the the backup chain to the hardened repository:

Once the move processes finished, the backup job has been updated to point to the new repository. Let's check what happened on the Linux hardened repo. 

Find the backup chain in our repo:

Check the immutability flag:

The restore points are set as immutable. The metadata file is not since this file is modified during each backup operation. Trying to delete any of the restore points will fail:
 

We can also check that XFS fast clone is working by looking at the used space on the repo which is less that the sum of the 2 full backups:

In this post we've looked at the features of hardened repository and how they work. To implement a hardened repository in your environment follow the steps in the user guide  

Backup vSAN 7 File Share with Veeam Backup & Replication 10

This is a blog post about two of the new features that were released this year in vSAN 7 and in Veeam Backup & Replication v10:

• vSAN File Service
• Veeam NAS Backup 

As you've already guessed, we're going to create a file share on vSAN, put files on it and backup it up with Veeam Backup & Replication (VBR) using the native NAS support.

At a high level the environment looks as following:

A 2 node vSAN 7 cluster runs in a nested environment. Each node has 4 vCPU, 32 GB of RAM, 20 GB Cache SSD and 200 GB Data SSD. A witness appliance has been also deployed. VBR v10 is configured with the minimum hardware: 2 vCPU and 8GB. Both proxy and repository roles are installed on the same VM with backup server, as well as the embedded MSSQL Express DB. Network connectivity between vSAN and VBR is 1 Gbit.

The configuration of the vSAN cluster or the installation of VBR are not in scope of this blog as there are many resources out there. I've used William Lam's nested library for the vSAN nodes and downloaded the witness appliance directly from VMware site.

The prerequisites for the following steps are: a running vSAN cluster and a VBR installation. You will also need 2 IP's reserved for File Server virutal appliances (one for each node) and a working DNS server with records added for the 2 IP's

Part 1 - vSAN File Service

vSAN File Service provides file shares on top of vSAN storage. It supports NFS 3 an NFS 4 protocols. It comprises of vSAN Distributed File System (vDFS) and it is integrated with vSAN Storage Policy Based Management.

Enable vSAN File Services

At the cluster level go to Configure - vSAN - Services and press enable File Service

Select whether the File Server agent OVF appliance will be download automatically or it will be manually uploaded:

Give a namespace for file share, type in DNS server address and domain

Select a portgroup, type the netmask and default gateway IP

Type in the IP addresses for each of the nodes

Review the configuration and press finish. Installation process will start deploying the agents on each of the vSAN nodes.

Once it is finished you will 2 new components deployed on the vSAN cluster: the File Service Agents. These are virtual appliances running Photon OS 1 and Docker and they act as NFS file servers.

Create NFS file share

It's time to create the share and put some files to it. To create the share Configure - vSAN - File Services Share. Enter the name of the share, storage policy, soft and hard quota limits.

You can also assign labels to the file share. Quotas can be changed at a later time. Next define access control: what IP addresses have access to this share, what type of access and if to protect the share with root squash.

Lastly review the settings and create the share. Once created you will see it vSAN - File Services Share:

Now the file share is ready to be used. Because in part 2 we will use Veeam's NAS backup, we've added a small load of files on the share using a linux VM. First get the share path by selecting the file share and pressing copy on URL:

Next ssh to your favorite Linux VM and mount the file share:

To create random files with random size, we've used a simple script found on here and which was slightly adapted.

for n in {1..500}; do
    dd if=/dev/urandom of=file$( printf %03d "$n" ).bin bs=100 count=$(( RANDOM + 1024 ))
done

The workload tested was:
- 200 files ranging  between a few hundreds of KB and a few MB
- 50 files ranging from a few MB and few tens of M

The script can be adapted to create larger workloads with thousands of files. Since the whole lab is nested, performance testing was not in scope.

Part 2 - Veeam NAS Backup 

The architecture for NAS backup requires at a minimum a file share (in our case NFS from vSAN), file proxy, cache repository, backup repository and a Veeam Backup Server.

Additionally, secondary and archive repositories can be added in infrastructure. In our lab infrastructure all components are deployed on a single VM - the backup server.

File proxy - component that acts as the data mover transporting data from the source (file share) to the backup proxy. It is used in backup and restore activities.

Cache repository - is a location to store temporary metadata. This metadata is used to reduce the load on the file share during incremental backups.

Backup repository - main location of the backups

Add File Share 
In VBR Console go to Inventory - Add File Share

Select NFS share and specify the path to the share

Select the File proxy, cache repository and backup speed

Review the summary and finish the configuration.

Having the file share added, we will add it to a backup job. Right click it and add it to a new backup job (or use an existing one).

Select the file share to backup

Select destination repository, backup policy and archive repository if any.

Select a secondary target if required (backup copy for short term)

Enable the job to run automatically

Review the settings and run the job.

The initial execution backups up all 250 files on the share. For the incremental, we've removed and recreated the first 50 files on the share:

for n in {1..50}; do 
  rm -f file$( printf %03d "$n" ).bin
  dd if=/dev/urandom of=file$( printf %03d "$n" ).bin bs=100 count=$(( RANDOM + 1024 ))
done

As you can see the cache repository is pretty effective in determining which files to be backed up:

Troubleshoot WordPress Connectivity Error to Database

I am an irregular user of Linux. For some demo scenario, I needed to install WordPress. All good and pretty straight forward. I've deployed two CentOS VMs, installed MySQL on one (DB layer) and Apache and PHP on the the other (front end). I've configured MySQL, created DB, added users. I've also tested connectivity from front end to the DB layer. I've also installed WordPress, but when connecting to wp-admin site, I got a connection error message to the DB.

Weird, since I could actually connect to the DB using mysql client and the same credentials.

So it is not a connection error or a problem with the DB.

Since mysql client from front end works, I've tested php connectivity. For this I've put together the following small script in a file called conn.php and put the file in /var/www/html/wordpress/

<?php
try {
    $dbh = new PDO('mysql:host=IP_ADDR;port:3306;dbname=WP_DB_NAME','USER', 'PASS');
    print "Connected succefully";
}
catch (PDOException $e) {
    print "Error!: " . $e->getMessage() . "<br/>";
    die();
}
?>

Next I opened the browser and executed the php script which resulted in an expected connection error:

At this point I just run the same script, but this time interactively from php command line:

And I got a less expected connection successful message. By default SELinux is enabled and it will not allow httpd to talk to the mysql service. A quick test is to temporarily disable SELinux:

setenforce 0

and running again the script will connect successfully: 

So the main culprit is SELinux. Going forward there are 2 options: disable SELinux permanently (which is acceptable in demo labs) or enable Apache to connect to other service by changing the SELinux policy (which will be another post)

Enable RDP in SLES 12

Haven't been working with SuSE Linux for a while and when this new project came along I got my hands on SLES 12 SP3.

I've created an account on suse.com, downloaded the ISO, created the VM and installed from scratch. It went pretty smoothly. The suse account gives a 60 day trial activation key which will allow to update the server. After the 60 days, no more update, but it's just for a lab setup.

The one thing I need from the server is to provide terminal services. So I installed xrdp from using Software Manager in Yast 2

Next, enable xrdp service and start it:

sudo systemctl enable xrdp
sudo systemctl start xrdp

Last thing to do is to configure the firewall. It is enabled by default and will not allow connection to the server. Check that the xrdp service definition exists. If not create a file that looks like this:

Configure the firewall to accept xrdp (and if you are at it, add ssh also). Open the file /etc/sysconfig/SuSEfirewall2 and change the following lines to include your services:

FW_CONFIGURATIONS_EXT="sshd xrdp"
FW_CONFIGURATIONS_DMZ="sshd xrdp"
FW_CONFIGURATIONS_INT="sshd xrdp"

Save the changes and restart the firewall service:

sudo systemctl restart SuSEfirewall2

Now, if all went good open your preferred RPD client and give it a try:

Linux - adding a user to multiple servers

When I am not working on VMWare infrastructure, I face administration tasks at OS level. Since I am writing the scripts, I will try to post some on the blog from time to time. Following, a simple way of adding a new user to multiple servers (I am not using dsh, rather a for loop):

for i in `cat server_list`; do ssh $i  "echo 'john:johnpass:30000:30000::/home/john:/bin/bash' | /usr/sbin/newusers;

sed -i '/AllowUsers/s/root/root\ john/' /etc/ssh/sshd_config;

sed -i '/AllowGroups/s/root/root\ john/' /etc/ssh/sshd_config;

service sshd reload;

sed -i -e '/root\tALL=(ALL)/a john\tALL=(ALL)\tALL' /etc/sudoers;

"; done

It adds user information (username, password,UID,GID,home,shell) using newuser command, then it adds user and group in sshd_config (if it is the case). The last sed adds the new user to sudoers file right after it matches a string with tab (\t) in it.

Rsync cyrus volumes from EMC to NetAPP

A couple of weeks ago I was faced with the following problem: clean up the EMC filer that hosts one of our client`s cyrus e-mail mailboxes.Putting it all on paper I got to the following:

- there were more than 3 million files and folders

- the only available protocol is NFS

- the bandwidth is 1 Gbit

- source filer is EMC, destination is NetApp

The chosen option was a copy-paste operation from one filer to another with service interruption to avoid synchronization problems. One question was raised: how long it will take to do it and what tool to use? 

After some parking lot meetings and discussions, the shortlist for tools included cp, rsync and cpio. First thing first, speed tests. I have generated a workload of 50000 files having the following composition: 40% @ 1 KB, 20% @ 2 KB, 20% @ 3 KB and 20% @ 10 KB using for and dd:

### first 20000 files 1K, then increase seq and bs 

for i in $(seq 1 1 20000); do dd if=/dev/zero of=/tmp/FILER/vol99/$i count=1 bs=1024 2>/dev/null; done

Next, the tests, using time command:

### cp

time  cp -a vol99/ /tmp/DEST/vol99

### rsync

time rsync -a vol99/ /tmp/DEST/vol99/ 

### cpio 

time find . -print -depth | cpio -padmuv /tmp/DEST/vol99/

cp -a does the same thing as cp -dpR - preserves links, permissions and is recursive. rsync -a does the same as rsync -rlptgoD and I was mainly interested to ensure recursive transfer, preservation of permissions, modification times, groups and owners. cpio -padmu ensures creation of directories where needed, preservation of mtime.

The fastest was cp and it was considered the baseline. rsync was 7% slower, while cpio was 46% slower (because of the big find piped to it). Since the obvious choice was cp, it was decided to use rsync (mostly because of the possibility to continue interrupted transfers and to do check sums - which was never used). The speed tests also allowed to approximate transfer times of the real data at around 100 minutes. This way I could go ahead with the formalities on the client side.

When rsync was tested against real data (around 400K files and directories), it generated a bit of a surprise. It started to build up its file list, it took more than 15 minutes to do it and it started a very slow transfer process - after 1 hour it did not finish one volume. And we had 5.  The main cause was the partition structure used by cyrus imap for mail store and the big number of files. To get an idea: vol1 has 20 imap partitions: cyrus1 to cyrus20. Each cyrusN partition has the the mailbox structure from A to Z. The first rsync test was:

nohup rsync -a /SRC/vol1/ /DEST/vol1/ &

and this got stuck into creating the file list and transferring very slow. The idea was to have fast transfer, so for the second test rsync was run in parallel for each cyrus partition from 1 to 20:

nohup rsync -a /SRC/vol1/cyrus1/ /DEST/vol1/cyrus1/ &

nohup rsync -a /SRC/vol1/cyrus2/ /DEST/vol1/cyrus2/ &

...

nohup rsync -a /SRC/vol1/cyrus20/ /DEST/vol1/cyrus20/ &

This time it worked just as expected. Actually, during the operation we managed to increase the CPU on a NetApp 6280 by 10%-15% using parallel rsync.

One more thing to be said: rsync syntax is a bit tricky. If the slash at the end of source directory is omitted, then it will create the directory itself in the destination and you will end up with a shifted folder structure: /DEST/vol99/cyrus20/cyrus20/.