Veeam Backup & Recovery - Change Block Tracking Reset

Change Block Tracking (CBT) is a feature that allows tracking of changed disk sectors. Tracking for virtual disks is done in the virtualization layer. CBT is exposed through vSphere APIs for Data Protection (VADP) to 3rd party applications. 

VBR uses this feature to track changes between incremental backups and make those backups faster. Instead of reading the whole vmdk, it will ask and receive only the changed blocks from the last incremental backup. How do you know CBT is enabled and used? 

From VMware point of view, you will see for each vmdk a file with the  vmdk_name-ctk.vmdk

From Veeam point of view, you will see in the backup job statistics next to the disk details CBT. Let's look at what actually happens in the backup job.

CBT on, active full backup, read 20 GB

CBT on, incremental backup, read 11 MB

CBT off, incremental backup, read 20 GB

This is CBT in action. The first picture is an active full, so the whole disk has been read. The second one is an incremental where only changes from last backup have been read. When CBT was disabled, the second incremental also read 20GB.  Now extrapolate this to hundreds of VMs to understand the importance of CBT: less data read means not only a shorter backup window, but also less load on the production storage. 

Luckily, CBT is enabled by default for all newly created backup jobs. It can be found in the vSphere integration tab, in Storage > Advanced. 

CBT is not supported for physical mode RDMs and if a VM has snapshots when it's activated the first time. Sometimes CBT can get corrupted and the only way to solve it is to reset CBT. This is not easily done A new feature in VBR v10 allows to automatically reset CBT on all VMs when after an active full backup is executed. Remembering how many times I heard support guys advising a CBT reset, I think this is a cool feature to add. The CBT reset action is caught in the backup stats window

The downside is that the active full backup will take a bit longer, but you will be protected against potential CBT corruptions.

Veeam Backup and Replication v10 - General Options

I finally got v10 up and running in my lab. I started randomly clicking around the interface and noticed a couple of changes in the General Options menu. I think it's one of the most ignored part of Veeam Backup & Replication while it has a lot of important settings. So I decided to pay it the proper respect and write a post about it. Just a heads up: if you are interested in VBR notifications, alerts, audits, reports, then this it where you should look.

I/O Control
By default it is disabled. However, if you are running in a production environment and backups may overlap with any business hours, this should be enabled. It ensure backups are not going to impact the production datastores by imposing latency threshold and throttling down backup jobs. The thresholds should be set accordingly with the underlying physical storage and current production latency.

Notifications
This tab is enabled by default. Besides license and update related notifications, there are 3 settings regarding notifications on space usage:

• warn when backup storage space is less than 10% 
• warn when production storage is less than 10% - this will cause your backup jobs to finish with a warning 
• skip VM processing when free disk space is below 5% - in this case a backup job will fail for the VMs on those datastores 

As you can see these notifications are not only simple notifications, but also determine the result of backup jobs. You can tweak the default settings, but it is highly recommended to leave enough free space on datastores for snapshots to grow during a backup.Otherwise you may run into an out of space situation.

Security

This is used to change VBR SSL certificate (default is self signed), set up trust for Linux discovered hosts and it also has a new setting that appeared in v10: audit logs location. 

The audit logs are used to track file level restore activities. By changing the location to a centralized repository or even a WORM (write once read many) device, you can ensure a proper audit trail and archival. In case you run a file level restore, the following information will be logged in a CSV format file: Time, User, SID, Operation, Result, Object.

29.03.2020 18:35:51Z, MYLAB\Administrator, S-1-5-21-AAAAAAA-XXXXXXX-NNNNNNN-500, Restore, Success, C:\Users\Administrator.MYLAB\Desktop\order_form_7393 VMUG Feb 2019.pdf

However there is a workaround - you can open a file directly from backup in explorer and read its content. This activity will not be logged. 

E-mail Settings and SMNP Settings

These two tabs are dedicated to reporting over e-mail and alerting over SNMP. When enabling e-mail reporting you may choose the level of granularity. By default you will receive e-mails for any job status: success, warning or fail.

For SNMP you may configure up to 5 different receivers, as well as their community string and listening port.

Session history

Session history configure the number of session to display and the retention period. Up to 9.5U4 the default retention period was 53 weeks. With v10 this changed to 13 weeks. 

From an operational point of view, the most important settings are related to I/O control, notifications and e-mail/SNMP while from auditing point of view, the audit log and the history retention period.