Monitoring Azure Blob Storage - Daily Change Rate

Lately more and more projects are looking at backing up object storage. When starting a backup project one of the first questions you need to answer is what is the size of the protected workloads and what is the daily change rate. If the total size is usually monitored using different tools or it can be extracted pretty easily, the daily change rate proves to be more challenging.

In this article, we’ll guide you step-by-step through monitoring daily change rates in Azure Blob Storage using diagnostic settings, the Log Analytics workspace, and Kusto Query Language (KQL). If you’re not looking at backing up object storage, you can still read the article since real time monitoring of object storage can be applied in other situations.

Why Daily Change Rates?

The daily change rates of the source object storage will directly impact required space on the backup repository, and it will also have a direct influence on the sizing of components such as proxies and cache repositories.

Why Log Analytics?

Log Analytics provides near real time tracking of blob operations as it ingests Azure Monitor logs (storage diagnostics logs), while KQL offers powerful filtering and aggregation functionalities. It’s easy to setup and to extract data without external processing tools. There is a downside, Log Analytics may incur additional costs as detailed in here.

The proposed solution is intended for a short period of time, such as 7 days, to extract the daily change rates for the protected blobs.

Step 1: Configuring a Log Analytics Workspace

Before you can send diagnostic logs to Log Analytics, ensure that you have a Log Analytics workspace set up:

·         In the Azure Portal, navigate to Log Analytics workspaces using the global search bar.

·         Click + Create to start a new workspace, or select an existing one if you already have one in your desired subscription and region.

·         For a new workspace, provide a Name, choose a Subscription, select a Resource Group (or create one), and pick a Region that aligns with your storage account for optimal performance.

·         Review the settings and click Review + Create, then Create to provision the workspace.

Once your Log Analytics workspace is ready, you can proceed with connecting your Storage account’s diagnostic settings to this workspace.

For compliance meets, performance and backup sizing needs, use regional workspaces to determine requirements in each region.

Step 2: Enable Diagnostic Settings for Azure Blob Storage

Azure Storage accounts provide diagnostic logs that capture activity at both the account and blob level, including read, write, and delete operations. To monitor daily change rates, you need to ensure that diagnostic logging is enabled and that logs are routed to a Log Analytics workspace.

Configuring Diagnostic Settings

·         Navigate to your Azure Storage account in the Azure Portal.

·         In the left pane, click on Monitoring > Diagnostic settings.

·         Click on the blob

·         Click + Add diagnostic setting.

·         Provide a name for your setting (e.g., BlobActivityLogs).

·         Under Log, select categories to monitor (Storage Write).

·         Choose Send to Log Analytics workspace and select your target workspace.

·         Click Save to apply your configuration.

Once enabled, the diagnostic settings will begin sending blob activity logs to the designated Log Analytics workspace.

Step 3: Understanding the Logged Data

After a short period (5-10 minutes), data will start accumulating in your Log Analytics workspace. The logs sent from Azure Blob Storage include a range of operation types, such as PutBlob (create), SetBlobProperties (modify), among others.

Typically, these logs land in the StorageBlobLogs table. Key columns to pay attention to include:

·         TimeGenerated: The timestamp of the action.

·         OperationName: The specific blob operation (e.g., PutBlob).

Step 4: Querying the Data with Kusto Query Language (KQL)

KQL is a powerful query language designed for ad-hoc analysis of structured, semi-structured, and unstructured data. With KQL, you can aggregate and filter Blob Storage changes to calculate daily change rates.

Here’s a basic KQL query to get data for the last 7 days and present changes on a daily basis:

StorageBlobLogs

| where TimeGenerated > ago(7d)

| where OperationName in ("PutBlob", "PutBlock", "PutBlockList", "SetBlobProperties", "SetBlobMetadata")

| summarize TotalChangedGB = sum(todouble(ContentLengthHeader)) / 1073741824

          by bin(TimeGenerated, 1d)

| sort by TimeGenerated asc

 

This query looks only at new or modify operations:

• PutBlob: Uploads a new blob or overwrites an existing one.
• PutBlock: Uploads a block (part of a blob) — useful for tracking new blob uploads in block blob scenarios.
• PutBlockList: Commits the uploaded blocks into a blob.
• SetBlobProperties: Updates system properties of a blob (e.g., content type, cache control). These are metadata-level changes, not content changes.
• SetBlobMetadata: Updates user-defined metadata associated with a blob. Again, this doesn’t alter the blob’s content but changes its descriptive attributes.

 

To run the query, go to your Log Analytics workspace > Logs > New Query. Paste the query from above, select the time range for the last 7 days, and press Run:

The output will give you the changed size in GB and that you can use to estimate required resources for blob storage backups. It shows only the last 3 days because the logs have been sent only in the last 3 days to the workspace.

You can further expand the way you are using the data by visualizing it in Azure Dashboards or creating alerts. However for the purpose of deploying the backup infrastructure advanced data manipulation is not needed.

Best Practices & Considerations

·         Retention: Ensure your Log Analytics workspace retains data for as long as you need for analysis, auditing, or compliance. Remember longer retention incurs larger costs

·         Cost: Frequent and verbose logging can incur costs. Scope logs and queries to only what you require and check your costs.

·         Security: Protect access to diagnostic logs as they may contain sensitive activity information.

·         Implementation: Start on per region basis. Select the most relevant storage accounts in that region that could be used for a generalization.

Conclusion

Monitoring Azure Blob daily change rates using diagnostic settings, Log Analytics, and KQL can be easily setup and doesn’t require external processing of data although it may incur additional costs. Other possibilities exist such as Blob Inventory, but in this case additional data processing will be needed.

Veeam Backup for AWS: Comprehensive Cloud Data Protection

In today's cloud-dependent world, data protection is essential for maintaining business continuity. Veeam Backup for AWS (VBA) offers an AWS-native, highly adaptable solution designed to protect, manage, and recover data within AWS environments. Its main purpose is to help organizations address the unique data protection needs of AWS workloads, ensuring that cloud data remains resilient against threats like accidental deletion, cyberattacks, or service interruptions.

Key Components of Veeam Backup for AWS

1. Automated Backup and Recovery: Veeam allows for fully automated backup processes, supporting Amazon EC2, RDS, Dynamo DB, Redshift, EFS, FSx and VPC. With policies and schedules, users can customize backups to fit business needs and ensure their critical data is consistently protected.

2. Cost Optimization: Veeam uses Amazon S3 and its various storage classes, such as Glacier and Glacier Deep Archive, to optimize storage costs. Users can automatically tier their data to lower-cost storage options, making cloud backups more affordable without sacrificing accessibility.

3. Immutability and Security: Leveraging Amazon S3 Object Lock, Veeam ensures that backups remain immutable, providing a strong defense against ransomware and other cyber threats. This feature prevents any changes or deletions to stored data within a specified timeframe, securing it from unauthorized access or malicious attacks.

4. Cross-Region and Cross-Account Recovery: In case of an outage or disaster, Veeam enables cross-region and cross-account recovery, allowing users to restore data quickly and securely across different AWS accounts or regions, thereby meeting stringent recovery objectives.

5. User-Friendly Interface and Self-Service: The solution includes a streamlined interface that simplifies backup setup and monitoring. Additionally, self-service recovery options allow users to restore their data with minimal intervention, enabling faster response times in critical situations.

Starting with version 7.0, Veeam Backup for AWS is part of the Veeam Backup & Replication (VBR) solution. AWS Plug-in for Veeam Backup & Replication extends the Veeam Backup & Replication functionality and allows you to add backup appliances to Veeam Backup & Replication. The entire lifecycle of VBA is managed from VBR through AWS Plug-in. 

Deployment, update and management of VBA is done from VBR console. Currently you can still deploy VBA from AWS marketplace, connect it to VBR and upgrade it to the latest version. However this process is deprecated and only VBR console should be used to manage VBA. One or multiple VBA appliances can be managed from the same VBR server. 

Additionally, Veeam ONE can offer enhanced monitoring and reporting capabilities for VBA by collecting date about protected AWS resources. 

By combining these components, Veeam Backup for AWS provides an end-to-end backup and disaster recovery solution tailored for AWS cloud environments, balancing security, cost, and ease of use. 

In the following posts we will take a deeper look at Veeam Backup for AWS architecture and operations.

Integrating Veeam Backup Infrastructure with Security Information and Event Management Solution

Security Information and Event Management (SIEM) systems provide comprehensive security monitoring, threat detection, and incident response capabilities. The main features a SIEM should provide are:

• Data Collection from a wide range of sources, including log files, network traffic, system events and security alerts.
• Normalization and Correlation where collected data is standardized, stored and events are correlated.
• Threat Detection and Analysis using predefined rules, statistical analysis, machine learning algorithms, and threat intelligence feeds.
• Alerting and Reporting to notify security personnel in real-time
• Incident Response and Forensics tools for investigating security incidents, conducting forensic analysis

All of the above make SIEM complex systems and also not cheap. However this does not minimize the importance of having SIEM deployed in your enterprise environment as a proactive line of defense against attacks that can compromise services availability and integrity.

Data protection solutions play another crucial role in making sure your environment is safe and can recover in case of any incident ranging from unintentional deletion to a sophisticated malware attack. It makes sense to integrate your backup infrastructure into your SIEM. Veeam offers the possibility to send events and alerts to a SIEM solution using syslog protocol. 

Install Graylog

Let's look at first at the SIEM solution. For our lab environment, we chose Graylog Open as it offers  a basic and free SIEM solution that can run on top of Ubuntu. We are using an Ubuntu 22.04 template with a static IP address that is resolvable via DNS. To install Graylog Open we have followed the instructions from this link. Please note that MongoDB and OpenSearch (or ElastichSearch) are required on the Graylog server. We will not repeat the steps since it does not make any sense to duplicate the content. 

Configure Graylog for Veeam data ingestion

Once Graylog is installed, login to the admin interface using the http_bind_address configured in /etc/graylog/server/server.conf. For our lab it would be http://graylog_ip_address:9000, for example. 

Go to System > Inputs. Select Syslog TCP  and press Launch New Input

Type in a name, add the bind address where to listen for incoming connections and makes sure to use a TCP port above 1024. By default Graylog runs under a normal user and cannot bind to ports under 1024. You may leave the rest of the settings as default. Press Launch Input and make sure it shows as running 

When the input is running, Graylog is ready to receive messages from Veeam infrastructure. 

Configure Veeam Backup & Replication syslog integration

First, open the VBR console and go to Main menu > Options

In the new window got to Event Forwarding and under Syslog servers press Add. Configure the Graylog input: IP address or FQDN, port and protocol, Please note that only one syslog server can be configured 

Press OK to save the syslog configuration and save it to VBR. You are now ready to test. The simplest way is to enable/disable Malware Detection configuration in VBR console. On main menu, press Malware Detection and in the new window make a change to your current configuration, for example deselect Enable file system activity analyses. Press OK in VBR console. Go back to Graylog console > Search and you will see an new event created. The event is generated by Veeam_MP application and the message description contains information about the VBR console event "Malware detection settings have been changed"

Configure Veeam ONE syslog integration

If you have Veeam ONE in your deployed, you can also configure it to send messages to Graylog. Open Veeam ONE client and from the main menu go to Settings > Server Settings (or just press CTRL+S)

Go to Syslog, and press Enable Syslog. Then add the syslog server FQDN or IP address, leave the syslog facility as mail, select the transport protocol and port (we are using TCP 1514).

Then select the syslog audit events that you want to send:

• Access to data
• Changes to data
• Privileged activities
• Login sessions

To test the connections settings, click Test Syslog Integration. Press OK to save the settings. Back in Graylog console you will see the test message:

Additionally you can select to send syslog messages whenever a Veeam ONE alert is triggered. Go to Alarm Management and in the filter field type "malware". You will be presented with the list malware related alarms available out of the box in Veeam ONE:

Let's change the settings for a couple of alarms. Select: "Veeam malware detection exclusion change tracking", right click on it and press Edit. In Alarm Settings window go to Notifications tab. From the Action drop down list select Send Syslog message and leave the Condition as Any state. This will enable sending syslog messages regardless of the alarm state: error, warning or resolved

Press OK to save. Repeat the steps from above  for "Veeam malware detection change tracking" alarm. To trigger a malware configuration change alarm in Veeam ONE we need to change something in VBR console. So back to VBR console, from main menu press Malware Detection and change your current configuration. Remember to press OK to save the change. The change will trigger an alert in Veeam ONE - open the alert to see more details:

Open Graylog console and notice that the Veeam ONE message was received:

In our example we have integrated only 2 components from the infrastructure, but it is easy to understand how SIEM systems are critical to good security posture in the company by allowing the integration of alerts and events from different components. A simple infrastructure configuration change in the backup server could be correlated with an out of hours login on a jump server, and some suspicious network traffic. Having SIEM in place could help detect all of these events, notify operations teams and assist them in  mitigating the breach. 

Veeam Backup & Replication Architecture for Disaster Recovery in Google Cloud

In the following article we look at a DR architecture for Veeam Backup & Replication using Google Cloud as a disaster recovery location and implementing read only access to shared backup repository. 

Having a disaster recovery (DR) plan is not a nice to have, but a core requirement for any business that wants to survive a crisis situation. For any disaster recovery plan we need a secondary location where to restart the services. This secondary location can actually be a public cloud service provider. Veeam Backup & Replication enables recovery of virtual machines backups and agent based backups directly to the cloud. 

We propose to implement a solution with two backup servers (VBR) accessing the same backup data. We deploy one backup server on premises (ON PREM VBR in the following diagram). It acts as our operational server managing backups, backup copy jobs and restores. The second backup server (DR VBR) is deployed in Google Cloud (GCE). It acts as our DR backup server. For the most of the time it will not be used. It becomes active during testing or during a real DR situation. 

On premises VBR is configured to write backups to a local repository. A backup copy job creates a copy of the primary backups to a Google Cloud Storage repository. To write data to the Cloud Storage repository, the on premises VBR will use a HMAC key associated with a service account that has read/write permissions to that bucket. Since we do not plan to use on premises VBR to restore to Google Cloud, these are the only permissions that it needs. It also needs to be the only VBR that has write permission to that bucket. 

The DR VBR is deployed on a GCE VM in backup project. We are using a separate projects to host the backup infrastructure. It uses a service account with read only permissions to Cloud Storage to access data copied by the on premises VBR. Using the read only account we make sure that there will be no incompatibility or data corruption at the repository level. 

Since the cloud VBR is used to recover VMs in case of a DR situation, it needs an additional service account with restore to GCE permissions (listed here). The service account is configured in the project where we will restore the VMs (production project) and added to VBR using service account key. 

The proposed implementation can be further adapted for other scenarios such as sending backups directly to cloud or even cloud only environments.

By using the proposed architecture, we implement 3-2-1 rule and enable fast and secure restores in case of a disaster while keeping flexibility, low costs and RTO/RPO for on premises restores.

Backup vSAN 7 File Share with Veeam Backup & Replication 10

This is a blog post about two of the new features that were released this year in vSAN 7 and in Veeam Backup & Replication v10:

• vSAN File Service
• Veeam NAS Backup 

As you've already guessed, we're going to create a file share on vSAN, put files on it and backup it up with Veeam Backup & Replication (VBR) using the native NAS support.

At a high level the environment looks as following:

A 2 node vSAN 7 cluster runs in a nested environment. Each node has 4 vCPU, 32 GB of RAM, 20 GB Cache SSD and 200 GB Data SSD. A witness appliance has been also deployed. VBR v10 is configured with the minimum hardware: 2 vCPU and 8GB. Both proxy and repository roles are installed on the same VM with backup server, as well as the embedded MSSQL Express DB. Network connectivity between vSAN and VBR is 1 Gbit.

The configuration of the vSAN cluster or the installation of VBR are not in scope of this blog as there are many resources out there. I've used William Lam's nested library for the vSAN nodes and downloaded the witness appliance directly from VMware site.

The prerequisites for the following steps are: a running vSAN cluster and a VBR installation. You will also need 2 IP's reserved for File Server virutal appliances (one for each node) and a working DNS server with records added for the 2 IP's

Part 1 - vSAN File Service

vSAN File Service provides file shares on top of vSAN storage. It supports NFS 3 an NFS 4 protocols. It comprises of vSAN Distributed File System (vDFS) and it is integrated with vSAN Storage Policy Based Management.

Enable vSAN File Services

At the cluster level go to Configure - vSAN - Services and press enable File Service

Select whether the File Server agent OVF appliance will be download automatically or it will be manually uploaded:

Give a namespace for file share, type in DNS server address and domain

Select a portgroup, type the netmask and default gateway IP

Type in the IP addresses for each of the nodes

Review the configuration and press finish. Installation process will start deploying the agents on each of the vSAN nodes.

Once it is finished you will 2 new components deployed on the vSAN cluster: the File Service Agents. These are virtual appliances running Photon OS 1 and Docker and they act as NFS file servers.

Create NFS file share

It's time to create the share and put some files to it. To create the share Configure - vSAN - File Services Share. Enter the name of the share, storage policy, soft and hard quota limits.

You can also assign labels to the file share. Quotas can be changed at a later time. Next define access control: what IP addresses have access to this share, what type of access and if to protect the share with root squash.

Lastly review the settings and create the share. Once created you will see it vSAN - File Services Share:

Now the file share is ready to be used. Because in part 2 we will use Veeam's NAS backup, we've added a small load of files on the share using a linux VM. First get the share path by selecting the file share and pressing copy on URL:

Next ssh to your favorite Linux VM and mount the file share:

To create random files with random size, we've used a simple script found on here and which was slightly adapted.

for n in {1..500}; do
    dd if=/dev/urandom of=file$( printf %03d "$n" ).bin bs=100 count=$(( RANDOM + 1024 ))
done

The workload tested was:
- 200 files ranging  between a few hundreds of KB and a few MB
- 50 files ranging from a few MB and few tens of M

The script can be adapted to create larger workloads with thousands of files. Since the whole lab is nested, performance testing was not in scope.

Part 2 - Veeam NAS Backup 

The architecture for NAS backup requires at a minimum a file share (in our case NFS from vSAN), file proxy, cache repository, backup repository and a Veeam Backup Server.

Additionally, secondary and archive repositories can be added in infrastructure. In our lab infrastructure all components are deployed on a single VM - the backup server.

File proxy - component that acts as the data mover transporting data from the source (file share) to the backup proxy. It is used in backup and restore activities.

Cache repository - is a location to store temporary metadata. This metadata is used to reduce the load on the file share during incremental backups.

Backup repository - main location of the backups

Add File Share 
In VBR Console go to Inventory - Add File Share

Select NFS share and specify the path to the share

Select the File proxy, cache repository and backup speed

Review the summary and finish the configuration.

Having the file share added, we will add it to a backup job. Right click it and add it to a new backup job (or use an existing one).

Select the file share to backup

Select destination repository, backup policy and archive repository if any.

Select a secondary target if required (backup copy for short term)

Enable the job to run automatically

Review the settings and run the job.

The initial execution backups up all 250 files on the share. For the incremental, we've removed and recreated the first 50 files on the share:

for n in {1..50}; do 
  rm -f file$( printf %03d "$n" ).bin
  dd if=/dev/urandom of=file$( printf %03d "$n" ).bin bs=100 count=$(( RANDOM + 1024 ))
done

As you can see the cache repository is pretty effective in determining which files to be backed up:

Veeam Backup & Recovery - Change Block Tracking Reset

Change Block Tracking (CBT) is a feature that allows tracking of changed disk sectors. Tracking for virtual disks is done in the virtualization layer. CBT is exposed through vSphere APIs for Data Protection (VADP) to 3rd party applications. 

VBR uses this feature to track changes between incremental backups and make those backups faster. Instead of reading the whole vmdk, it will ask and receive only the changed blocks from the last incremental backup. How do you know CBT is enabled and used? 

From VMware point of view, you will see for each vmdk a file with the  vmdk_name-ctk.vmdk

From Veeam point of view, you will see in the backup job statistics next to the disk details CBT. Let's look at what actually happens in the backup job.

CBT on, active full backup, read 20 GB

CBT on, incremental backup, read 11 MB

CBT off, incremental backup, read 20 GB

This is CBT in action. The first picture is an active full, so the whole disk has been read. The second one is an incremental where only changes from last backup have been read. When CBT was disabled, the second incremental also read 20GB.  Now extrapolate this to hundreds of VMs to understand the importance of CBT: less data read means not only a shorter backup window, but also less load on the production storage. 

Luckily, CBT is enabled by default for all newly created backup jobs. It can be found in the vSphere integration tab, in Storage > Advanced. 

CBT is not supported for physical mode RDMs and if a VM has snapshots when it's activated the first time. Sometimes CBT can get corrupted and the only way to solve it is to reset CBT. This is not easily done A new feature in VBR v10 allows to automatically reset CBT on all VMs when after an active full backup is executed. Remembering how many times I heard support guys advising a CBT reset, I think this is a cool feature to add. The CBT reset action is caught in the backup stats window

The downside is that the active full backup will take a bit longer, but you will be protected against potential CBT corruptions.