Veeam Backup for AWS: Workers

In the previous post we looked at Veeam Backup for AWS (VBA) appliance architecture and deployment. To protect AWS resources VBA uses two methods: worker based and AWS native protection (snapshots and backup vaults). We will treat native protection in future posts where we will look at the backup process for different types of resources.

Workers are part of the VBA architecture. They are temporary instances deployed automatically at the beginning of a backup, restore or retention activity. They are removed at the end of the operation. Workers are used for the following operations:

• image-level backups for EC2 and RDS
• restore data from an image-level backup
• index EFS 
• retention task

Backups created by workers are sent to a backup repository which uses Amazon S3.    

Workers are based on standard amzn-linux-v2 and the specific Veeam binaries are installed from a dedicated S3 bucket (configuration bucket). They run the following services:

Veeam Data Mover —  retrieves data of protected AWS resources and transfers it to backup repositories. During restore, it transfers backed-up data from backup repositories to the target location.

File-level recovery browser — web service that allows to find and save files and folders of a backed-up EC2 instance to the local machine or to the original location.

Worker are deployed in either the backup account or in the production account. As a best practice, a backup account should be implemented to deploy the VBA appliance and the workers. Additionally a dedicated repository account should be created for the S3 buckets used as backup repositories. This account segmentation is aligned with an increased security posture and helps prevent catastrophic situations in case of an account breach.

For the following operations the workers are deployed in the backup account:

• EC2 image-level backup
• entire EC2 instance restore from image-level backups
• EC2 volume-level restore from image-level backups
• EC2 file-level recovery
• EC2 backup retention tasks
• RDS archived backup

, 

By default, workers are deployed in production accounts for the following operations:

• EFS indexing
• RDS image-level backup
• RDS database restore from image-level backup

However, additional operations can have workers distributed across multiple production accounts:

• EC2 image-level backups
• EC2 instance restore from image-level backups
• EC2 volume-level restore from image-level backups 
• EC2 file-level recovery from snapshots

During deployment operations, VBA assumes a worker deployment role in the account where it needs to deploy the workers. A temporary IAM role is attached to each deployed worker. By default, VBA uses the "Default Backup Restore" role that has all the permissions required to perform data protection and disaster recovery operations.

Workers are created with default profiles based on the size of processed data: 

• Small profile (c5.large) - total EBS volume size < 1024 GB
• Medium profile (c5.2xlarge) - 1024 GB < total EBS volume size < 16 TB
• Large profile (c5.4xlarge) - total EBS volume size > 16 TB
• Archiving profile (c5.2xlarge) is used for transferring data to archive repositories

The default profiles can be customized per region level. However, make sure you understand the impact of changing the worker size on the overall costs for the operation.

Tags can be automatically added to worker instances and the following parameters are available for use:

• %applianceId% - unique VBA appliance ID
• %policyId% - policy ID for which the worker is deployed
• %policyName% - policy name for which the worker is deployed

From networking point of view, you can select per region VPC, subnet and security group where to connect the instances:

VBA uses public IP to communicate with workers which means the selected VPC must have internet gateway and routing tables and security groups should permit access. This behavior can be changed and worker instances can be deployed in a private network by enabling private network deployment functionality in VBA. To ensure your VBA can work in private environment, configure private endpoints as presented here.

Workers are a key architectural component of VBA. Understanding their use cases and how to customize a standard deployment is essential to creating a proper VBA architecture. 

Veeam Backup for AWS: Backup Appliance Architecture and Deployment

We continue our series of posts on Veeam Backup for AWS (VBA) and we look at its architecture and deployment process. Veeam Backup for AWS appliance coordinates backup and recovery tasks, controls scheduling, notifications, manages retention tasks and controls deployment of additional components used during backup and restore operations. 

Veeam Backup for AWS appliance is a Linux-based EC2 instance running several services:

• Backup service - coordinates backup and recovery operations.
• Configuration database (PostgreSQL) - stores configuration of backup policies, worker instance, added IAM roles, sessions and so on
• Configuration restore service — allows to back up and restore the configuration of the backup appliance. While connected to a VBR server, the configuration backup will be configured and executed from VBR console.
• Web UI  - web interface to access VBA
• Veeam Updater service - install product and package updates
• Veeam FLR service — restore individual files and folders of protected EC2 instances
• Self Backup service — allows Veeam Backup for AWS to back up and restore the configuration database of the backup appliance.
• REST API service

Veeam Backup & Replication server (VBR) centrally manages VBA appliances. One or more instances can be managed from the same VBR allowing for multi-region deployment. The backup server (VBR) can be deployed either in AWS or on-premises. Backup server uses Veeam Plugin for AWS to connect to AWS services and manage the lifecycle of Veeam Backup for AWS components. 

To start VBA deployment from VBR  make sure you have access to an AWS account. During deployment process, Veeam Plugin for AWS connects to multiple AWS APIs and services:

• AWS IAM  - create IAM roles for VBA (impersonation role and backup/restore access role)
• Amazon EC2 - deploy the appliance from an EC2 Ubuntu image and create networking if necessary 
• AWS SSM - run scripts and configure the VBA appliance 
• Amazon S3 - deployment scripts and binaries
• Amazon KMS 
• Amazon STS 
• CloudWatch 
• Service Quota

You can choose to connect to an existing appliance or deploy a new one. Next add AWS account access key and secret key. Remember the account needs permissions to the above services. See below a list of all the APIs called during installation:

• cloudtrail.amazonaws.com
• config.amazonaws.com
• ec2.amazonaws.com
• kms.amazonaws.com
• monitoring.amazonaws.com
• notifications.amazonaws.com
• resource-explorer-2.amazonaws.com
• s3.amazonaws.com
• servicequotas.amazonaws.com
• ssm.amazonaws.com
• sso.amazonaws.com
• sts.amazonaws.com

Select the region where you want to deploy VBA, enter an instance name for the appliance and choose how to connect to it - dynamic public IP, static public IP or private IP. In the  case you choose private IP you need to make sure you have access to that IP range. The deployment process can also create networking infrastructure if it does not exist - VPC, subnets, security groups, internet gateway and so on. Finally enter the guest OS administrator credentials and the key pair to encrypt credentials with. 

The deployment process automatically goes through several stages: 

• get and deploy an EC2 Ubuntu image 
• create IAM roles
• prepare the instance (deploy and configure the required binaries in the appliance)
• create administrator credentials for the appliance 
• initialize services in the appliances
• check for updates for Ubuntu Linux, Microsoft Windows (ASP.NET Core Runtime) and PostgreSQL
• install updates 
• reboot appliance
• add permissions to IAM role

The installation can take some time, so be patient. During the installation IAM roles are automatically created.

"Impersonation role" is attached to the backup appliance and it is used to assume other IAM roles. It has the following permissions:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": "sts:AssumeRole",

            "Resource": "*"

        }

    ]

}

and the following trust relationship:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Principal": {

                "Service": "ec2.amazonaws.com"

            },

            "Action": "sts:AssumeRole"

        }

    ]

}

Another 2 roles are created: "VeeamInstanceBackupRestoreAccessRole" and "VcbSnapshotsRole"

VeeamInstanceBackupRestoreAccessRole - is the default role that will allow VBA to perform all backup and restore operations. It must be granted permissions to assume the IAM roles:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Principal": {

                "Service": [

                    "ec2.amazonaws.com",

                    "backup.amazonaws.com",

                    "batchoperations.s3.amazonaws.com"

                ],

                "AWS": "arn:aws:iam::1111111:role/VeeamImpersonationRole"

            },

            "Action": "sts:AssumeRole"

        }

    ]

}

The permissions list is pretty long and it can be found here. In IAM a series of policies are created and named with vbaws_policy_* and vcb_policy_* in order to add all required permissions. 

VcbSnapshotsRole - gives permissions to manage ec2 snapshots and it also needs assume permissions:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Principal": {

                "Service": "dlm.amazonaws.com"

            },

            "Action": "sts:AssumeRole"

        }

    ]

}

After VBA appliance is deployed, you may choose to create and configure a S3 bucket to use as a backup repository and start the wizard from the VBR console or open VBA console and start configuring the appliance. 

Veeam ONE for NIS 2 Compliance: Key Benefits and Features

The NIS 2 Directive, an EU regulatory standard for cybersecurity, has brought new compliance requirements that impact organizations across critical sectors. Enforced to enhance cybersecurity across the Union, the directive applies to both "essential" and "important" entities, which must now align with stricter guidelines on incident reporting, risk management, and continuous security monitoring. Articles 21 and 23, in particular, detail essential cybersecurity risk management measures and rigorous reporting practices. For companies looking to meet these mandates, Veeam ONE offers a robust solution through its in-depth monitoring, alerting, and reporting capabilities. In this post, we’ll explore how Veeam ONE aligns with NIS 2 compliance, enabling organizations to efficiently manage cybersecurity risks and meet their regulatory requirements.

Let’s first examine Articles 21 and 23 of the NIS 2 Directive:

• Article 21 establishes the need for essential cybersecurity measures and risk management practices. This includes continuous monitoring, incident prevention and detection, vulnerability management, and real-time security alerts to help manage cybersecurity risks.

• Article 23 outlines the expectations for incident handling and reporting. Organizations are required to have mechanisms in place for reporting cybersecurity incidents in a timely manner, providing detailed information to relevant authorities, and ensuring these incidents are contained and managed to prevent widespread impact.

In essence, these articles push organizations to adopt an end-to-end approach to cybersecurity, emphasizing visibility, timely reporting, and proactive threat management—all of which can be effectively addressed by Veeam ONE.

Designed to deliver insights, monitoring, and proactive alerting, Veeam ONE aligns well with the NIS 2 Directive’s requirements for both risk management and incident reporting. Here’s how:

1. Continuous Monitoring and Real-Time Visibility

Article 21 mandates the implementation of continuous monitoring for risk management, and Veeam ONE’s advanced monitoring features fulfill this requirement. It provides a unified view of the virtual and backup infrastructure, allowing organizations to observe system performance, detect anomalies, and gain visibility into potential vulnerabilities across both virtual and physical environments. 

Out of the box and customizable dashboards are used to monitor in real time the state of your infrastructure as seen below in vSphere Alarms dashboard:

2. Advanced Alerting for Proactive Risk Management

Veeam ONE supports compliance with Article 21’s focus on risk management through its advanced alerting capabilities. Organizations can configure specific alerts for potential threats or anomalies such as ransomware attacks, allowing IT teams to respond to risks promptly. These alerts can be customized to focus on critical NIS 2 areas, such as system failures, unauthorized access attempts, or irregular system behaviors. The ability to set thresholds for alerts ensures that teams are notified about risks before they escalate, enhancing response time and preventing potential incidents.

Alerts can be used immediately and notifications received in real time via syslog, SNMP traps, e-mail or ServiceNow. A deep view on any possible threats is provided, such as failed login attempts:

3. Comprehensive Reporting and Documentation for Compliance

One of the key aspects of Article 23 is the ability to report incidents comprehensively. Veeam ONE’s robust reporting features make it easier to document and communicate relevant cybersecurity events. Reports can be generated on demand, providing detailed documentation of incidents, system performance, and overall cybersecurity health. These reports not only help in complying with NIS 2 but also streamline communication with stakeholders and regulatory authorities.

More than a hundred predefined reports that can be scheduled to run periodically are available:

4. Resource Optimization and Capacity Planning

To meet NIS 2 compliance, organizations must also ensure that they have the resources to manage cybersecurity effectively. Veeam ONE’s capacity planning tools allow organizations to forecast resource needs, identify areas of potential overuse or underutilization, and optimize resource allocation across critical areas. 

5. Enhanced Incident Response and Management

Veeam ONE supports Article 23’s emphasis on efficient incident response by providing a detailed view of system health and performance, enabling organizations to react promptly to cybersecurity events. Veeam ONE’s integration with Veeam Backup & Replication also allows organizations to automate responses for certain incidents, ensuring continuity and minimizing the impact of any security event

By providing critical insights, proactive alerts, and robust reporting capabilities, Veeam ONE empowers organizations to meet NIS 2 mandates while maintaining optimal security and performance. As cybersecurity threats continue to evolve, solutions like Veeam ONE can be invaluable assets for organizations looking to stay compliant and protect their operations.

Veeam Backup for AWS: Comprehensive Cloud Data Protection

In today's cloud-dependent world, data protection is essential for maintaining business continuity. Veeam Backup for AWS (VBA) offers an AWS-native, highly adaptable solution designed to protect, manage, and recover data within AWS environments. Its main purpose is to help organizations address the unique data protection needs of AWS workloads, ensuring that cloud data remains resilient against threats like accidental deletion, cyberattacks, or service interruptions.

Key Components of Veeam Backup for AWS

1. Automated Backup and Recovery: Veeam allows for fully automated backup processes, supporting Amazon EC2, RDS, Dynamo DB, Redshift, EFS, FSx and VPC. With policies and schedules, users can customize backups to fit business needs and ensure their critical data is consistently protected.

2. Cost Optimization: Veeam uses Amazon S3 and its various storage classes, such as Glacier and Glacier Deep Archive, to optimize storage costs. Users can automatically tier their data to lower-cost storage options, making cloud backups more affordable without sacrificing accessibility.

3. Immutability and Security: Leveraging Amazon S3 Object Lock, Veeam ensures that backups remain immutable, providing a strong defense against ransomware and other cyber threats. This feature prevents any changes or deletions to stored data within a specified timeframe, securing it from unauthorized access or malicious attacks.

4. Cross-Region and Cross-Account Recovery: In case of an outage or disaster, Veeam enables cross-region and cross-account recovery, allowing users to restore data quickly and securely across different AWS accounts or regions, thereby meeting stringent recovery objectives.

5. User-Friendly Interface and Self-Service: The solution includes a streamlined interface that simplifies backup setup and monitoring. Additionally, self-service recovery options allow users to restore their data with minimal intervention, enabling faster response times in critical situations.

Starting with version 7.0, Veeam Backup for AWS is part of the Veeam Backup & Replication (VBR) solution. AWS Plug-in for Veeam Backup & Replication extends the Veeam Backup & Replication functionality and allows you to add backup appliances to Veeam Backup & Replication. The entire lifecycle of VBA is managed from VBR through AWS Plug-in. 

Deployment, update and management of VBA is done from VBR console. Currently you can still deploy VBA from AWS marketplace, connect it to VBR and upgrade it to the latest version. However this process is deprecated and only VBR console should be used to manage VBA. One or multiple VBA appliances can be managed from the same VBR server. 

Additionally, Veeam ONE can offer enhanced monitoring and reporting capabilities for VBA by collecting date about protected AWS resources. 

By combining these components, Veeam Backup for AWS provides an end-to-end backup and disaster recovery solution tailored for AWS cloud environments, balancing security, cost, and ease of use. 

In the following posts we will take a deeper look at Veeam Backup for AWS architecture and operations.

Integrating Veeam Backup Infrastructure with Security Information and Event Management Solution

Security Information and Event Management (SIEM) systems provide comprehensive security monitoring, threat detection, and incident response capabilities. The main features a SIEM should provide are:

• Data Collection from a wide range of sources, including log files, network traffic, system events and security alerts.
• Normalization and Correlation where collected data is standardized, stored and events are correlated.
• Threat Detection and Analysis using predefined rules, statistical analysis, machine learning algorithms, and threat intelligence feeds.
• Alerting and Reporting to notify security personnel in real-time
• Incident Response and Forensics tools for investigating security incidents, conducting forensic analysis

All of the above make SIEM complex systems and also not cheap. However this does not minimize the importance of having SIEM deployed in your enterprise environment as a proactive line of defense against attacks that can compromise services availability and integrity.

Data protection solutions play another crucial role in making sure your environment is safe and can recover in case of any incident ranging from unintentional deletion to a sophisticated malware attack. It makes sense to integrate your backup infrastructure into your SIEM. Veeam offers the possibility to send events and alerts to a SIEM solution using syslog protocol. 

Install Graylog

Let's look at first at the SIEM solution. For our lab environment, we chose Graylog Open as it offers  a basic and free SIEM solution that can run on top of Ubuntu. We are using an Ubuntu 22.04 template with a static IP address that is resolvable via DNS. To install Graylog Open we have followed the instructions from this link. Please note that MongoDB and OpenSearch (or ElastichSearch) are required on the Graylog server. We will not repeat the steps since it does not make any sense to duplicate the content. 

Configure Graylog for Veeam data ingestion

Once Graylog is installed, login to the admin interface using the http_bind_address configured in /etc/graylog/server/server.conf. For our lab it would be http://graylog_ip_address:9000, for example. 

Go to System > Inputs. Select Syslog TCP  and press Launch New Input

Type in a name, add the bind address where to listen for incoming connections and makes sure to use a TCP port above 1024. By default Graylog runs under a normal user and cannot bind to ports under 1024. You may leave the rest of the settings as default. Press Launch Input and make sure it shows as running 

When the input is running, Graylog is ready to receive messages from Veeam infrastructure. 

Configure Veeam Backup & Replication syslog integration

First, open the VBR console and go to Main menu > Options

In the new window got to Event Forwarding and under Syslog servers press Add. Configure the Graylog input: IP address or FQDN, port and protocol, Please note that only one syslog server can be configured 

Press OK to save the syslog configuration and save it to VBR. You are now ready to test. The simplest way is to enable/disable Malware Detection configuration in VBR console. On main menu, press Malware Detection and in the new window make a change to your current configuration, for example deselect Enable file system activity analyses. Press OK in VBR console. Go back to Graylog console > Search and you will see an new event created. The event is generated by Veeam_MP application and the message description contains information about the VBR console event "Malware detection settings have been changed"

Configure Veeam ONE syslog integration

If you have Veeam ONE in your deployed, you can also configure it to send messages to Graylog. Open Veeam ONE client and from the main menu go to Settings > Server Settings (or just press CTRL+S)

Go to Syslog, and press Enable Syslog. Then add the syslog server FQDN or IP address, leave the syslog facility as mail, select the transport protocol and port (we are using TCP 1514).

Then select the syslog audit events that you want to send:

• Access to data
• Changes to data
• Privileged activities
• Login sessions

To test the connections settings, click Test Syslog Integration. Press OK to save the settings. Back in Graylog console you will see the test message:

Additionally you can select to send syslog messages whenever a Veeam ONE alert is triggered. Go to Alarm Management and in the filter field type "malware". You will be presented with the list malware related alarms available out of the box in Veeam ONE:

Let's change the settings for a couple of alarms. Select: "Veeam malware detection exclusion change tracking", right click on it and press Edit. In Alarm Settings window go to Notifications tab. From the Action drop down list select Send Syslog message and leave the Condition as Any state. This will enable sending syslog messages regardless of the alarm state: error, warning or resolved

Press OK to save. Repeat the steps from above  for "Veeam malware detection change tracking" alarm. To trigger a malware configuration change alarm in Veeam ONE we need to change something in VBR console. So back to VBR console, from main menu press Malware Detection and change your current configuration. Remember to press OK to save the change. The change will trigger an alert in Veeam ONE - open the alert to see more details:

Open Graylog console and notice that the Veeam ONE message was received:

In our example we have integrated only 2 components from the infrastructure, but it is easy to understand how SIEM systems are critical to good security posture in the company by allowing the integration of alerts and events from different components. A simple infrastructure configuration change in the backup server could be correlated with an out of hours login on a jump server, and some suspicious network traffic. Having SIEM in place could help detect all of these events, notify operations teams and assist them in  mitigating the breach. 

Veeam Backup & Replication Architecture for Disaster Recovery in Google Cloud

In the following article we look at a DR architecture for Veeam Backup & Replication using Google Cloud as a disaster recovery location and implementing read only access to shared backup repository. 

Having a disaster recovery (DR) plan is not a nice to have, but a core requirement for any business that wants to survive a crisis situation. For any disaster recovery plan we need a secondary location where to restart the services. This secondary location can actually be a public cloud service provider. Veeam Backup & Replication enables recovery of virtual machines backups and agent based backups directly to the cloud. 

We propose to implement a solution with two backup servers (VBR) accessing the same backup data. We deploy one backup server on premises (ON PREM VBR in the following diagram). It acts as our operational server managing backups, backup copy jobs and restores. The second backup server (DR VBR) is deployed in Google Cloud (GCE). It acts as our DR backup server. For the most of the time it will not be used. It becomes active during testing or during a real DR situation. 

On premises VBR is configured to write backups to a local repository. A backup copy job creates a copy of the primary backups to a Google Cloud Storage repository. To write data to the Cloud Storage repository, the on premises VBR will use a HMAC key associated with a service account that has read/write permissions to that bucket. Since we do not plan to use on premises VBR to restore to Google Cloud, these are the only permissions that it needs. It also needs to be the only VBR that has write permission to that bucket. 

The DR VBR is deployed on a GCE VM in backup project. We are using a separate projects to host the backup infrastructure. It uses a service account with read only permissions to Cloud Storage to access data copied by the on premises VBR. Using the read only account we make sure that there will be no incompatibility or data corruption at the repository level. 

Since the cloud VBR is used to recover VMs in case of a DR situation, it needs an additional service account with restore to GCE permissions (listed here). The service account is configured in the project where we will restore the VMs (production project) and added to VBR using service account key. 

The proposed implementation can be further adapted for other scenarios such as sending backups directly to cloud or even cloud only environments.

By using the proposed architecture, we implement 3-2-1 rule and enable fast and secure restores in case of a disaster while keeping flexibility, low costs and RTO/RPO for on premises restores.