Integrating Veeam Backup Infrastructure with Security Information and Event Management Solution

Security Information and Event Management (SIEM) systems provide comprehensive security monitoring, threat detection, and incident response capabilities. The main features a SIEM should provide are:

• Data Collection from a wide range of sources, including log files, network traffic, system events and security alerts.
• Normalization and Correlation where collected data is standardized, stored and events are correlated.
• Threat Detection and Analysis using predefined rules, statistical analysis, machine learning algorithms, and threat intelligence feeds.
• Alerting and Reporting to notify security personnel in real-time
• Incident Response and Forensics tools for investigating security incidents, conducting forensic analysis

All of the above make SIEM complex systems and also not cheap. However this does not minimize the importance of having SIEM deployed in your enterprise environment as a proactive line of defense against attacks that can compromise services availability and integrity.

Data protection solutions play another crucial role in making sure your environment is safe and can recover in case of any incident ranging from unintentional deletion to a sophisticated malware attack. It makes sense to integrate your backup infrastructure into your SIEM. Veeam offers the possibility to send events and alerts to a SIEM solution using syslog protocol. 

Install Graylog

Let's look at first at the SIEM solution. For our lab environment, we chose Graylog Open as it offers  a basic and free SIEM solution that can run on top of Ubuntu. We are using an Ubuntu 22.04 template with a static IP address that is resolvable via DNS. To install Graylog Open we have followed the instructions from this link. Please note that MongoDB and OpenSearch (or ElastichSearch) are required on the Graylog server. We will not repeat the steps since it does not make any sense to duplicate the content. 

Configure Graylog for Veeam data ingestion

Once Graylog is installed, login to the admin interface using the http_bind_address configured in /etc/graylog/server/server.conf. For our lab it would be http://graylog_ip_address:9000, for example. 

Go to System > Inputs. Select Syslog TCP  and press Launch New Input

Type in a name, add the bind address where to listen for incoming connections and makes sure to use a TCP port above 1024. By default Graylog runs under a normal user and cannot bind to ports under 1024. You may leave the rest of the settings as default. Press Launch Input and make sure it shows as running 

When the input is running, Graylog is ready to receive messages from Veeam infrastructure. 

Configure Veeam Backup & Replication syslog integration

First, open the VBR console and go to Main menu > Options

In the new window got to Event Forwarding and under Syslog servers press Add. Configure the Graylog input: IP address or FQDN, port and protocol, Please note that only one syslog server can be configured 

Press OK to save the syslog configuration and save it to VBR. You are now ready to test. The simplest way is to enable/disable Malware Detection configuration in VBR console. On main menu, press Malware Detection and in the new window make a change to your current configuration, for example deselect Enable file system activity analyses. Press OK in VBR console. Go back to Graylog console > Search and you will see an new event created. The event is generated by Veeam_MP application and the message description contains information about the VBR console event "Malware detection settings have been changed"

Configure Veeam ONE syslog integration

If you have Veeam ONE in your deployed, you can also configure it to send messages to Graylog. Open Veeam ONE client and from the main menu go to Settings > Server Settings (or just press CTRL+S)

Go to Syslog, and press Enable Syslog. Then add the syslog server FQDN or IP address, leave the syslog facility as mail, select the transport protocol and port (we are using TCP 1514).

Then select the syslog audit events that you want to send:

• Access to data
• Changes to data
• Privileged activities
• Login sessions

To test the connections settings, click Test Syslog Integration. Press OK to save the settings. Back in Graylog console you will see the test message:

Additionally you can select to send syslog messages whenever a Veeam ONE alert is triggered. Go to Alarm Management and in the filter field type "malware". You will be presented with the list malware related alarms available out of the box in Veeam ONE:

Let's change the settings for a couple of alarms. Select: "Veeam malware detection exclusion change tracking", right click on it and press Edit. In Alarm Settings window go to Notifications tab. From the Action drop down list select Send Syslog message and leave the Condition as Any state. This will enable sending syslog messages regardless of the alarm state: error, warning or resolved

Press OK to save. Repeat the steps from above  for "Veeam malware detection change tracking" alarm. To trigger a malware configuration change alarm in Veeam ONE we need to change something in VBR console. So back to VBR console, from main menu press Malware Detection and change your current configuration. Remember to press OK to save the change. The change will trigger an alert in Veeam ONE - open the alert to see more details:

Open Graylog console and notice that the Veeam ONE message was received:

In our example we have integrated only 2 components from the infrastructure, but it is easy to understand how SIEM systems are critical to good security posture in the company by allowing the integration of alerts and events from different components. A simple infrastructure configuration change in the backup server could be correlated with an out of hours login on a jump server, and some suspicious network traffic. Having SIEM in place could help detect all of these events, notify operations teams and assist them in  mitigating the breach. 

Veeam Backup for Google Cloud - Zero Trust Architecture with Cloud Identity-Aware Proxy

Having security embedded by design into your architecture is more than just a best practice. It is how any one should actually start their work in any kind of project in public, private or hybrid cloud. Veeam Backup for Google Cloud (VBG) is one of the technologies that enables data security and resiliency by backing up and protecting your data running in the cloud. However, VBG is also residing in the same cloud and one of the first things is to make sure it is deployed and accessed in a secure manner. 

The challenge rises from the need to access VBG console for configuration and operation activities. The focus of this post is securing this access. 

In a standard deployment you would have your VBG appliance installed in a VPC, apply firewall rules to restrict access to VBG and then using an SSL encrypted web browser connect to the console. This connectivity can be done over Internet or in some more complex scenarios over VPN or interconnect links.  If you are connecting to VBG over Internet, you would need to expose VBG using a public IP address and restrict access to that IP address from your source IP. This is the use case that we are treating in our article. Another scenario using bastion servers and private connectivity is not treated now, however principals and mechanisms learned here can still apply. 

As you can easily see there are some disadvantages in the having VBG directly accessed from Internet. First, VBG is directly accessed from Internet. Having a firewall rule that limits source IP addresses allowed to connect to the external IP address of VBG increases the security trust, but it does not apply zero trust principles. We don't know who is hiding behind that allowed source IP address. There is no user identification and authorization in place before allowing the user to open a session to VBG console. Anyone connecting from that specific source IP address is automatically trusted.

How can we make sure that whoever or whatever trying to connect to VBG is actually allowed to do it? Please mind that we are talking about the connection to VBG console before any authentication and authorization into VBG is applied. We want to make sure that whoever tries to enter credentials in VBG console is identified and has the permissions to do that action. 

Think of use cases where your user has lost his rights to manage backups, however still has access to the backup infrastructure. You would want to have a secure and simple way of controlling that access and being able to easily revoke it. In this situation we can use Cloud Identity and Access Management (IAP) and Cloud Identity-Aware Proxy (IAP).

How does it work?

Cloud IAP implements TCP forwarding which encrypts any type of TCP traffic between the client initiating the session and IAP using HTTPS. In our case we normally connect to VBG console using HTTPS (web browser). Adding IAP TCP forwarding, the initial HTTPS traffic will be encrypted in another HTTPS connection. From IAP to VBG, the traffic will be sent without the additional layer of encryption. The purpose of using IAP is to keep VBG connected to private networks only and control which users can actually connect by using permissions and IAM users. 

Public IP of VBG will be removed and if outbound connectivity is needed, then use a NAT gateway to enable it, but this is out of scope for the current post.

To summarize, instead of allowing anyone behind an IP address to connect to our VBG portal, we restrict this connectivity to specific IAM users. Additionally we keep VBG on a private network.

Guide

Start by preparing the project: enable Cloud Identity-Aware Proxy API. In the console 

• APIs & Services > Enable APIs and Services 
• search for Cloud Identity-Aware Proxy API  and press enable.

Once enabled you will see it displayed in the list APIs

Allow IAP to connect to your VM by creating a firewall rule. In console go to VPC network > Firewall and press Create Firewall Rule

• name: allow-ingress-from-iap
• targets: Specifed target tags and select the tag of your VBG instance. We are using "vbg-europe" network tag. If you don't use network tags you can select "All instances in the network"
• source IPv4 ranges: Add the range 35.235.240.0/20 which contains all IP addresses that IAP uses for TCP forwarding.
• protocols and ports - specify the port you want to access - TCP 443
• press Save

Grant to users (or groups) permissions to use IAP TCP forwarding and specify to which instance to make it as restrictive as possible. Grant the roles/iap.tunnelResourceAccessor role to VBG instance by opening IAP admin page in console (Security > Identity-Aware Proxy). Go to SSH and TCP Resources page (you may ignore the OAuth warning).

Select your VBG instance and press Add principal. Give to the IAM principal IAP-Secured Tunnel User role. You may want to restrict access to VBG to specific periods of time or days of the week. In this case add an IAM time based condition as seen in the example below.

Save the configuration and now you are ready to connect to your isolated VBG instance. On the machine where you want to initialize the connection you would need to have gcloud cli installed (Cloud SDK). Run the following command to open a TCP forwarding tunnel to VBG instance on port 443.

gcloud compute start-iap-tunnel  your-vbg-instance-name 443 --local-host-port=localhost:0  --zone=your-instance-zone

When the tunnel is established you will see a message in the console with the local TCP port that is used for forwarding, similar to below image:

To be able to execute gcloud compute start-iap-tunnel you need to have compute.instances.get and compute.instances.list permissions on the project where VBG instance runs. You may grant the permissions to the users or groups using a custom role.

In case the user is not authorized in IAP or an IAM condition applied, then you will get the following message when trying to start the tunnel:

Finally it's time to open your browser, point it to the localhost and TCP port returned by gcloud command and connect to your VBG instance in the cloud: 

The proposed solution is suitable for management and operations of VBG. However, please keep in mind that IAP TCP forwarding is not intended for bulk data data transfer. Also, IAP automatically disconnects sessions after one hour of inactivity. 

In this post we've seen how to use Cloud IAP and Cloud IAM to enable secure access to Veeam Backup for Google Cloud console using zero trust architecture principals.

Create vCenter Server Roles Using PowerCLI - Applied to Veeam Backup & Replication

Security is important and having a minimal set of permissions is a requirement, not an option. Having this in mind (and being asked a few times by customers), I put together a short script that will create the vCenter Server roles required by Veeam Backup & Replication service account and Veeam ONE service account. The two accounts have different requirements, with Veeam ONE being the most restrictive as it needs mostly read only.

The script itself is pretty straight forward, the more time consuming is getting the privilege lists. So here you are:

• Veeam ONE role
• Veeam Backup & Replication role

And now for the actual scripting part:

$role = "Veeam Backup Server role"
$rolePrivilegesFile = "veeam_vc_privileges.txt"
$vCenterServer = "your-vcenter-server-FQDN"
Connect-VIServer -server $vCenterServer
$roleIds = @()
Get-Content $rolePrivilegesFile | Foreach-Object{
    $roleIds += $_
}
New-VIRole -name $role -Privilege (Get-VIPrivilege -Server $vCenterServer -id $roleIds) -Server $vCenterServer

The script will create a new vCenter Server role assigning it privileges from the file given as input.

If you ever require to get the privileges from vCenter Server then the next piece of code will help (thanks to VMware communities)

$role = "VBR Role"
Get-VIPrivilege -Role $role | Select @{N="Privilege Name";E={$_.Name}},@{N="Privilege ID";E={$_.ID}}

You will use the privilege ID format for creating the new role.

Running Veeam PowerShell Scripts in Non-Interactive Mode - Credentials

This time we get back to some PowerShell basics and how to run scripts in non-interactive mode.

Veeam Backup & Replication and scripting go hand in hand really well. And the first thing you do when running a script is to connect to backup server

Connect-VBRServer -Server $ServerName -User $User -Password $Pass

Placing user and pass in clear text in a script may not be the best option. We could use PSCredential object to hold the username and password.

$PSCredential = Get-Credential
Connect-VBRServer -Server $ServerName -Credential $PSCredential

However getting the credentials in the objects implies using Get-Credential cmdlet which is interactive and will prompt to type the user and password. This makes it hard to run the script in non-interactive mode.

To make it non-interactive we need password saved somewhere. To save and encrypted password we can use ConvertFrom-SecureString cmdlet and pipe the output to a file:

(Get-Credential).Password | ConvertFrom-SecureString | Set-Content $encPassFileName

Since the username could be stored in the script itself, we retrieve only the secure string for the password, pipe it to the encrypting cmdlet and then output it to a file. Opening the output file will list something similar to:

000d08c9ddf0115d1118c7a00c04fc297eb01000

Now we need to create the PScredential object:

$password = Get-Content $encPassFileName | ConvertTo-SecureString 
$psCredential = New-Object System.Management.Automation.PsCredential($username,$password)

First, we loaded the content of the file storing the encrypted password and convert it to a secure string (PScredential objects use secure strings). Next we created the object with the username and the password. The generated object can be used to connect to VBR Server

Connect-VBRServer -Server $ServerName -Credential $PSCredential

By default, ConvertFrom-SecureString encryption is done using Windows Data Protection API (DPAPI). For this reason the file cannot be moved to another computer and used from that one. On each computer from where the script is being run, the password must be encrypted separately.

Updating vSphere 6.5 with vSphere Update Manager

Update Manager has been and is the tool for upgrading and patching ESXi hosts, virtual appliances and VMs. There was only one caveat: it needed a Windows server to run on. It needed it, because starting with version 6.5, vSphere Update Manager is no longer dependent on Microsoft Windows. Update Manager 6.5 is embedded in vCenter Server Appliance 6.5 and uses the internal vPostgres database. The Windows offering still exists, but now there is a choice between going all Windows or all Linux.

From the point of view of functionality, one of the features Update Manger offers is orchestrated ESXi host updates. Since I got a few questions related to host updates, I will describe in the post how to do the update. In my case I updated from vSphere 6.5 to 6.5a. Not a big step, but enough to prove a point.

I've started by downloading the latest ESXi ISO from VMware site, then logged in vCenter Web Client and uploade the image in Update Manager. In Web Client go to Home - Update Manager - Manage tab - ESXi Images and select Import ESXi Image. Browse to the location of the ISO image and upload it.

After the upload is successful, the image will be displayed in the list imported images. Select it and you can see the software packages that are included in the image:

Now the desired image is in the repository. To use the image, we need to attach it to a baseline. Then the baseline will be attached to the hosts or clusters that we want to update. Right click on the image and select create baseline, type the name of the baseline, optionally add a description and press OK. 

The baseline is displayed in Hosts Baseline tab:

Attach the new baseline to cluster of hosts. In Web Client go to Home - Hosts and Cluster, right click cluster, in the action list go to Update Manager and select Attach Baseline. Select the appropriate baseline an click OK. Next, go to Update Manager tab and press Scan for Updates to check the compliance status of the hosts in the cluster.

After the scan finishes, the Non-Compliant message is displayed and we can start the remediation process. Pressing Remediate starts the wizard. First, select the baseline to apply (there might be other baselines attached to the hosts). 

Then select the targets, in my case I've selected all the hosts in the cluster:

Accept the EULA, go to Adanced options and schedule the remediation to take place during a specific maintenance window or whether to ignore warnings that may appear during update. In this case we'll run the remediation immediately:

On Host remediation options select what to do with the VMs when hosts are put into maintenance mode: leave them on, suspend or power off. Removable media mounted to VMs can be automatically disabled. In case entering maintenance mode fails, we can specify how many times to retry and how long to wait between retries. The settings can be saved as default host remediation options for future upgrades.

Finally, the Cluster remediation options enables changes at cluster level - disable DPM, disable Fault Tolerance, disable HA admission control. Powered off and suspended VMs can be migrated to other hosts in the cluster while a host enters maintenance mode. Finally you may select to enable parallel cluster remediation and to specify how many hosts to process in parallel or let Update Manager decide the number based on the cluster settings. Parallel remediation is applied only on hosts where VMs are powered off or suspended. Also, VSAN cluster allows only for one host to be in maintenance mode at a time. Hosts in VSAN cluster will be updated sequentially.

If remediation is sequential and one of the host fails to enter maintenance mode, Update Manager will report an error and stop the remediation process. When parallel processing is selected, if Update Manager encounters an error, it ignores the host and it continues with the next host in the cluster. 

On the summary page you can run a a pre-check remediation report which will provide information about issues with the environment and will suggest what changes to apply:

Press OK and the remediation process will start. At the end of the process, the status of overall compliance in Update Manger tab will display Compliant.