Thursday, March 7, 2019

vCenter Server Restore with Veeam Backup & Replication

Recently I went through the process of testing vCenter Server appliance restore in the most unfortunate case when the actual vCenter Server was not there. Since the tests were being done for a prod appliance, it was decided to restore it without connectivity to the network. Let's see how this went on.

Test scenario
  • distributed switches only
  • VCSA
  • Simple restore test: put VCSA back in production using a standalone host connected to VBR
Since vCenter is "gone", first thing to do is to directly attach a standalone ESXi host to VBR. The host will be used for restores (this is a good argument for network team's "why do you need connectivity to ESXi, you have vCenter Server"). The process is simple, open VBR console go to Backup Infrastructure and add ESXi host. 

You will need to type in the hostname or IP and root account. Since vCenter Server was not actually gone, we had to use the IP  instead of FQDN as it was seen through the vCenter Server connection with the FQDN. 

Next, start the an entire VM restore

During the restore wizard, select the point in time (by default last one), then select Restore to a different location or with different settings:

Make sure to select the standalone host:

Leave default Resource Pool and datastores, but check the selected datastore has sufficient space. Leave the default folder, however if you still have the source VM change the restored VM's name:

Select the network to connect to. Actually disconnect the network of the restored VM. That was the scenario, right? Since the purpose of this article is not to make you go through the same experience we had, let's not disconnect it. And you will see why immediately:

Keep defaults for the next screens and start the restore (without automatically powering on the VM after restore). 

A few minutes later the VM is restored and connected to the distributed port group. 

We started by testing a disconnected restored VM, but during the article we didn't disconnect it. And here is why: when we initially disconnected the network of the restored VM, we got an error right after the VM was registered with the host and the restore failed. 

Same error was received trying to connect to a distributed portgroup configured with ephemeral binding. The logs show the restore process actually tries to modify network configuration of an existing VM and that makes it fail when VBR is connected directly to the console.When the portgroup is not changed for the restored VM, then the restore process skips updating network configuration. Of course, updating works with standard switch port group. 

In short, the following restore scenarios will work when restoring directly through a standalone host:
  • restore VCSA to the same distributed port group to which the source VM is connected
  • restore VCSA to a standard portgroup

Tuesday, February 19, 2019

Running Veeam PowerShell Scripts in Non-Interactive Mode - Credentials

This time we get back to some PowerShell basics and how to run scripts in non-interactive mode.

Veeam Backup & Replication and scripting go hand in hand really well. And the first thing you do when running a script is to connect to backup server

Connect-VBRServer -Server $ServerName -User $User -Password $Pass

Placing user and pass in clear text in a script may not be the best option. We could use PSCredential object to hold the username and password.

$PSCredential = Get-Credential
Connect-VBRServer -Server $ServerName -Credential $PSCredential

However getting the credentials in the objects implies using Get-Credential cmdlet which is interactive and will prompt to type the user and password. This makes it hard to run the script in non-interactive mode.

To make it non-interactive we need password saved somewhere. To save and encrypted password we can use ConvertFrom-SecureString cmdlet and pipe the output to a file:

(Get-Credential).Password | ConvertFrom-SecureString | Set-Content $encPassFileName

Since the username could be stored in the script itself, we retrieve only the secure string for the password, pipe it to the encrypting cmdlet and then output it to a file. Opening the output file will list something similar to:


Now we need to create the PScredential object:

$password = Get-Content $encPassFileName | ConvertTo-SecureString 
$psCredential = New-Object System.Management.Automation.PsCredential($username,$password)

First, we loaded the content of the file storing the encrypted password and convert it to a secure string (PScredential objects use secure strings). Next we created the object with the username and the password. The generated object can be used to connect to VBR Server

Connect-VBRServer -Server $ServerName -Credential $PSCredential

By default, ConvertFrom-SecureString encryption is done using Windows Data Protection API (DPAPI). For this reason the file cannot be moved to another computer and used from that one. On each computer from where the script is being run, the password must be encrypted separately.

Wednesday, January 30, 2019

NSX-T Part 1 - Architectural Overview

This is is the first in a series of blogs about NSX-T installation and configuration. Although similar in concepts, NSX-T bring a series of differences from NSX-V. One of them is that it is independent from vCenter Server. And this brings also a lot of changes from what I was used to - some changes are architectural, others are related to functionality and user interface.

The blog series will start with architectural overview, then go to deployment of NSX-T Manager and configuration of controllers, followed by basic configuration of NSX-T environment. Once we get to use NSX-T overlay network between two VMs, we will continue with exploring routers and edges.

At a high level, the architecture is made of the same 3 components as in NSX-V:
  • management plane - single API entry point, user configuration, handles user queries, and performs operational tasks on all management, control, and data plane nodes in the system
  • control plane - computes all ephemeral runtime state based on configuration from the management plane, disseminates topology information reported by the data plane elements, and pushes stateless configuration to forwarding engines
  • data plane - stateless forwarding/transformation of packets based on tables populated by the control plane and reports topology information to the control plane

Management Plane Agent (MP Agent) is a component that runs on controllers and transport nodes (hypervisors) and is responsible for executing tasks requested by Management Plane:

  • desired state of the system
  • message communication to management plane (configuration, statistics, status, real time data)
Control plane is made of Central Control Plane (CCP) running on the Controller cluster and the Local Control Plane (LCP) running on transport nodes. CCP provides configuration to other controllers and LCP and it is detached from data plane (failure in CCP will not impact data plane). LCP is responsible for most of the ephemeral runtime state computation and pushing stateless configuration to forwarding engine in data plane. LCP is linked to the Transport Node hosting it.

Data plane's forwarding engine performs stateless forwarding of packets based on tables populated by control plane. Data plane maintains state for features like TCP termination, but this state is about payload manipulation and is different from the state maintained at Control Plane's level which is about how to forward packets (see MAC:IP tables)

N-VDS (NSX Managed Virtual Distributed Switch or KVM Open vSwitch)
N-VDS provides traffic forwarding. It is a component presented as a hidden distributed switch (it will not appear in vCenter Server for example). It is always created and configured by NSX manager in vSphere environments. It can be predefined in KVM. This is a major change from NSX-V. It also implies that you will need to have at least one not utilized vmnic on the ESXi host for the N-VDS. 
Another change is that the encapsulation protocol used now is GENEVE (latest IETF draft here)

Virtual Tunnel Endpoint (VTEP)
VTEP is the connection point at which the encapsulation and decapsulation takes place.

Transport Node
A node capable of participating in an NSX-T Data Center overlay or NSX-T Data Center VLAN networking. 

Transport Zone
Collection of transport nodes that defines the maximum span for logical switches. 

Uplink Profile
Defines settings for the links from hypervisor hosts to NSX-T Data Center logical switches or from NSX Edge nodes to top-of-rack switches (active/standby links, VLAN ID, MTU).

Logical switches - provide Layer 2 connectivity across multiple hosts 

Logical router - provides connectivity North-Sourh enabling access to external networks and East-West for networks within the same tenant. A logical router consists of two optional parts: 
  • a distributed router (DR) - one-hop distributed routing between logical switches and/or logical routers connected to it
  • one or more service routers (SR) - delivers services that are not currently implemented in a distributed router
A logical router always has a DR, and it has SR if any of the following is true:
  • it is a Tier-0 router, even if stateful services are not configured
  • it is a Tier-1 router linked to a Tier-0 router and has stateful services configured (such as NAT)
NSX Edge - provides routing services and connectivity to external networks as well as non-distributed services. 

In the next post will see how to install NSX-T manager and how all the above mentioned concepts are coming together when configuring the manager 

Wednesday, January 23, 2019

Role Based Access for VMware in Veeam Backup & Replication 9.5 Update 4 - Advanced Backup Features and Recovery

In previous post we looked at how to configure the integration between Veeam and vCenter Server and how to create a simple backup job in self service portal.

In this post we'll go further and look at advanced configuration options available for backup jobs as well at recovery features.

1. VM restore
The user restore the VMs he is in charge of . Login to self service portal (https://em-address:9443/backup) and go to VMs tab. Select the VM you want to restore and press Restore. Two choices are given - overwrite and keep original VM.

Once the restore point has been selected and whether to power on or not the VM, the wizard starts the restore process to production environment. Choosing "keep" will result in a new VM being created with the name: originalVMname_restoredDateTime
In this case pay attention not to power it on after restore as it is connected to the same network with the original VM. If you choose "overwrite", powering on maybe a good idea. 

2. Guest file system indexing and file restore

When enabled,  VBR will create an index of files and folders on the VM fuest OS during backup. Guest file indexing allows you to search for VM guest OS files inside VM backups and perform 1-click restore in the self service portal.Initially the indexes are created on VBR server, but later on moved to Enterprise Manager server.

Let's take our job created initially and add guest indexing to it. Logon the self service portal, go to jobsa and edit the job and on guest processing tab tick the boc "Enable guest file system indexing".

Add user credentials and if different credentials are needed for VMs in the job press customize credentials. Lastly, define what to index. By default everything is indexed except system folders (Program files, Temp, Windows). You can select to index only specific folders or to index all folders.

Save the configuration and run the backup job. Once the job runs successfully, in self service portal  got to Files tab and type in the name of the file you want to find. In my case, the indexed VM is running IIS  so I will search for iisstart.htm file.

After finding the file we can now restore the file directly to production VM overwriting the originla file or keeping it. The file can be also downloaded to the local computer in an archive having the name FLR_DATE_HOUR

If we have more files to restore, we can add the file to a Restore list and execute the restore on multiple files at a time:

File level restore is possible without indexing, but it needs to mount the backup and let you manually search the file in the file system.

3. Application item restore

If using Veeam explores allows for a very good granularity and flexibility regarding supported applications, the self service portal is restricted to MSSQL and Oracle applications. On the Items of the self service portal, select the database, the location where to restore (original or alternate) and the point in time.

We'll restore the DB on the same server, but with a different name. For this reason choose restore to another location and press Restore. Specify the server name and the credentials for connecting to admin share and MSSQL DB

Specify the MDF and log file names - change them is the same location with original DB is being used:

Press Finish and watch the progress in the status bar:

When it is finished, you can connect to MSSQL server and check the DB has been created:

The new integration with vCenter Server offers a powerful tool for job management delegation to the users others than backup administrators. 

Tuesday, January 22, 2019

Role Based Access for VMware in Veeam Backup & Replication 9.5 Update 4

One of the cool features that Veeam Backup & Replication 9.5 Update 4 comes with is integration with vCenter Server role based access. What does it mean? It allows to delegate permissions to users and groups in Veeam Enterprise Manager based on their permissions in vCenter Server.

A user or group of users is now able to monitor and control the backup and restore of their own VMs in a vSphere environment based on a predefined policy. A policy can be defined through vSphere tags, a role in vCenter or as granular as a single permission in vCenter Server. Delegation is done through the self service portal in  Enterprise Manager.

Cool thing is the integration is actually extending vCenter Server access control by adding vSphere tags as control mechanisms. For example DBA's want to do their backup and restore, then just assign a tag to the VMs and create the policy in Enterprise Manager. It's that simple.

Since my environment uses tags, we will test the following scenario: all developers will have access to development VMs which are tagged with "SLA3" vSphere tags.

First, make sure tags exist in vCenter Server and are assigned to the VMs in scope.

Next, you need to install and configure VBR and Enterprise Manager (EM). This is not in the scope of the current article.

Once EM is installed and configured  login to the EM portal (https://em-address:9443/) and go to Configuration. Check VBR and vCenter Server are available and reachable.

On Self Service tab you will see the default configuration for all Domain Users group (my lab is AD integrated).

For the test we will create a new configuration.

Let's look at the Delegation Mode - what mechanism we use to define access:

By default, it uses VM privilege and the selected privilege is VirtualMachine.Interact.Backup, but you can choose to use any privilege available in vCenter Server. Once you need more flexibility, you can define roles in vCenter Server and use the delegation based on those roles (a set of privileges applied to an object). Finally, you can use vSphere tags and allow access based on the tags. Once the preferred method of delegation is chosen, it will apply to all self service configurations. So be careful when changing the method. 

Now open, the default self service configuration and let's take a look at it. We see it assigns a repository and a quota on that repository. The quota can be global (for all users in group) or individual (per each user). 

You also define the default settings for advanced job settings such as compression , deduplication, scheduling of full backups and so on. The settings can be copied from Veeam default or from an existing job. 

There are 4 job scheduling options available from allowing full access to the scheduler to no access to the scheduler. We will use the default one (full access to job scheduling). Choose wisely what you want your users to do.

vSphere tags drop down list appeared because I chose the Delegation Method as vSphere tags, but it's left empty. 

Let's create a new self service configuration for developers group. Press Add and then:
1. select the Type - user or group and search it in AD
2. select the repository and define the quota
3. select job scheduling options
4. select the vSphere tag
5. configure the advanced settings

Press Save and open the self service portal (https://em-address:9443/backup/). Login with one of the users from the group (Developers in my case). Since user is member of 2 configurations, select which configuration to logon:

Once logged in, the portal displays 5 tabs: Dashboards, Jobs, VMs, Files and Items

Go to Jobs and create a new backup job. A process similar to VBR console job creation will start. First give the job a name, description and decide how many restore points to keep:

Then add the VMs you want to backup. Only VMs with SLA3 tag will be displayed.

If required, enable application aware processing and add credentials for guest processing:

Schedule the job (in this configuration you are allowed):

Lastly, enable notifications (if you want to be alerted about the status of the job)

Now that the job has been created, you can run it:

Meanwhile, we can take a look at how things look in VBR console

You'll notice the running job named with the following convention: Domain\GroupName_Domain\UserName_JobName. So, all is good and running smoothly.

Back in the self service portal, pressing on the job statistics we can see what happened:

There you go, just had a user define a backup job and backup its own VM using a simple vSphere tag and no other settings at either vCenter Server or VBR level. Next time we'll take a look at restore options.

Tuesday, December 4, 2018

Configure Veeam Backup for Microsoft Office 365

In the previous post, we've gone through the steps to install VBO365 on a Windows Core VM. Now we'll look at configuration.

After installation, a default backup proxy and default backup repository are created on the server. Proxies are responsible for handling backup traffic from Office 365 to repository and restore traffic from repository to Veeam explorers. For small deployments the same server (management server) can be used as proxy and repo server. However it is recommended to use external proxies and repositories.

One very important aspect of the repository is that it defines the retention policy. This means that all backup jobs pointing to one repository will get the retention defined at that repository level.

More important, in VBO365 retention policy is defined as the number of years/days since the object has been changed. In the case of a 3 year retention policy, only e-mail that have been touched in the past 3 years will be backed up. In case all emails are required for a certain mailbox, keep forever policy can be selected.

Next we will add the O365 organization that we want to backup

Once the organization has been added, we can create backup jobs. Although it is possible to backup everything in one job, it is recommended to create separate backup jobs for Exchange, SharePoint , Archive and OneDrive and point them to different repositories. 

Next we'll create a backup job for emails of only 4 users in the organization. Start the wizard and give the job a name:

Select the users you want to backup:

Select the items to backup for the users (by default All, we'll select only e-mails):

Add exclusions (in our case not required):

Choose the proxy and repository (remember the repo sets the retention):\

Confirm the job execution schedule:

The job will be created and stopped. Right click on it and start to backup. 

Once the backup finishes, you can right click on the job and open Exchange explorer

Tuesday, November 27, 2018

Enable RDP in SLES 12

Haven't been working with SuSE Linux for a while and when this new project came along I got my hands on SLES 12 SP3.

I've created an account on, downloaded the ISO, created the VM and installed from scratch. It went pretty smoothly. The suse account gives a 60 day trial activation key which will allow to update the server. After the 60 days, no more update, but it's just for a lab setup.

The one thing I need from the server is to provide terminal services. So I installed xrdp from using Software Manager in Yast 2

Next, enable xrdp service and start it:
sudo systemctl enable xrdp
sudo systemctl start xrdp

Last thing to do is to configure the firewall. It is enabled by default and will not allow connection to the server. Check that the xrdp service definition exists. If not create a file that looks like this:

Configure the firewall to accept xrdp (and if you are at it, add ssh also). Open the file /etc/sysconfig/SuSEfirewall2 and change the following lines to include your services:

Save the changes and restart the firewall service:
sudo systemctl restart SuSEfirewall2

Now, if all went good open your preferred RPD client and give it a try: