Thursday, August 17, 2017

vRealize Business update checking hangs

Trying to update vRealize Business 7.2 I noticed that it hangs during update check with the message "Checking for available updates...".

No matter what I did - changed the update options, reboot the appliance - didn't help. Looking on I found the post 477322 which presented the same behavior, only for vCenter Server Appliance and it was from 2014. So I tried (a bit skeptically) to start the update from CLI as suggested in the post:

  • ssh to vRB appliance 
  • /opt/vmware/bin/vamicli update --check
  • /opt/vmware/bin/vamicli --install latest --accepteula
And it worked:

I suppose this workaround can be applied to most of the appliances. 

Monday, August 14, 2017

New and cool features in vRealize Automation 7.3 - parameterized blueprints

vRealize Automation 7.3 has been released for a few months, but only last week I got time to update the lab and take a more serious look at it. And I really liked what I saw.

One of the first features that I saw made me smile because I remembered the times when clients requested it and I was going through the processes of explaining that it is possible, but it will need some customization and some workflow development. Now, in 7.3 there are parameterized blueprints which allows to define t-shirt sizes for VMs, It also provides image parameters - how that image is being built. This way the configuration of the VM can be hidden from the service consumer and using image parameters organization policies can be implemented without the need to create new blueprints. You could have only one Windows template that provides 2012, 2012 R2 and 2016 as simple as selecting from a drop down menu. 

In the end, the user gets the possibility to select whatever OS version is entitled to and the t-shirt size of that deployment.   

Let's see how we can get to this nice item request screen.

First we define the parameters. In vRA portal, go to Administration -> Property Dictionary -> Component Profiles. There are two component profiles already defined: Image and Size. 

We need to edit each one. By default, the component profiles have no values defined. Edit Size profile where we will configure CPU, memory, storage for vSphere virtual machines to be used in blueprints. Go to Value sets tab and press New:

Type in a display name (what the service consumer will see in the request form), a description (optional), configure the values for CPU, memory and storage and select the status (by default it is active and it can be used in blueprints). Press Save if you want to add more value sets or Finish to save and exit. Once a value set has been defined, it can be edited, deactivated or deleted.

Now let's define value sets for Image component profile and configure the build information for vSphere VMs. It is the same build information that traditionally configured at each blueprint's level, but this time it is defined as a series of parameters. Edit Image component profile, go to Value Sets tab and press New:

Type in the name display name, a description and then select how to build the VM. In my case, I've  selected Linked Clone and filled in the necessary parameters: VMware template to clone from, the snapshot to use, customization specification name. All other options existing in a blueprint still available:  create, clone, NetApp flex clone. 

Once we have defined the value sets and made them active, we can use them in blueprints. 

Go to Design - Blueprints and there is the choice to either modify and existing blue print or create a new one. I will modify an existing blueprint, since I want to reduce blueprint sprawl :-) 
Go to Design - Blueprints - Edit, in the blueprint select the vSphere machine component and on the tab go to  Profiles.

By default, no component profiles are selected, press Add and select which component profiles to use: Size, Image or both. 

Press OK and select from each component profile the value sets to use for this particular blueprint. I've used only two of the t-shirt sizes (large and xlarge) and selected large to be the default one:

Press Finish to save and exit. Since the blueprint was already published, we can go directly to Catalog and request the Item using the new t-shirt sizes. For a new item, you need first to publish it, map it to a service and ensure the users are entitled to request the item. 

Thursday, May 4, 2017

Virtual Machine Encryption

A new security feature introduced in vSphere 6.5 is virtual machine encryption. The encryption is VM agnostic as it takes place at hypervisor level before the I/O is stored to disk. It uses vSphere APIs for I/O filtering framework that allows interception of VM I/Os in the virtual SCSI emulation (vSCSI) layer. It encrypts virtual machine files (nvram, vswp), virtual disk files and core dump files. However it does not support log files, VM configuration files or virtual disk descriptor files since these are considered to contain non-sensitive data.

How it works

There are several components necessary to implement VM encryption. The process uses two different sets of keys - key encryption keys (KEKs) and data encryption keys (DEKs). The components are:
  • external key management server (KMS) - generates and stores key encryption keys (KEKs) 
  • vCenter Server - requests KEKs from KMS and distributes them to ESXi hosts; Key Management Interoperability Protocol (KMIP) v1.1 is supported
  • ESXi hosts - generates data encryption keys (DEKs) and encrypts them with KEK; encrypted DEKs are stored in configuration files. DEKs are used to encrypt/decrypt virtual machine files. KEKs need to be in ESXi memory for a VM to be powered on.

Since KEKs are only stored in KMS and they are used to encrypt/decrypt DEKs, KMS should be made highly available. Loosing KMS generated keys means DEKs cannot be decrypted and access to VM data is gone.

Another important aspect is VM encryption uses the data block's address to protect against snooping by generating different encrypted data for identical data blocks. However it does not provide protection against data corruption.

How to configure

First we need to configure KMS solution. For demo purpose I've used the docker container created by William Lam. Please note that the keys for this KMS are held in memory and they will be lost on restart. To configure it, logon to your docker host and start the docker image with KMS by running the following commands:
docker pull lamw/vmwkmip
docker run -d -p 5696:5696 law/lamw/vmwkmip

Check the container is running by executing the following command on the docker host: docker ps. 

Next configure vCenter Server. Login to web client, select the vCenter Server in Hosts view, go to configure tab, Key Management Servers and press Add KMS. In the window add the KMS cluster name, server alias, server address, TCP port and optionally proxy details:

Press Yes to set the KMS as your default KMS cluster:

Trust the certificate presented by KMS:

vCenter Server is now configured to use KMS and the details are displayed in the web client:

To encrypt a VM becomes a matter of applying the correct storage policy to the VM. Before applying the encryption policy make sure the VM is powered off, otherwise you will get the following error:

To change the storage policy, in web client right click the VM you want to encrypt, go to VM Policies -> Edit VM Storage Policies. Change the default policy with VM Encrpyption Policy (which is the default encryption policy) and press OK button:

The encryption process will take some time. You can monitor it in events log:

Once the process is finished you can power on the VM. On the summary tab of the VM you can also check that the VM is encrypted:

Access control
Since cryptography is used when one needs to restrict access to certain data, the question is do we need all admins to have access to cryptographic functions in vCenter Server. To restrict access, a new role has been introduced - No cryptography administrator. It does not have the following privileges: 

  • Cryptographic Operations
  • Global.Diagnostics
  • Host.Inventory.Add host to cluster
  • Host.Inventory.Add standalone host
  • Host.Local operations.Manage user groups

To further restrict the access, the role can be cloned and modified accordingly.

The are restrictions and limitations when VM encryption is being used. One of the most important is that backup solutions using VMware vSphere Storage API - Data Protection are restricted to hot add backup and NBD-SSL network transport mode. SAN backup is not supported.

VMs with existing snapshots cannot by encrypted. All snapshots must be first consolidated. Guest memory cannot be saved during a snapshot of encrypted VMs.

If you are looking for details on performance impact of VM encryption, there is a performance study from VMware. Dedicated encryption hardware is not necessary, but using a processor that support AES-NI instruction set will speed up encryption and decryption.

Tuesday, April 18, 2017

Updating vSphere 6.5 with vSphere Update Manager

Update Manager has been and is the tool for upgrading and patching ESXi hosts, virtual appliances and VMs. There was only one caveat: it needed a Windows server to run on. It needed it, because starting with version 6.5, vSphere Update Manager is no longer dependent on Microsoft Windows. Update Manager 6.5 is embedded in vCenter Server Appliance 6.5 and uses the internal vPostgres database. The Windows offering still exists, but now there is a choice between going all Windows or all Linux.

From the point of view of functionality, one of the features Update Manger offers is orchestrated ESXi host updates. Since I got a few questions related to host updates, I will describe in the post how to do the update. In my case I updated from vSphere 6.5 to 6.5a. Not a big step, but enough to prove a point.

I've started by downloading the latest ESXi ISO from VMware site, then logged in vCenter Web Client and uploade the image in Update Manager. In Web Client go to Home - Update Manager - Manage tab - ESXi Images and select Import ESXi Image. Browse to the location of the ISO image and upload it.

After the upload is successful, the image will be displayed in the list imported images. Select it and you can see the software packages that are included in the image:

Now the desired image is in the repository. To use the image, we need to attach it to a baseline. Then the baseline will be attached to the hosts or clusters that we want to update. Right click on the image and select create baseline, type the name of the baseline, optionally add a description and press OK. 

The baseline is displayed in Hosts Baseline tab:

Attach the new baseline to cluster of hosts. In Web Client go to Home - Hosts and Cluster, right click cluster, in the action list go to Update Manager and select Attach Baseline. Select the appropriate baseline an click OK. Next, go to Update Manager tab and press Scan for Updates to check the compliance status of the hosts in the cluster.

After the scan finishes, the Non-Compliant message is displayed and we can start the remediation process. Pressing Remediate starts the wizard. First, select the baseline to apply (there might be other baselines attached to the hosts). 

Then select the targets, in my case I've selected all the hosts in the cluster:

Accept the EULA, go to Adanced options and schedule the remediation to take place during a specific maintenance window or whether to ignore warnings that may appear during update. In this case we'll run the remediation immediately:

On Host remediation options select what to do with the VMs when hosts are put into maintenance mode: leave them on, suspend or power off. Removable media mounted to VMs can be automatically disabled. In case entering maintenance mode fails, we can specify how many times to retry and how long to wait between retries. The settings can be saved as default host remediation options for future upgrades.

Finally, the Cluster remediation options enables changes at cluster level - disable DPM, disable Fault Tolerance, disable HA admission control. Powered off and suspended VMs can be migrated to other hosts in the cluster while a host enters maintenance mode. Finally you may select to enable parallel cluster remediation and to specify how many hosts to process in parallel or let Update Manager decide the number based on the cluster settings. Parallel remediation is applied only on hosts where VMs are powered off or suspended. Also, VSAN cluster allows only for one host to be in maintenance mode at a time. Hosts in VSAN cluster will be updated sequentially.

If remediation is sequential and one of the host fails to enter maintenance mode, Update Manager will report an error and stop the remediation process. When parallel processing is selected, if Update Manager encounters an error, it ignores the host and it continues with the next host in the cluster. 

On the summary page you can run a a pre-check remediation report which will provide information about issues with the environment and will suggest what changes to apply:

Press OK and the remediation process will start. At the end of the process, the status of overall compliance in Update Manger tab will display Compliant. 

Monday, April 10, 2017

Veeam VMware vSphere Web Client Plug-in

Somehow I keep on getting back to restarting this activity, which is very rewarding, but also time consuming. Since I've accepted a new position with another company, I am reviving the blog. Again. I'm starting to see a pattern here. 

First, let me start with a disclaimer: the posts and opinions herein are my own. Not representing the company I work for, or the vendor, or anything official.

First topic on the list is Veeam vSphere Web Client Plugin. I've chosen this because it integrates Veeam Backup & Replication console (VBR from now on) with vSphere Web Client allowing the virtualization admin to have an understanding of what is going on with "those backups".

The plugin provides the following widgets:
- VMs Overview - details about protected VMs, restore points, source VM size, backup size
- Processed VMs - backed up and replicated
- Repositories - capacity, bakup size, free space
- Jobs statistics - running jobs,  successful jobs, errors

In addition to all the information displayed, it also enables the virtualization admin to run VeeamZIP (full backup file that acts as an independent restore point) and QuickBackup (on-demand incremental backup - needs the existence of a full backup) tasks.

Now let's see how we get the plugin installed and configured.

First, we need to have Veeam Enterprise Manager (EM) installed in the infrastructure. It is not usually installed since EM is utilized to manage multiple VBR installations from a single console. Having only one VBR instance doesn't make a use case for EM, unless you are using encryption in your environment. In this case, EM is used in the encryption/decryption process. There are a few other considerations for Enterprise Manager, one them being deploying vSphere Web Client Plugin. The installation of EM is not covered in this article. It needs a Windows server and SQL database. In our demo lab, EM was installed on the same server with Veeam ONE using the already deployed SQL Express.

After installation, open EM console and login. Go to Configuration and add VBR servers that you will manage from EM:

Next, go to vCenter Server tab and you will see the list of Veeam managed servers populated. Looking at the plugin status you will see "Unknown" meaning no check has been done for that particular vCenter Server:

In the above image, there are 2 servers that have already been configured with the plugin. For those it shows the status "Installed" and the plugin version. In order to install the plugin on a new vCenter Server, select the server and then press Check version. Fill in username and password for connecting to vCenter Server and press ok. The status will change from "Pending" to "Not installed".

You will notice the "Install" button is activated. Press "Install" and wait for the status to change to "Installed":

Next, login to vSphere Web Client and on you should see the plugin on the home page of vCenter Server client:

The plugin displays two tabs - Summary and Settings. The Summary tab contains four widgets:
- VMs Overview

- Jobs statistics
- Processed VMs

- Repositories 

Settings tab allows configuration of EM server, the user account for web plugin to connect to EM and Veeam ONE (if it exists in the infrastructure). 
When Veeam ONE is configured, predefined reports can be accessed directly from the widgets. For example the Repositories widget provides a link to Capacity Planning for Backup Repositories report:

Finally, the web client plugin implements VeeamZIP and Quick Backup functionality in the VM context menu, empowering the virtualization administrator to create full and incremental on demand backups. 

Before being able to run the backup actions, we need to configure the plugin by selecting VeeamZIP to.... Next you configure the destination backup server, repository, retention policy and compression level 

After selecting the desired values, press VeeamZIP button to save the configuration. From now on, you can run backups directly from vSphere Web Client. 

Wednesday, March 29, 2017

Veeam Backup and Replication: Offsite Backup Repository

One of the common scenarios for backup infrastructures is to send the local backups to a secondary site. In a Veeam Backup and Replication (VBR) environment, this can be easily done by deploying a backup repository in the secondary location and then configuring backup copy jobs for the backups that you want to be sent offsite. VBR  v9.5 supports 4 types of backup repositories:

  • Windows server with local or direct attached storage - local disk, USB drive, iSCSI/FC LUN
  • Linux server with local, direct attached storage or mounted NFS - local disk, USB drive, iSCSI/FC LUN, NFS mounts
  • CIFS (SMB) share
  • deduplicating storage appliance - only the following three are supported EMC Data Domain, ExaGrid and HPE StoreOnce and there is also Enterprise license required.
For the current implementation, the chosen solution is implemented in a VMware environment across two vCenter Servers. VBR Server and main repository are located in the primary site. In the secondary site a backup repository has been installed on top of a Windows VM. Data mover service is installed in both sites. 

Having a data mover service in secondary site also enables backups directly to secondary site. Now, let's see how to configure the offsite repository.

First we need to deploy the Windows VM. The process is "standard" procedure: reserve an IP address, deploy from template, select VM name, compute resource and storage, customize the guest OS (including joining to AD).

After the VM has been deployed, configure the VM hardware if necessary: repository space depending on the size of the backups and RAM (4GB for OS and up to 4GB for each concurrent backup job).

Once the Windows VM is up, go to VBR management console, backup infrastructure tab and start repository configuration wizard by right clicking on Backup Repository -> Add backup repository.
Add a name and a description for the new repository:

Select the type of repository (Windows, Linux, CIFS, Appliance):

On the repository server list page press "Add new"

This will open a new wizard that configures a new windows server repository. Add DNS name or IP address of the repository server (optionally a description):

Add the credentials to use for connecting to the VM. If you've saved them in the credential manager select them from the drop down list, otherwise click Add button and enter the username and password.

Review the components to be installed and press Apply. The wizard will install VBR components on the repository server and it displays the progress in the window:

A summary page is displayed with info about the target server:

Once server is configured the  new backup repository is displayed in the list of servers. Press Populate button to retrieve all the available storage locations. Select the appropriate storage, press Next.

Configure the repository parameters: backup folder path, maximum number of concurrent tasks, read and write data rates (if necessary). 

Advanced configuration features are realated mostly to storage appliances. Pressing Advanced button allows to select the following:
  • Align backup file data blocks - useful for better deduplication ratios on storage appliances that use constant block size deduplication
  • decompress backup data before storing - achieves better deduplication ratio on most storage appliances at the cost of performance
  • this repository is backed by rotated hard drives - if hard drives are rotated and removed from the server
  • user per-VM backup files - multiple I/O streams per VM will improve performance with storage appliances 
Next, select the mount server and whether to enable vPower NFS or not. Default TCP ports for mount server and vPower NFS could be changed if necessary (press Ports button).

Review the configuration page of the server and press Apply.

The summary page will display the tasks, the progress and their status - creating repository folder, installing components (mount server, vPower NFS), configuring components. 

The following services are installed on the repository server:
  • Veeam Data Mover Service - sends and receives data 
  • Veeam Mount Server - mounts backups and replicas for file-level access
  • Veeam Installer Service - installs, updates and configures VBR components
  • Veeam vPower NFS Service - enables running VMs directly from backup files by "publishing" VM vdmk's from backup files to vPower NFS datastore. The datastore is then mounted on the ESXi host.
Once the process finishes, the new repository appears in the repository list of the VBR console and it is ready to use. Use it as a destination for a copy job or as an offsite backup job.