Clear DNS Cache on VCSA after ESXi IP Address Update

I've recently had to do some changes and modify the ESXi management IP address. Once the ESXi host has been put to maintenance mode and removed from vCenter Server inventory I've updated the DNS server records: A and PTR records. Checking DNS resolution works, I've tried to re-add servers to vCenter Server using their FQDN, but it errored with no route to host. This is because of the DNS client cache on VCSA. 

To solve it fast and not wait too long you need to ssh to VCSA appliance and run the following commands: 

systemctl restart dnsmasq

systemctl restart systemd-resolved

Once the services are restarted you can add again the ESXi hosts using the FQDN.

PowerCLI - Optimizing Scripts

Not sure when and how this drive to optimize appeared in me, but I've seen it taking control in different situations and driving other people crazy. So I thought why not give it a try on the blog also. Maybe something good comes out of it. 

I will go over a couple of simple concepts that will help with the execution of PowerCLI scripts. The first one is API calls and to be more specific the number of calls. 

Let's use the following example: 

• we have a list of VMs and we need to get the total used space by those VMs

The use case can be approached in two ways:

• get the size for each VM from the list and make the sum 

foreach ($vmName in $vmList) {
	$totalUsedSpace  += (Get-VM $vmName).UsedSpaceGB
}

  

In this case we would use Get-VM cmdlet for each VM in the list. When we use Get-VM we actually make an API call to vCenter Server. So that means for a list of 10 VMs we do 10 API calls. If we do this with 100 VMs that means 100 calls. The immediate effect is that the script takes a long time to execute and we increase the load on vCenter Server with each call. 

But we can get all VMs from vCenter Server in one call and then go through the VMs

• get all VMs in vCenter Server and then check each VM in the list 

$allVms = Get-VM

foreach ($vm in $allVms) {
	foreach ($vmName in $vmList) {
		if ($vm.Name -eq $vmName) {
			$totalUsedSpace  += $vm.UsedSpaceGB
		}
	} 
	
}

The advantage of the previous example is we do only one API call. The disadvantage is we take all objects and then we have imbricated "for" loops. And this can take a long time especially when there are a lot of objects in vCenter Server. Let's take a look at some execution times.

The data set is made of more than 6000 VMs in vCenter Server ($allVms) and we are looking for the size of a list of 100 VMs ($vmList)

For 100 VMs, the first script (on API call for each VM in list) takes around 275 seconds. The second script takes 14 seconds. Even if we do "for" in "for", it takes way less time than to do hundreds of API calls. Obviously, if we increase the $vmList size to 300 VMs the first script will take almost 3 times the time while the second will only increase by a few seconds to 16 seconds. That comes from the increased complexity of running the imbricate loops. The more we increase $vmList, the more time it will take. At 600 VMs in $vmList, it takes 21 seconds to run. At this moment we'd like to see if there is another way to actually decrease the complexity of the script and make it faster. 

Let's see how to get rid of the imbricated loops. For this to work, we'll use a hash table (key-value pair) created from vm names and their used space. Once the hash table is created then we search in it only for the vms in $vmList:

$allVms = get-vm 
$hashVmSize = @{}
foreach ($vm in $allVms) {
	$hashVmSize.Add($vm.Name,$vm.UsedSpaceGB)
}
$totalUsedSpace = 0 
foreach ($vmName in $vmList) {
	$totalUsedSpace  += $hashVmSize[$vmName]
}

For the new script the time to execute for the 600 VMs in $vmList is less than 12 seconds (almost half the time of the imbricated loops script). The complexity reduction comes from the number of operations executed. If we count inside the loops how many times it is executed, we'll see that for the hash table script we run the "for" loops for almost 7000 times, while for the second script (imbricated loops), we run the loop for more than 38 million times (yes, millions). 

What we've seen so far:

• calling the API is expensive - 1 call is better than 10 calls 
• imbricated loops are not scalable (not even nice) 
• hash tables can help

This doesn't mean that if you want to see the size of 2 VMs you should bring 6000 VMs from vCenter Server. It means that next time when your script takes 20 minutes, maybe there is a faster way to do it. 

vCenter Server Appliance 7.0 Command Line Installer

One of my favorite features in vSphere is command line install of vCenter Server. It first appeared with vCenter Server 6.0. It is based on a JSON file input and can be use to do a fresh install of vCSA 7.0, upgrade an existing vCSA 6.5 or 6.7 installation to 7.0 or migrate a Windows vCenter Server 6.5 or 6.7 to vCSA 7.0.

The installer can be run from Windows, Linux or Mac. To access it, you need the vCSA iso file and locate folder vcsa-cli-installer\win32 (Windows users). JSON templates are found in templates folder. You need to modify the JSON template that fits your use case. I will do a fresh install of vCSA 7.0 in my lab so I will be using the template embedded_vCSA_on_VC.json which deploys the new vCSA inside an existing vCenter Server. The template is commented very well, however I will post here an example of what a simple configuration looks like. Please be aware that this is just a snippet of the actual template and some parts have been left out for ease of reading.

    "new_vcsa": {
        "vc": {
            "hostname": "vcsa67.mylab.com",
            "username": "administrator@mylab.com",
            "password": "",
            "deployment_network": "VM Network",
            "datacenter": [
                "VDC-1"
            ],
            "datastore": "DATASTORE-1",
            "target": [
                "CLUSTER-1"
            ]
        },
        "appliance": {
            "thin_disk_mode": true,
            "deployment_option": "small",
            "name": "vcsa70"
        },
        "network": {
            "ip_family": "ipv4",
            "mode": "static",
            "system_name": "vcsa70.mylab.com",
            "ip": "192.168.100.1",
            "prefix": "24",
            "gateway": "192.168.100.254",
            "dns_servers": [
                "192.168.1.10"
            ]
        },
        "os": {
            "password": "",
            "ntp_servers": "0.ro.pool.ntp.org",
            "ssh_enable": true
        },
        "sso": {
            "password": "",
            "domain_name": "vsphere.local"
        }
    }

As you can see, once you create the template it can reused a lot of times. What for you may ask and one answer is nested labs. If you are unsure what size the vCSA should be, the installer will tell you:
.\vcsa-deploy.exe --supported-deployment-sizes

The installer takes different parameters besides the JSON file:
.\vcsa-deploy.exe install --accept-eula [--verify-template-only|--precheck-only][file_path_to_json] 

If you want to automatically accept SSL certificate thumbprint, you can add --no-ssl-certificate-verification parameter.

As seen above, the installer comes with 2 options that enable you to check that everything is fine before actually starting the install:

• verify-template-only - will run a JSON file verification to validate the structure and input parameters (e.g. password strength, IP address, netmask). The final check result is displayed along with the path to the log file. The log file contains all required details. For example if you typed an IP address that does not exist, the following message is displayed in log file:

2020-03-27 19:44:06,232 - vCSACliInstallLogger - ERROR - The value '192.268.100.1' of the key 'ip' in section 'new_vcsa', subsection 'network' is invalid. Correct the value and rerun the script.

• precheck-only - will do a dry run of the installer. This time it will connect to vCenter server and check that the environment values are actually correct: for example that you don't have another VM with the same name, vCenter objects are correct (datacenter, datastore, cluster or host). It also does a ping test to validate the IP/FQDN entered for the new vCSA are available.

================ [FAILED] Task: PrecheckTask: Running prechecks. execution

Error message: ApplianceName: A virtual machine with the name 'vcsa70' already

exists on the target ESXi host or cluster. Choose a different name for the

vCenter Server Appliance (case-insensitive).

Of course, you don't have to run both checks or even any check if you are confident enough. For me, precheck-only helped since I didn't understand how to fill in the JSON file from the first time (I will blame it on a barrier language). One very important aspect of installing is to have DNS records setup and working. If you don't, even if the prechecks and the actual install will work, first boot of vCSA will most likely fail.

Having all setup up and checked, you  just run the install command and that's it. I like the CLI installer because it is simple, powerful and repeatable. No more filling in fields in a GUI and waiting for the lines on the screen.

PBKAC and Nested ESXi VSAN

This post is a short one about a PBKAC (Problem Between Keyboard and Chair) and how I discovered it while setting up a nested VSAN cluster in my home lab.

I've recently decided to install a nested VSAN environment. I've created 4 nested ESXi hosts running 6.7 U3, added disks to them (flash disks) and tried to configure VSAN. And it was not smooth at all.

First, I've noticed I couldn't add flash devices as capacity tier. Disks were seen only when I manually marked them as HDD in vCenter Server. Even after doing this, the "NEXT" button was grayed out and couldn't complete the cluster creation.

Being a stubborn engineer, I did not read the manual, but I've dropped the UI and went all the way esxcli. Created the VSAN cluster in command line with no luck this time also. The cluster ended up in a split cluster scenario while UI showed confusing health messages.

Decided there is something wrong with my 6.7 U3 image (?!?) and Sunday coming to an end, I've temporarily postponed the project. Restarted it a few days later to find out that I was using vCenter Server 6.7 build 13639324 (U2a).So yeah, vCenter Server 6.7 U2a and VSAN 6.7 U3a doesn't sound like a good match. Immediately, I've updated vCenter Server to U3a and VSAN cluster creation took 5 minutes.

It wasn't a waste of time, as I've dusted off some powershell and I've learnt a valuable lesson: always check the version of your components. But it was definitely a PBKAC.

Create vCenter Server Roles Using PowerCLI - Applied to Veeam Backup & Replication

Security is important and having a minimal set of permissions is a requirement, not an option. Having this in mind (and being asked a few times by customers), I put together a short script that will create the vCenter Server roles required by Veeam Backup & Replication service account and Veeam ONE service account. The two accounts have different requirements, with Veeam ONE being the most restrictive as it needs mostly read only.

The script itself is pretty straight forward, the more time consuming is getting the privilege lists. So here you are:

• Veeam ONE role
• Veeam Backup & Replication role

And now for the actual scripting part:

$role = "Veeam Backup Server role"
$rolePrivilegesFile = "veeam_vc_privileges.txt"
$vCenterServer = "your-vcenter-server-FQDN"
Connect-VIServer -server $vCenterServer
$roleIds = @()
Get-Content $rolePrivilegesFile | Foreach-Object{
    $roleIds += $_
}
New-VIRole -name $role -Privilege (Get-VIPrivilege -Server $vCenterServer -id $roleIds) -Server $vCenterServer

The script will create a new vCenter Server role assigning it privileges from the file given as input.

If you ever require to get the privileges from vCenter Server then the next piece of code will help (thanks to VMware communities)

$role = "VBR Role"
Get-VIPrivilege -Role $role | Select @{N="Privilege Name";E={$_.Name}},@{N="Privilege ID";E={$_.ID}}

You will use the privilege ID format for creating the new role.

Veeam Replication - Automatically Fix Detected Invalid Snapshot Configuration

This one comes directly from the field. It is about  that moment when something goes wrong at a client's site and you need to fix. In this case, replicas went wrong. The link between the two sites not being reliable at all and some other networking issues at the DR site (yes, it's always network fault) determined replica VMs to end up in a corrupted state. If you want to keep the replica VMs that are already in the DR site then the fix is pretty simple: remove snapshots and remap vmdk's still pointing to delta files. Manually doing this is cumbersome, especially since some VMs can have multiple disks and when we are talking about tens or hundreds of  VMs. Luckily, some things can be scripted.

We will present some of the modules in the script, since the whole script is published here on GitHub. For ease of readability we will remove some of the try-catch blocks and logging messages from the original script and comment here only the logical part.

Script is given as input parameters vCenter Server where VM replicas are, backup server hostname, replica job status, replica job fail message and replica suffix. The replica suffix is important since it uses it to find VMs

$vbrServer = "vbr1"
$vcServer = "vc1"
$status = "Failed"
$reason = "Detected an invalid snapshot configuration."
$replicaSuffix = "_replica"

The script needs to be executed from a place where both vCenter Server and Veeam backup server are reachable and where PowerCLI module is imported as well as Veeam PowerShell snapin.

Add-PSSnapIn -Name VeeamPSSnapin
Connect-VBRServer -Server $vbrServer
Connect-VIServer -Server $vcServer

Next, the script will get all VMs for which the replication job has failed ($status) with the given reason ($status). For this, we use Get-VBRJob cmdlet, FindLastSession() and GetTaskSessions() methods. Once the VM in a replica job matches the chosen criteria, it i added to an array ($vmList)

$vmList = @()
# get failed replica VM names 
$jobs = Get-VBRJob  | Where {$_.JobType -eq "Replica"}
foreach($job in $jobs)
{
 $session = $job.FindLastSession()
 if(!$session){continue;}
 $tasks = $session.GetTaskSessions() 
 $tasks | foreach { 
        if (($_.Status -eq $status) -and ($_.Info.Reason -match $reason)) {
            $vmList += $_.Name
            }
        }
}

Once we have the list of VMs who's replica failed, it's time to get dirty. The VM name we are looking for is made of the original VM name (from the array) and the replica suffix.

$replicaName = $_ + $replicaSuffix

First, delete the snapshots. Fot this we use PowerCLI cmdlets: Get-VM, Get-Snapshots, Remove-Snapshot.

$replica = Get-VM -Name $replicaName -ea Stop
$replica | Get-Snapshot -ea Stop | Sort-Object -Property Created | Select -First 1 | Remove-Snapshot -RemoveChildren -Confirm:$false -ea Stop

Next, remap the disks if they are still pointing to the delta file. In order to do that, we get all the disks for the replica VM (Get-HardDisk) and we check if the disk name of the replica VM contains the specific delta file characters ("-0000"). This is how we determine if it's a delta disk or a normal disk. Delta disk name is parsed to generate the source disk name (removing the delta characters from vmdk name). Once this is done, it's just a matter of reattaching the source disk to the VM (Remove-HardDisk, New-HardDisk)

# get disks for a replica VM
$disk = $replica |  Get-HardDisk -ea Stop
# process each disk 
$disk | foreach {
    $diskPath = $_.Filename
    # check if disk is delta file
    if ($diskPath -Match "-0000") {
        $diskPath = $_.Filename
        # for each delta file parse the original vmdk name
        $sourceDisk = $diskPath.substring(0,$diskPath.length-12) + ".vmdk"
        # get the datastore where the delta is 
        $datastore = Get-Datastore -Id $_.ExtensionData.Backing.Datastore
        # check the original vmdk still exists on that datastore
        if (Get-HardDisk -Datastore $datastore.Name -DatastorePath $sourceDisk) {
            # remove delta
            Remove-HardDisk -HardDisk $_ -Confirm:$false -ea stop
            # attach original disk
            $newDisk = New-HardDisk -VM $replica -DiskPath $sourceDisk -ea Stop
        } else {
            Write-Host "WARN Could not find $($sourceDisk) on $($datastore.Name) "
            Add-Content -Path  $logFile -Value "WARN: Could not find $($sourceDisk) on $($datastore.Name) "
        }
    }
}

Last thing to do is to consolidate disks. We run Get-VM and check if consolidation is needed. If it's needed, we just run ConsolidateVMDisks_Task() method.

$vm = Get-VM -Name $replicaName -ea Stop
if ($vm.Extensiondata.Runtime.ConsolidationNeeded) {
    $vm.ExtensionData.ConsolidateVMDisks_Task()
}

Now the replica VMs are re-usable. There is some manual job to be done, though. You need to map the replica VM in the replication job and run the job.

vCenter Server Restore with Veeam Backup & Replication

Recently I went through the process of testing vCenter Server appliance restore in the most unfortunate case when the actual vCenter Server was not there. Since the tests were being done for a prod appliance, it was decided to restore it without connectivity to the network. Let's see how this went on.

Test scenario

• distributed switches only
• VCSA
• Simple restore test: put VCSA back in production using a standalone host connected to VBR

Since vCenter is "gone", first thing to do is to directly attach a standalone ESXi host to VBR. The host will be used for restores (this is a good argument for network team's "why do you need connectivity to ESXi, you have vCenter Server"). The process is simple, open VBR console go to Backup Infrastructure and add ESXi host. 

You will need to type in the hostname or IP and root account. Since vCenter Server was not actually gone, we had to use the IP  instead of FQDN as it was seen through the vCenter Server connection with the FQDN. 

Next, start the an entire VM restore

During the restore wizard, select the point in time (by default last one), then select Restore to a different location or with different settings:

Make sure to select the standalone host:

Leave default Resource Pool and datastores, but check the selected datastore has sufficient space. Leave the default folder, however if you still have the source VM change the restored VM's name:

Select the network to connect to. Actually disconnect the network of the restored VM. That was the scenario, right? Since the purpose of this article is not to make you go through the same experience we had, let's not disconnect it. And you will see why immediately:

Keep defaults for the next screens and start the restore (without automatically powering on the VM after restore). 

A few minutes later the VM is restored and connected to the distributed port group. 

We started by testing a disconnected restored VM, but during the article we didn't disconnect it. And here is why: when we initially disconnected the network of the restored VM, we got an error right after the VM was registered with the host and the restore failed. 

Same error was received trying to connect to a distributed portgroup configured with ephemeral binding. The logs show the restore process actually tries to modify network configuration of an existing VM and that makes it fail when VBR is connected directly to the console.When the portgroup is not changed for the restored VM, then the restore process skips updating network configuration. Of course, updating works with standard switch port group. 

In short, the following restore scenarios will work when restoring directly through a standalone host:

• restore VCSA to the same distributed port group to which the source VM is connected
• restore VCSA to a standard portgroup

Role Based Access for VMware in Veeam Backup & Replication 9.5 Update 4 - Advanced Backup Features and Recovery

In previous post we looked at how to configure the integration between Veeam and vCenter Server and how to create a simple backup job in self service portal.

In this post we'll go further and look at advanced configuration options available for backup jobs as well at recovery features.

1. VM restore
The user restore the VMs he is in charge of . Login to self service portal (https://em-address:9443/backup) and go to VMs tab. Select the VM you want to restore and press Restore. Two choices are given - overwrite and keep original VM.

Once the restore point has been selected and whether to power on or not the VM, the wizard starts the restore process to production environment. Choosing "keep" will result in a new VM being created with the name: originalVMname_restoredDateTime

In this case pay attention not to power it on after restore as it is connected to the same network with the original VM. If you choose "overwrite", powering on maybe a good idea. 

2. Guest file system indexing and file restore

When enabled,  VBR will create an index of files and folders on the VM fuest OS during backup. Guest file indexing allows you to search for VM guest OS files inside VM backups and perform 1-click restore in the self service portal.Initially the indexes are created on VBR server, but later on moved to Enterprise Manager server.

Let's take our job created initially and add guest indexing to it. Logon the self service portal, go to jobsa and edit the job and on guest processing tab tick the boc "Enable guest file system indexing".

Add user credentials and if different credentials are needed for VMs in the job press customize credentials. Lastly, define what to index. By default everything is indexed except system folders (Program files, Temp, Windows). You can select to index only specific folders or to index all folders.

Save the configuration and run the backup job. Once the job runs successfully, in self service portal  got to Files tab and type in the name of the file you want to find. In my case, the indexed VM is running IIS  so I will search for iisstart.htm file.

After finding the file we can now restore the file directly to production VM overwriting the originla file or keeping it. The file can be also downloaded to the local computer in an archive having the name FLR_DATE_HOUR

If we have more files to restore, we can add the file to a Restore list and execute the restore on multiple files at a time:

File level restore is possible without indexing, but it needs to mount the backup and let you manually search the file in the file system.

3. Application item restore

If using Veeam explores allows for a very good granularity and flexibility regarding supported applications, the self service portal is restricted to MSSQL and Oracle applications. On the Items of the self service portal, select the database, the location where to restore (original or alternate) and the point in time.

We'll restore the DB on the same server, but with a different name. For this reason choose restore to another location and press Restore. Specify the server name and the credentials for connecting to admin share and MSSQL DB

Specify the MDF and log file names - change them is the same location with original DB is being used:

Press Finish and watch the progress in the status bar:

When it is finished, you can connect to MSSQL server and check the DB has been created:

The new integration with vCenter Server offers a powerful tool for job management delegation to the users others than backup administrators. 

Role Based Access for VMware in Veeam Backup & Replication 9.5 Update 4

One of the cool features that Veeam Backup & Replication 9.5 Update 4 comes with is integration with vCenter Server role based access. What does it mean? It allows to delegate permissions to users and groups in Veeam Enterprise Manager based on their permissions in vCenter Server.

A user or group of users is now able to monitor and control the backup and restore of their own VMs in a vSphere environment based on a predefined policy. A policy can be defined through vSphere tags, a role in vCenter or as granular as a single permission in vCenter Server. Delegation is done through the self service portal in  Enterprise Manager.

Cool thing is the integration is actually extending vCenter Server access control by adding vSphere tags as control mechanisms. For example DBA's want to do their backup and restore, then just assign a tag to the VMs and create the policy in Enterprise Manager. It's that simple.

Since my environment uses tags, we will test the following scenario: all developers will have access to development VMs which are tagged with "SLA3" vSphere tags.

First, make sure tags exist in vCenter Server and are assigned to the VMs in scope.

Next, you need to install and configure VBR and Enterprise Manager (EM). This is not in the scope of the current article.

Once EM is installed and configured  login to the EM portal (https://em-address:9443/) and go to Configuration. Check VBR and vCenter Server are available and reachable.

On Self Service tab you will see the default configuration for all Domain Users group (my lab is AD integrated).

For the test we will create a new configuration.

Let's look at the Delegation Mode - what mechanism we use to define access:

By default, it uses VM privilege and the selected privilege is VirtualMachine.Interact.Backup, but you can choose to use any privilege available in vCenter Server. Once you need more flexibility, you can define roles in vCenter Server and use the delegation based on those roles (a set of privileges applied to an object). Finally, you can use vSphere tags and allow access based on the tags. Once the preferred method of delegation is chosen, it will apply to all self service configurations. So be careful when changing the method. 

Now open, the default self service configuration and let's take a look at it. We see it assigns a repository and a quota on that repository. The quota can be global (for all users in group) or individual (per each user). 

You also define the default settings for advanced job settings such as compression , deduplication, scheduling of full backups and so on. The settings can be copied from Veeam default or from an existing job. 

There are 4 job scheduling options available from allowing full access to the scheduler to no access to the scheduler. We will use the default one (full access to job scheduling). Choose wisely what you want your users to do.

vSphere tags drop down list appeared because I chose the Delegation Method as vSphere tags, but it's left empty. 

Let's create a new self service configuration for developers group. Press Add and then:

1. select the Type - user or group and search it in AD

2. select the repository and define the quota

3. select job scheduling options

4. select the vSphere tag

5. configure the advanced settings

Press Save and open the self service portal (https://em-address:9443/backup/). Login with one of the users from the group (Developers in my case). Since user is member of 2 configurations, select which configuration to logon:

Once logged in, the portal displays 5 tabs: Dashboards, Jobs, VMs, Files and Items

Go to Jobs and create a new backup job. A process similar to VBR console job creation will start. First give the job a name, description and decide how many restore points to keep:

Then add the VMs you want to backup. Only VMs with SLA3 tag will be displayed.

If required, enable application aware processing and add credentials for guest processing:

Schedule the job (in this configuration you are allowed):

Lastly, enable notifications (if you want to be alerted about the status of the job)

Now that the job has been created, you can run it:

Meanwhile, we can take a look at how things look in VBR console

You'll notice the running job named with the following convention: Domain\GroupName_Domain\UserName_JobName. So, all is good and running smoothly.

Back in the self service portal, pressing on the job statistics we can see what happened:

There you go, just had a user define a backup job and backup its own VM using a simple vSphere tag and no other settings at either vCenter Server or VBR level. Next time we'll take a look at restore options.

PowerCLI - Get the sizing of virtual machines

Any backup project needs an answer to at least the following questions: what is the total number of VMs, what is total backup size, what is the number of virtual disks per VM. It doesn't mean that it is that easy, since there are more questions that need answers, but these three are the base. Sometimes there is a fast answer to them, but there are situations when the information is unknown, or more difficult to find. To make life easier, I've put together a small script that is based on PowerCLI commandlets Get-VM and Get-HardDisk.

The script runs against vCenter Server, retrieves each VM, parses the data and sends the needed information to a CSV file. The file has the following structure: VM name, number of disks attached to the VM, used space without swap file (in GB), total used space (in GB), provisioned space (in GB).

A short explanation on the 3 different values for space:

• provisioned space is the space requested by the VM on the datastore. For thick provisioned VMs the allocated (used) space is equal to requested space. Thin provisioned ones do not receive the space unless they consume it. Hence the difference between provisioned space and used space (allocated) columns
• disk space reported in used column contains other files than the VM disks (vmdk). One of this files is the VM swap file, which is equal to the difference between the VM memory and the amount of VM reserved memory. This means that if there are no reservations on the VM memory, each VM will have the swap space equal to the size of memory. This file is not part of the backup and can be excluded. If we do a math exercise - 150 VMs with 8 GB each means more than 1 TB of data which we can ignore from calculations
• finally, looking at vm10 in the image above, we see the 2 columns (used and used without swap) being equal. Since vm10 is powered on, this means it has a memory reservation equal to the whole memory. For powered off VMs, the 2 columns will always be equal since swap file is created only for running VMs.

Let's take a look at the code. First we define a function that takes as input a VM object and returns a custom made power shell object with the required properties of the VM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

function GetVMData($v) { $vmResMemory = [math]::Round($v.ExtensionData.ResourceConfig.MemoryAllocation.Reservation/1024,2) $vmMem = [math]::Round($v.MemoryMB/1024,2) $vmUsedSpace = [math]::Round($v.UsedSpaceGB,2) if ($v.PowerState -match "PoweredOn") { $vmUsedSpaceNoSwap = $vmUsedSpace - $vmMem + $vmResMemory # removing swap space from calculations } else { $vmUsedSpaceNoSwap = $vmUsedSpace } $vmProvSpace = [math]::Round($v.ProvisionedSpaceGB,2) # swap space included $vmName = $v.Name $vmNoDisks = ($v | Get-HardDisk).count $hash = New-Object PSObject -property @{Vm=$v.Name;NoDisks=$vmNoDisks;UsedSpaceNoSwap=$vmUsedSpaceNoSwap;UsedSpace=$vmUsedSpace;ProvSpace=$vmProvSpace} return $hash }

We look at several parameters of the VM: memory reservation, allocated memory, used space, number of disks. We take the parameters and create a new PowerShell object that the function returns as a result.

The main body of the script is pretty simple. First, we define the format of the output CSV file and the we take every VM from vCenter Server (you need to be connected to vCenter Server before running this script) and process it:

1 2 3 4 5 6 7 8 9

$vmData = @('"Name","NoDisks","UsedSpaceGB(noSwap)","UsedSpaceGB","ProvisionedSpaceGB"') $csvFile = ($MyInvocation.MyCommand.Path | Split-Path -Parent)+"\vmData.csv" foreach ($v in get-vm) { $hash = GetVMData -v $v $item = $hash.Vm + "," + $hash.NoDisks + "," + $hash.UsedSpaceNoSwap + "," + $hash.UsedSpace + "," + $hash.ProvSpace $vmData += $item } $vmData | foreach { Add-Content -Path $csvFile -Value $_ }

If you want to look only at VMs that are powered on, then you need to add an IF clause that checks the VM power state in the FOR loop before processing each VM:

1 2 3

if ($v.PowerState -match "PoweredOn") { # process $v }

Not it's time to look at the output CSV file and start sizing for that backup solution.

NSX integration with vRealize Automation 7.4 - part 2

In part 1 of this post we presented the configuration at vRA level. In this post we'll see how to create a service in the Service Catalog for  programmatic NSX consumption.

First let's remember the main concepts of vRealize Automation service catalog:

• catalog items are published in the service catalog for user consumption e.g Linux VM or 3 tier web app

• catalog items can be grouped under different services: QA, Test&Dev, Web Apps, Linux Servers

• a user is allowed to request a service item based on his entitlements; Entitlements define who has access to catalog items and what actions he can do 

For start, we'll create a service called Linux VMs and a new Entitlement called Allow Linux VMs. We'll entitle all users of the business group to the Linux VMs service. Using services in the entitlement instead of individual items we make sure that every new item mapped to this service will be automatically accessible to the users.

Administration > Catalog Management > Services

Administration > Catalog Management > Entitlements

Next we'll create a blueprint that deploys vSphere VMs. There are several ways to provision vSphere VMs, we will use linked clones because they are very fast and use deltas to keep the changes (which is good in labs). To use linked clones we need to create a golden image: a VM configured to the desired state.

First create the VM: deploy it from an existing template or create it from scratch. VM hostname and networking details will be configured at deployment during guest OS customization. For this to work we need VMware tools installed in the VM and a customization specification created in vCenter Server. 

No other special configuration is needed for the VM.

Optional step (vRA agent installation): if you don't plan to run scripts inside the guest OS of the vRA manged VM, you can skip this step. The installation should be pretty easy since VMware already provides a script that can handle it. Go to vRA appliance URL and download the script on your Linux VM:

 wget  https://vra_app_fqdn/software/download/prepare_vra_template_linux.tar.gz --no-check-certificate

Then extract the script from the archive and run it:

tar -xvzf prepare_vra_template_linux.tar.gz

cd prepare_vra_template_linux

./prepare_vra_template.sh  

Choose default agent type (vSphere), add the address of vRealize Appliance, manager service, accept the key fingerprints for the certificates, set the download timeout, and install JRE (if not already in the VM)

Now we have a VM with all the utils inside (VMware tools and optionally vRA agent) and we create the snapshot that will be the base for linked clones.

At this point we login to vRA portal and we start working on our service creation. Go to Design > Blueprints. Start creating a New blueprint. Type the name of the blueprint, assign an unique ID or leave the automatically generated one, limit the number of deployments per request (if you want). Add lease days to control the spawn of deployed VMs (especially for temporary environments) and add a period of time you want the item to be archived before deleting (when lease expires).

Since this is for demo, I've added a default lease of 1 day and no archival (automatic deletion after lease expires). On NSX Settings tab, choose the NSX transport zone and if you want to isolate the VMs deployed from this blueprint (allow only internal traffic between VMs).

Pressing OK button will take you to the canvas. From Machine Types category, drag and drop vSphere (vCenter) Machine

From Network&Security category, drag and drop On-Demand Routed Network

Select the vSphere__vCenter__Machine_1 components form the canvas and fill in the configuration details. Add the number of instances that can be deployed in a request 

Add build information: how the VM will be created (linked clone), where to clone it from, what customization specification to use:

Type the VM consumed resources: number of CPUs, memory, storage. Take care when configuring these values: if you allow 10 instances in a deployment, each instance with a maximum of 8 vCPU and 32 GB of RAM you may end up with a deployment using 80 vCPU and 320 GB of RAM. This is a good moment when approval workflows come into place.

Finally we need to connect the VM to the network. But first we'll configure the network component. On canvas select the On-Demand_Routed_Network_1 components and choose the parent network profile (profile that has been created in part 1) 

Go back to vSphere component, go to Network tab and click New. From drop down box select the network name

Lastly, add a custom property for the VM to define the operating system that is being used

At this moment we've configured how to create the VM, how to create the network and linked the VM to the network. Press Finish and then Publish the blueprint:

Once the blueprint has been published, it will appear under Administration > Catalog Management > Catalog Items. Select the new catalog item, press Configure and map it to the service created earlier at the beginning of the post. 

The service will appear in the Catalog tab and you can press Request to deploy a new instance of it. To see what is happening, go to Requests tab, select the request, press View Details and when the request details open press Execution Information

Here you will see that the vxlan has been created on demand and DLR reconfigured. Also the VM has been created and mapped to the new vxlan. The process can also be monitored in vCenter Server

After the provisioning finished successfully, the components will be displayed in Items tab from where they can be managed using day 2 operations.