vCenter Server 8 Upgrade - Unknown Host Error

 I've recently upgraded my vCenter Server 7.0.3 to 8.0 and during the process I've encountered the following error: Error in method invocation [Errno 1] Unknown host

The error is related to using IP address instead of FQDN as shown in the image below . It will appear in stage 2, after VCSA VM is deployed in the environment. 

To avoid it, just use FQDN everywhere. This error was first mentioned on upgrades from 6.7 to 7.0, but somehow in my lab got ported to upgrade from 7.0 to 8.0.  

vSphere 7 Local Disk Fresh Install and VMFS-L

I just got my new Intel NUC and it was time to install it. So I popped the vSphere 7 USB stick and started installing. Ten minutes later I was looking at the freshly installed system and noticed that the hard drive was much smaller than expected - 337 GB from a 500 GB raw drive. The vSphere 6.7 NUC with the same drive had a capacity of 458 GB. So what happened to 120 GB of space?

Looking at the partition layout the new 120GB VMFSL caught my attention.

Because I am an engineer and I read the manual after the fact, I started reading vSphere 7 storage requirements in the official documentation. The VMFS-L is used as ESX-OSData partition instead of the scratch partition. It stores logs, coredumps and configuration. However I cannot loose 120 GB on each of my NUCs and I am already running vSphere 7 from a USB stick . So two questions came up:

1.  How to recover some of the 120GB from VMFS-L

2. How to install vSphere 7

1.  How to recover some of the 120GB from VMFS-L

I used the install USB stick to install vSphere 7 on it. However this didn't work since vSphere 7 would find the big VMFS-L partition and use it. Which made removing it not possible.

I also turned off scratch 

Then I actually booted up a vSphere 6.7 USB installation. Now I could access the disk and remove VMFS-L partition. 

2. How to install vSphere 7

Since I already had vSphere 7 running off a 16 GB USB stick, I figured out that it should be able to run with less capacity (again, it is my home lab, I wouldn't to all these tricks in production). Hence I installed vSphere 6.7 on the local disk and that got me to the following layout:

Then I added the host to vCSA 7.0 and upgraded to vSphere 7 using Lifecycle Manager which got me to a better looking final partition layout. The upgrade uses the existing core dump, locker, and scratch partitions to create the ESX-OSData volume. 

Seems that it's better to read first even if you are playing in your lab. In my defense, I had installed vSphere 7 before but it was a nested ESXi and they had a small dedicated boot drive.

For more details on how to change scratch partitions you can also look at the following KB 

Upgrading vCSA 6.7 to vCSA 7.0

First thing, first: backup vCSA 6.7. Use a backup solution or do a vCenter backup from VAMI. It's never bad to have one, even though you will see that it can be skipped.

Next, mount the ISO and start the installer UI. The UI presents the already known options: install, upgrade, migrate, restore. My target is to upgrade existing lab environment so, I am choosing upgrade. Before going further into the post, I want to clarify the upgrade process. It is not an in place upgrade, it is actually a migration and you will end up with a new VM running vCSA 7.0 alongside the old vCSA 6.7.

I don't intend to describe the step by step upgrade since there are already a few good blog posts and the process itself is pretty straight forward. I will however highlight some things I came across during the upgrade.

• you will end up with 2 VMs - a powered off old vCSA 6.7 and a powered on vCSA 7.0
• vCSA 7.0 will preserve in the end FQDN and IP of vCSA 6.7
• a temporary IP address is required for vCSA 7.0 to be used during the data migration from vCSA 6.7 (that moment in time when both VMs are up and running)
• it's a 2 stage process: first a new vCSA VM is deployed, then the data is migrated
• migration offers possibility to chose how much data you actually want transferred (the more you choose, the longer it takes)
• configuration and inventory 
• configuration, inventory, tasks and events 
• configuration, inventory, tasks, events and performance metrics

• the whole process took almost 2 hours (my lab), expect bigger times for real environments

During the upgrade itself I got a couple of warning at pre-check and a couple of notifications at the end for the rest being smooth and uneventful. The warning was about not having DRS enabled on my cluster, which was fine because I had a single node cluster where vCSA was running:

The notifications is about  TLS 1.0 and 1.1 being disabled and Auto-Deploy needing an update:

I did try to upgrade using CLI installer, however there are some issues with the upgrade templates and its schema in the GA version (15843807) and it kept on failing during JSON template precheck. I will come back to this topic once I figure it out.

vCenter Server Appliance 7.0 Command Line Installer

One of my favorite features in vSphere is command line install of vCenter Server. It first appeared with vCenter Server 6.0. It is based on a JSON file input and can be use to do a fresh install of vCSA 7.0, upgrade an existing vCSA 6.5 or 6.7 installation to 7.0 or migrate a Windows vCenter Server 6.5 or 6.7 to vCSA 7.0.

The installer can be run from Windows, Linux or Mac. To access it, you need the vCSA iso file and locate folder vcsa-cli-installer\win32 (Windows users). JSON templates are found in templates folder. You need to modify the JSON template that fits your use case. I will do a fresh install of vCSA 7.0 in my lab so I will be using the template embedded_vCSA_on_VC.json which deploys the new vCSA inside an existing vCenter Server. The template is commented very well, however I will post here an example of what a simple configuration looks like. Please be aware that this is just a snippet of the actual template and some parts have been left out for ease of reading.

    "new_vcsa": {
        "vc": {
            "hostname": "vcsa67.mylab.com",
            "username": "administrator@mylab.com",
            "password": "",
            "deployment_network": "VM Network",
            "datacenter": [
                "VDC-1"
            ],
            "datastore": "DATASTORE-1",
            "target": [
                "CLUSTER-1"
            ]
        },
        "appliance": {
            "thin_disk_mode": true,
            "deployment_option": "small",
            "name": "vcsa70"
        },
        "network": {
            "ip_family": "ipv4",
            "mode": "static",
            "system_name": "vcsa70.mylab.com",
            "ip": "192.168.100.1",
            "prefix": "24",
            "gateway": "192.168.100.254",
            "dns_servers": [
                "192.168.1.10"
            ]
        },
        "os": {
            "password": "",
            "ntp_servers": "0.ro.pool.ntp.org",
            "ssh_enable": true
        },
        "sso": {
            "password": "",
            "domain_name": "vsphere.local"
        }
    }

As you can see, once you create the template it can reused a lot of times. What for you may ask and one answer is nested labs. If you are unsure what size the vCSA should be, the installer will tell you:
.\vcsa-deploy.exe --supported-deployment-sizes

The installer takes different parameters besides the JSON file:
.\vcsa-deploy.exe install --accept-eula [--verify-template-only|--precheck-only][file_path_to_json] 

If you want to automatically accept SSL certificate thumbprint, you can add --no-ssl-certificate-verification parameter.

As seen above, the installer comes with 2 options that enable you to check that everything is fine before actually starting the install:

• verify-template-only - will run a JSON file verification to validate the structure and input parameters (e.g. password strength, IP address, netmask). The final check result is displayed along with the path to the log file. The log file contains all required details. For example if you typed an IP address that does not exist, the following message is displayed in log file:

2020-03-27 19:44:06,232 - vCSACliInstallLogger - ERROR - The value '192.268.100.1' of the key 'ip' in section 'new_vcsa', subsection 'network' is invalid. Correct the value and rerun the script.

• precheck-only - will do a dry run of the installer. This time it will connect to vCenter server and check that the environment values are actually correct: for example that you don't have another VM with the same name, vCenter objects are correct (datacenter, datastore, cluster or host). It also does a ping test to validate the IP/FQDN entered for the new vCSA are available.

================ [FAILED] Task: PrecheckTask: Running prechecks. execution

Error message: ApplianceName: A virtual machine with the name 'vcsa70' already

exists on the target ESXi host or cluster. Choose a different name for the

vCenter Server Appliance (case-insensitive).

Of course, you don't have to run both checks or even any check if you are confident enough. For me, precheck-only helped since I didn't understand how to fill in the JSON file from the first time (I will blame it on a barrier language). One very important aspect of installing is to have DNS records setup and working. If you don't, even if the prechecks and the actual install will work, first boot of vCSA will most likely fail.

Having all setup up and checked, you  just run the install command and that's it. I like the CLI installer because it is simple, powerful and repeatable. No more filling in fields in a GUI and waiting for the lines on the screen.

Distributed vRealize Automation 7.x Orchestrated Shutdown, Snapshot and Startup using PowerCLI

I will take a look at performing scheduled operations on vRealize Automation 7.6 (although  the article can apply to other versions). In a distributed architecture, vRA 7.6 can become a pretty big beast. Based on the requirements and the actual vSphere implementation (virtual datacenters and vCenter Servers), such a deployment can easily grow to 12-16 VMs. Scheduling operations that require restart of the environment requires careful preparation because of the dependencies between different components such as vRA server, IaaS components, MSSQL database. One of the most common and repetitive tasks is Windows patches requiring regular IaaS components reboots. But there are other activities that need to shutdown the whole environment and take a cold snapshot, for example a hot fix.

VMware documentation defines the proper way of restarting components in a vRA distribute environment. What I've done is to actually take those steps and put them in a PowerCLI script making the procedure reusable and predictable. A particular case is to detect if a VM is a placeholder VM (being a VM replica). Before going to the script itself, let's look at the whole workflow.

The first part is just a sequential shutdown and wait until the VMs poweroff to go to the next step. Then a cold snapshot is taken for all VMs. Lastly, VMs are powered on in an orchestrated sequence and wait times are implemented to allow for the services to come back up.

Getting to code part - first we define the list of VRA components, in this case proxies, DEM-workers, DEM-orchestrators, IaaS web, IaaS Managers and vRA Appliances.

# vCenter Servers
$vCSNames = ("vcssrv1", "vcssrv2", "vcssrv3","vcssrv4")

# vRA Components
$workers = @("vradem1", "vradem2","vraprx1","vraprx2", "vraprx3", "vraprx4", "vraprx5", "vraprx6")
$managerPrimary = @("vramgr1")
$managerSecondary = @("vramgr2")
$webPrimary = @("vraweb1")
$webSecondary = @("vraweb2")
$vraPrimary = @("vraapp1")
$vraSecondary = @("vraapp2")

# Snapshots
$snapName = "vra upgrade"
$snapDescription = "before 7.6 upgrade"

# Log file
$log = "coldSnapshotVra.log"

Next we define the 3 functions for shutdown, snapshot and start the VMs. Since in our environment we use SRM, I had to check for placeholder VMs when powering off and snapshotting the VMs. We'll take them one by one. First. shutdown VMs and wait for the VM to stop:

function shutdownVMandWait($vms,$log) {
    foreach ($vmName in $vms) {
        try {
            $vm = Get-VM -Name $vmName -ErrorAction Stop
            foreach ($o in $vm) {
                if($o.ExtensionData.Summary.Config.ManagedBy.Type -eq "placeholderVm") {
                    Write-Host "VM: '$($vmName)' is placeholderVm. Skipping."
                } else {
                    if (($o.PowerState) -eq "PoweredOn") {
                        $v = Shutdown-VMGuest -VM $o -Confirm:$false
                        Write-Host "Shutdown VM: '$($v.VM)' was issued"
                        Add-Content -Path $log -Value "$($v)"
                    } else {
                        Write-Host "VM '$($vmName)' is not powered on!"
                    }
                }   
            }
        } catch {
            Write-Host "VM '$($vmName)' not found!"
        }
    }
    foreach ($vmName in $vms) {
        try {
            $vm = Get-VM -Name $vmName -ErrorAction Stop
            while($vm.PowerState -eq 'PoweredOn') { 
                sleep 5
    Write-Host "VM '$($vmName)' is still on..."
                $vm = Get-VM -Name $vmName
            }
            Write-Host "VM '$($vmName)' is off!"
        } catch {
            Write-Host "VM '$($vmName)' not found!"
        }
    }
}

Next, take snapshots of the VMs

function snapshotVM($vms,$snapName,$snapDescription,$log) {
    foreach ($vmName in $vms) {
        try {
            $vm = Get-VM -Name $vmName -ErrorAction Stop
        } catch {
            Write-Host "VM '$($vmName)' not found!"
            Add-Content -Path $log -Value "VM '$($vmName)' not found!"
    
        }
        try {
            foreach ($o in $vm) {
                if($o.ExtensionData.Summary.Config.ManagedBy.Type -eq "placeholderVm") {
                    Write-Host "VM: '$($vmName)' is placeholderVm. Skipping."
                    Add-Content -Path $log -Value "VM: '$($vmName)' is placeholderVm. Skipping."
                } else {
                    New-Snapshot -VM $o -Name $snapName -Description $snapDescription -ErrorAction Stop
                }   
            }
        } catch {
            Write-Host "Could not snapshot '$($vmName)' !"
            Add-Content -Path $log -Value "Could not snapshot '$($vmName)' !"
    
        }
    }
}

And finally, power on the VMs:

function startupVM($vms,$log) {
    foreach ($vmName in $vms) {
        try {
            $vm = Get-VM -Name $vmName -ErrorAction Stop
            foreach ($o in $vm) {
                if($o.ExtensionData.Summary.Config.ManagedBy.Type -eq "placeholderVm") {
                    Write-Host "VM: '$($vmName)' is placeholderVm. Skipping."
                } else {
                    if (($o.PowerState) -eq "PoweredOff") {
                        Start-VM -VM $o -Confirm:$false -RunAsync
                    } else {
                        Write-Host "VM '$($vmName)' is not powered off!"
                    }
                }   
            }
        } catch {
            Write-Host "VM '$($vmName)' not found!"
        }
    } 
}

Last part of the script is the putting all the logic together. Connect to vCenter Server, orderly shutdown VMs, take the cold snapshots and bringing back the whole environment.

# MAIN
# Connect vCenter Server
$creds = Get-Credential
try {
    Connect-VIServer $vCSNames -Credential $creds
} catch {
    Write-Host $_.Exception.Message
}

# Stop VRA VMs
Write-Host "### Stopping DEM Workers an Proxies"
shutdownVMandWait -vms $workers -log $log
Write-Host "### Stopping Secondary Managers and Orchestrators"
shutdownVMandWait -vms $managerSecondary -log $log
Write-Host "### Stopping Primary Managers and Orchestrators"
shutdownVMandWait -vms $managerPrimary -log $log
Write-Host "### Stopping secondary Web"
shutdownVMandWait -vms $webSecondary -log $log
Write-Host "### Stopping primary Web"
shutdownVMandWait -vms $webPrimary -log $log
Write-Host "### Stopping secondary VRA"
shutdownVMandWait -vms $vraSecondary -log $log
Write-Host "### Stopping primary VRA"
shutdownVMandWait -vms $vraPrimary -log $log

# Snapshot VRA VMs
Write-Host "### Snapshotting DEM Workers an Proxies"
snapshotVM -vms $workers -snapName $snapName -snapDescription $snapDescription -log $log
Write-Host "### Snapshotting Secondary Managers and Orchestrators"
snapshotVM -vms $managerSecondary -snapName $snapName -snapDescription $snapDescription -log $log
Write-Host "### Snapshotting Primary Managers and Orchestrators"
snapshotVM -vms $managerPrimary -snapName $snapName -snapDescription $snapDescription -log $log
Write-Host "### Snapshotting secondary Web"
snapshotVM -vms $webSecondary -snapName $snapName -snapDescription $snapDescription -log $log
Write-Host "### Snapshotting primary Web"
snapshotVM -vms $webPrimary -snapName $snapName -snapDescription $snapDescription -log $log
Write-Host "### Snapshotting secondary VRA"
snapshotVM -vms $vraSecondary -snapName $snapName -snapDescription $snapDescription -log $log
Write-Host "### Snapshotting primary VRA"
snapshotVM -vms $vraPrimary -snapName $snapName -snapDescription $snapDescription -log $log

# Start VRA VMs
Write-Host "### Starting primary VRA"
startupVM -vms $vraPrimary -log $log
Write-Host  " Sleeping 5 minutes until Licensing service is registered"
Start-Sleep -s 300

Write-Host "### Starting secondary VRA"
startupVM -vms $vraSecondary -log $log
Write-Host  " Sleeping 15 minutes until ALL services are registered"
Start-Sleep -s 900

Write-Host "### Starting Web"
startupVM -vms $webPrimary -log $log
startupVM -vms $webSecondary -log $log
Write-Host  " Sleeping 5 minutes until services are up"
Start-Sleep -s 300

Write-Host "### Starting Primary manager"
startupVM -vms $managerPrimary -log $log
Write-Host  " Sleeping 3 minutes until manager is up"
Start-Sleep -s 180

Write-Host "### Starting Secondary manager"
startupVM -vms $managerSecondary -log $log
Write-Host  " Sleeping 3 minutes until manager is up"
Start-Sleep -s 180

Write-Host "### Starting DEM Workers an Proxies"
startupVM -vms $workers -log $log

Write-Host "### All components have been started"

# Disconnect vCenter 
Disconnect-VIServer * -Confirm:$false

You will notice that the orchestration logic is actually implemented here. This means you can easily add/remove/modify the VMs that the script targets. Let's say you only want to snapshot some proxies for which you don't need to bring everything down. Or you want to add external vRealize Orchestrators appliances. All changes take place in the main part by simply commenting out some steps.

This script helped a lot the nightly operations we had to do across our whole environment and I hope it will do the same for you.

Migrate vRealize Orchestrator 7.3 Cluster with External Database to 7.6

In every environment's life comes a time when you need to upgrade. So it did happen for our vRO setup. The source environment is running vRO 7.3.1 in a 2-node cluster and with and external MSSQL database. The target environment is a 2-node cluster running vRO 7.6 and internal PostgreSQL (since no external is supported).

Because source vRO is connected to external MSSQL server, in place upgrade is out of the question. So a migration is actually the only way to do it. A constraint for the migration is not to change hostnames or IP addresses. Well, the requirement is actually to get everything back to 7.6 with a minimum of reconfiguration and here we talk about plugins, workflows, actions, configuration elements, certificates and all that jazz.

Having laid down the task ahead we started working on the plan. And what we came up with was pretty interesting. 

vRO 7.6 appliance offers a migration tool that is accessible through VAMI. The tool needs access to source vRO and database. Since we have a 2-node cluster, we'll take advantage of that and do a rolling migration. Source vRO 7.3 cluster is made of nodes: node1 and node2. Target vRO 7.6 cluster will be running the same vRO nodes.

So how does this rolling migration look like:

In initial state there is a vRO 7.3.1 cluster with external MSSQL database. We start by redirecting traffic in load balancer to node1 and disabling monitoring on the pool. Then we break the cluster by removing node2 and powering it off. Next we deploy a new node2 running 7.6 and using the same hostname and IP address. If you want to reuse the same VM name, you just need to rename the old node in vCenter Server.

This takes us to intermediary state where we have one node1 in 7.3.1 and node2 in 7.6 with and embedded PostgreSQL DB. Once node2 in 7.6 is up an running, we connect to VAMI and fire up the migration tool. Migration tool will transfer all data from node1 and the external DB to node2. After vRO is migrated to 7.6, we proceed to power off node1 running 7.3.1 and deploy node1 in 7.6. Don't forget to switch over traffic to node2 in load balancer.

Now we are in final state where both nodes are running 7.6. Once node1 is up, we add it to the cluster that node2 is already member. The cluster is back and this time running vRO 7.6 with embedded PostgreSQL database. Simple, right?

Well, there are a few of gothcas:
- when running the migration tool, a pre-check is run that can fail if the vRO DB contains duplicates; it can be fixed by either removing duplicates from DB or connecting with vRO client to source vRO and renaming them
- SSL certificates are not migrated - you need to connect to source vRO keystore and export the certificates, then re-add them in control center
- there is a cumulative update that needs to be applied to the freshly deployed vRO 7.6
- dynamic types plugin configuration is not migrated - there is an updated plugin that needs to be installed
- in some cases you may need to update the appliance hardware including file system

Lastly enable traffic in load balancers to both nodes and re-enable health monitors. Do that after importing SSL certificate.

Happy migrations!

Command line upgrade of ESXi 5.1 to 5.1 U1

The reason I`ve chosen this method is not having VUM installed and also wanting to remember a bit of esxcli. The process is pretty simple:

• download the zip bundle (update-from-esxi5.1-5.1_update01.zip) and put it on a datastore accessible by all hosts (I am using a NFS share)
• check to see if the host has the datastore mounted, if not - add the datastore

~ # esxcli storage nfs list

~ # esxcli storage nfs add -H 192.168.1.1 -s /mnt/vol1-nfs -v shared1-nfs-sata

• check the host version

~ # esxcli system version get
   Product: VMware ESXi
   Version: 5.1.0
   Build: Releasebuild-799733
   Update: 0

• check the update file and see if it requires the host in maintenance mode (in this case, it does not - however since the update will request a system reboot, I think it is better to place the host in maintenance mode and have it cleaned up of VMs)

~ # esxcli software sources vib get --depot=/vmfs/volumes/shared1-nfs-sata/update-from-esxi5.1-5.1_update01.zip
...
   Maintenance Mode Required: False

• install the update

~ # esxcli software vib update --depot=/vmfs/volumes/shared1-nfs-sata/update-from-esxi5.1-5.1_update01.zip

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true

• reboot the system (after all VMs have been migrated to other hosts) and check the new host version

~ # reboot
...
~ # esxcli system version get
   Product: VMware ESXi
   Version: 5.1.0
   Build: Releasebuild-1065491
   Update: 1