Thursday, May 13, 2021

vSphere with Tanzu - Enable Supervisor Cluster using PowerCLI

In previous post we looked at how to manually enable Supervisor cluster on a vSphere cluster. Now we'll reproduce the same steps from GUI in a small script using PowerCLI. 

PowerCLI 12.1.0 brought new cmdlets for VMware.VimAutomation.WorkloadManagement module and one of this is Enable-WMCluster. We will be using this cmdlet to enable Tanzu supervisor cluster. In the following example we'll be using NSX-T, but the cmdlet can be used with distributed switches. 

The following script is very simple .First we need to connect to vCenter Server and NSX manager

Connect-VIServer -Server
Connect-NsxtServer -Server

Next we define the variables (all variable that were in the UI wizard).

The cluster where we enable Tanzu, the content library and the storage policies:

$vsphereCluster = Get-Cluster "MYCLUSTER"
$contentLibrary = "Tanzu subscribed"
$ephemeralStoragePolicy = "Tanzu gold"
$imageStoragePolicy = "Tanzu silver"
$masterStoragePolicy = "Tanzu gold"

Management network info for Supervisor Cluster VMs

$mgmtNetwork = Get-VirtualNetwork "Mgmt-Network"
$mgmtNetworkMode = "StaticRange"
$mgtmNetworkStartIPAddress = ""
$mgtmNetworkRangeSize = "5"
$mgtmNetworkGateway = ""
$mgtmNetworkSubnet = ""
$distributedSwitch = Get-VDSwitch -Name "Distributed-Switch"

DNS and NTP servers

$masterDnsSearchDomain = "my.lab"
$masterDnsServer = ""
$masterNtpServer = ""
$workerDnsServer = ""

Tanzu details - size and external and internal IP subnets

$size = "Tiny" 
$egressCIDR = ""
$ingressCIDR = ""
$serviceCIDR = ""
$podCIDR = ""

One more parameter needs to be provided: Edge cluster ID. For this we use NSX-T manager connectivity and 

$edgeClusterSvc = Get-NsxtService -Name com.vmware.nsx.edge_clusters
$results = $edgeClusterSvc.list().results
$edgeClusterId = ($results | Where {$_.display_name -eq "tanzu-edge-cluster"}).id

Last thing is to put all the parameters together in the cmdlet and run it against the vSphere cluster object

$vsphereCluster | Enable-WMCluster `
-SizeHint $size `
-ManagementVirtualNetwork $mgmtNetwork `
-ManagementNetworkMode $mgmtNetworkMode `
-ManagementNetworkStartIPAddress $mgtmNetworkStartIPAddress `
-ManagementNetworkAddressRangeSize $mgtmNetworkRangeSize `
-ManagementNetworkGateway $mgtmNetworkGateway `
-ManagementNetworkSubnetMask $mgtmNetworkSubnet `
-MasterDnsServerIPAddress $masterDnsServer `
-MasterNtpServer $masterNtpServer `
-MasterDnsSearchDomain $masterDnsSearchDomain `
-DistributedSwitch $distributedSwitch `
-NsxEdgeClusterId $edgeClusterId `
-ExternalEgressCIDRs $egressCIDR `
-ExternalIngressCIDRs $ingressCIDR `
-ServiceCIDR $serviceCIDR `
-PodCIDRs $podCIDR `
-WorkerDnsServer $workerDnsServer `
-EphemeralStoragePolicy $ephemeralStoragePolicy `
-ImageStoragePolicy $imageStoragePolicy `
-MasterStoragePolicy $masterStoragePolicy `
-ContentLibrary $contentLibrary

And as simple as that, the cluster will be enabled (in a scripted and repeatable way). 

Tuesday, May 4, 2021

vSphere with Tanzu - Enable Supervisor Cluster

Before diving head first into how to enable supervisor cluster it's important to clarify a few aspects. There are several great posts (here and here) on how to deploy automatically Tanzu on vSphere. The reason I choose to present a step by step guide is because going through the manual steps helped me clarifying some aspects. I will not be covering the networking part. There are two ways of enabling Tanzu on vSphere - using NSX-T or using vSphere networking and a load balancer

The Supervisor Cluster is a cluster enabled for vSphere with Tanzu. There is a one to one mapping between the Supervisor Cluster and the vSphere cluster. It is important because there features that are defined at Supervisor Cluster level only and inherited at Namespace level. A vSphere Namespace represents a set of resources where vSphere Pods, Tanzu Kubernetes clusters and VMs can run. It is similar to a resource pool in the sense that it brings together the compute and storage resources that can be consumed. A Supervisor Cluster can have many Namespaces, however at the time of writing there is a limit of 500 namespaces per vCenter Server. Depending on how you map namespaces to internal organizational units this can also be important. The high level architecture and components of a supervisor cluster can seen here


  • Configure NSX-T. Tanzu workloads need a T0 router configured on a edge cluster. All other objects (T1's, LB's, segments) are configured automatically during pod deployment. Edge recommended size is large, but it works with medium for lab deployments. Also for lab only, the edge cluster can run with a single edge node. Deploying and configuring NSX-T is not in the scope of this article
  • vCenter Server level
    • vSphere cluster with DRS and HA enabled
    • content library for Tanzu Kubernetes cluster images subscribed to In case you don't have Internet connectivity from vCenter Server you will need to download them offline and upload to the library. Check if you can have access to Internet via  a proxy and you can add the proxy in vCenter Server VAMI interface (https://vcs_fqdn:5480) 
    • storage policies - for lab purpose one policy can be created and used for all types of storage. Go to Policies and Profiles and create a new VM Storage Profile - Enable host based rules and select Storage I/O Control

  • IP's - for ingress and egress traffic (routed), pod and service (internal traffic) 
  • latest version of  vCenter Server  - 7.0 U2a (required for some of the new functionalities - vm operator and namespace self service)
  • NTP working and configured for vCenter Server  and NSX-T manager (and the rest of components) 

Enabling the Supervisor Cluster is pretty straight forward - go to workload management, clusters and add cluster. The wizard will take you through the following steps. 

First select vCenter Server and the type of networking. If you don't have NSX-T configured, then you can use vSphere Distributed Switch but first a load balancer needs to be installed (HAproxy or AVI)

Then you select the vSphere cluster where to enable the Supervisor cluster. 

Choose the size of the control plane VMs - the smaller they are the smaller the Kubernetes environment.

Map storage policies to types of storage in the Supervisor cluster

Add management network details. It is important to clarify that the supervisor VMs have 2 NIC's - one connected to vSphere distributed portgroup that has access to vCenter Server and NSX-T manager and another one connected to Kubernetes service network. Please check the "View Network Topology" in the step to have a clear picture of the configuration of the Supervisor VM. Also supervisor VMs need a range of 5 IPs free that will be use - in my case I am selecting a range from the management network.  

Next add the network details for ingress and egress networks and also for internal cluster networks (service and pod). Ingress and egress networks are used to access services inside the Kubernetes cluster  via DNAT (ingress) and by internal services to access outside world via SNAT (egress). 

In case you use the same DNS server for management and service networks, the server must be reachable over both interfaces. Service network will use the IP of the egress network to reach DNS. 

Lastly, add the content library, review the configuration and give it a run. 

 Once the cluster is deployed successfully you will see it in the ready state:

You can now create namespaces and Kubernetes guest clusters. To access the cluster you will need to connect to https://cluster_ip and download kubectl vSphere plugin. 

Since we got through all the manual steps, we can look next at automating the configuration using PowerCLI in the next post.

Monday, April 19, 2021

vRealize Automation 8.4 Disk Management

vRealize Automation 8.4 brings some enhancements to storage management at cloud template level. Since this a topic that I am particularly interested in, I've decided to take a look at the topic. I've focused on two cases cases:

  • cloud template with predefined number of disks
  • cloud template with dynamic number of disks 

Cloud template with predefined number of disks

First I've created a template with 2 additional disks attached to it. Both disk are attached to SCSI controller 1 and their size is given as input. Both disk are thin provisioned. The template looks as following:

Let's see the code behind the template. There are 2 main sections:

  • inputs: where the input parameters are defined
  • resources: where template resources are defined. 
Inputs section contains parameters for VM image flavor (defaults to micro) and disk sizes (default to 5GB each)

Resources section has 3 resources - the VM (Cloud_Machine_1) and its 2 additional disks (Cloud_Volume_1 and Cloud_Volume_2). Each resource is defined by a type and properties. 

The disks are mapped to the VM resource using attachedDisks property. The input parameters can be seen under each resource, for example for disk capacity: ${input.flavor}, ${input.disk1Capacity} and  ${input.disk2Capacity}. Please note that in this case the SCSI controller and the unit number are given in the template. 

    titleApp Disk Capacity GB
    titleLog Disk Capacity GB
        - tag'vmw:az1'
        - source'${}'
        - source'${}'

Once the template is created, you can run a test to see if all constraints are met and if code will run as expected. This is a useful feature and it is similar to unit tests used in development processes. 

If tests are successful, you can deploy the template. After the resources are provisioned, you can select in the topology view any of the resources and check the details and the available day 2 actions in the right pane. 

For the disks we can find out the resource name, its capacity, its state (if it is attached or not), if it is encrypted and to what machine it is associated.

More details are displayed under custom properties such as the controller name, datastore on which the disk is placed and so on. A lot more details are displayed under custom properties such as the controller name, datastore on which the disk is placed and so on.

We can resize the disks and also remove the disks from the machine (delete). You can see below a resize action where the existing value is displayed and the new value is typed:

Cloud template with dynamic number of disks 

The first example uses a predefined number of disks in the template even though the disk size is given as an input parameter. Another use case is to let the consumer specify how many disks he needs attached to the VM (obviously with some limitations). 

In this case the code is looking a bit different. We define an array as the input for the disk sizes. The array is dynamic, but in our case limited to maximum 6 values (6 disks). This array is then used to define the Cloud.Volume resource. 

          titleSize (GB)
        - tag'vmw:az1'
      attachedDisks'${map_to_object(resource.disk[*].id, "source")}'

When requesting the deployment, an user can leave the default disk in the VM image or add up to 6 more disks

Details about the disks and controllers can be seen directly from vRA. In the example below all disks are placed on the same controller:


When adding same size disks an error is displayed about "data provided already entered". Not clear at this time if it is my code or it is a limitation.

The controller type is automatically taken from the VM template (image). Being able to actually specify the controller type or even change it as a day 2 operation would be also helpful. 

Sunday, April 18, 2021

What's new in vRealize Automation 8.4

 Last Friday vRealize Automation 8.4 was released and we are going to take a look at some of the new features. 

vRA vRO Plugin

The vRO plugin for vRA is back and it seems it is here to stay for good. This is one of the long waited come backs. There are several phases of development for the plugin and what we get now is phase 1 functionalities:

  • management of vRA on-premises and vRA Cloud hosts
  • preserver authentication to the hosts and dynamic host creation
  • REST client available allowing requests to vRA

The plugin is supported in vRA 8.3, but it has to be downloaded and installed manually. There seems to be a plan for VRO especially if we look back at support added for other languages such as Node.js, Python and PowerShell.   

Storage Enhancements

At storage level there are new features that improve visibility and management:
  • specify order in which the disks are created 
  • choose SCSI controller to which the disk is connected  
  • day 2 actions on the disks part of image template

Deploy multiple disks blueprint:

A more detailed article about disk management can be found here 

Azure Provisioning Enhancements

A series of new features is available for Azure integration:
  • support for Azure shared images 
  • Azure disk encryption set - encrypt VMs and attached disks and support 3rd party KMS 
  • Azure disk snapshot - create and mange disk snapshots with Azure deployments

ITSM Integration with ServiceNow Enhancements 

Foo those of you using ServiceNow as portal new new enhancements are brought for the integration with vRA. 
  • Support for Catalog Items which has Custom Resource (without for vRO Objects)
  • Support for Catalog Items with Custom Day 2 actions
  • Ability to customize vRA Catalog by adding Edit Box and Drop down in ServiceNow.
  • Ability to add to attach a script to these fields.
  • Deployment Details on available in ServicePortal
If you are using on-premises ServiceNow the integration this is not yet validated (seems it's on the way though).

Enhancements to Configuration Management Tools

The configuration management eco-system supported with vRA also got its enhancements (Puppet, SaltStack, Ansible)

This was just a short overview of the new features brought in by vRA 8.4. The full list can be read in the release notes.

Monday, March 1, 2021

Deploy VCSA Appliance with Terraform

I am back to an older project involving VMware products and Terraform. For those of you new to the subject, Terraform is an open source infrastructure as code tool developed by HashiCorp. It allows to define the entire infrastructure in a language called HashiCorp Configuration Language (HCL) and JSON files (where HCL is not enough). 

The interest for Terraform is its ability to easily deliver infrastructure across different infrastructures: public cloud, private cloud, Kubernetes. You write your configuration files, test it (with plan) and then you apply it to the infrastructure to get your resources deployed. There are other software tools that can be used such as HashiCorp Vault which is a secret management solution that can be consumed programmatically. In my example I will be using Vault to store the passwords required for setting up VCSA. 

In this example we will use Terraform to update the VCSA JSON template with values provided in a variable file and then run the VCSA cli installer. So we are not using the vSphere provider, rather local provider for modifying the template file and null provider to run a local command. I chose this example though because it is something I struggled to get it working. 

I've used the following simple project structure:

Templates folder contains VCSA modified template. Although all .tf files could be made into one (, I prefer this way of making the code more readable (and yes, has variables and has the Vault provider definition and the keys to secrets) defines 2 resources: update a template file and a command to execute 

resource "local_file" "vcsa_json" {
    content = templatefile (
              vc_fqdn = var.vcenterserver,
              vc_user = var.vcenterserver_user
              vc_user_pass =["value"],
              vm_network = var.pg_mgmt,
              vdc = var.vdc,
              datastore = var.datastore,
              host =,
              cluster = var.cluster,
              vcsa_name = element(split(".", var.vcsa_fqdn),0),
              vcsa_fqdn = var.vcsa_fqdn,
              vcsa_ip = var.vcsa_ip,
              prefix = var.prefix,
              gateway = var.gateway,
              dns = var.dns,
              vcsa_root_pass =["value"],
              ntp_servers = var.ntp,
              sso_password =["value"]
    filename = var.config_file_path

resource "null_resource" "vcsa_install" {
  provisioner "local-exec" {
    command = "${var.installcmd_file_path}/vcsa-deploy install --accept-eula 
            --acknowledge-ceip --no-esx-ssl-verify ${var.config_file_path}"

Local_file resource takes the template given by template_file_path variable and creates a configuration file at the path given in config_file_path variable. Null_resource executes a local command, in this case vcsa-deploy command to which we input updated configuration file. 

Within the template file you can see references to variables from (var.something) and also to data from (data.vault_generic_secret.some_path). Let's look at the the two files. 

variable "template_file_path" {
  description = "JSON template file path"
  type = string
  default = "templates/vcsa70_embedded_vCSA_on_VC.json"

variable "config_file_path" {
  description = "vcsa configuration JSON file path"
  type = string
  default = "/data/build/vcsa01_embedded_vCSA_on_VC.json"

variable "installcmd_file_path" {
  description = "command line file path"
  type = string
  default = "/data/VMware-VCSA-all-7.0.1-17491101/vcsa-cli-installer/lin64"

variable "vcsa_fqdn" {
  description = "vcsa hostname"
  default = "vcsa01.mylab.local"

variable "vcsa_ip" {
  description = "vcsa ip address"
  default = ""

variable "prefix" {
  description = "IP prefix"
  default = "24"

Each variable is defined by a name and a value. It can also have a description and a type (Please note that not all variables have been posted in this listing)

provider "vault" {
    address = ""
    token = "ABCD"
    skip_tls_verify = true

# vcsa deploy
data "vault_generic_secret" "vcsa_admin" {
    path = "kv-vmware-stgdev/administrator@vsphere.local"

data "vault_generic_secret" "vcsa_root" {
    path = "kv-vmware-stgdev/root"

The file contains the Vault provider definition and two keys for the VCSA admin and root passwords. 

template file (vcsa70_embedded_vCSA_on_VC.json) 

The values from and are updated in the template. To be able to update the default  template, you need to modify it first by adding keys that can be interpreted by Terraform provider. In my case I took the VCSA 7.0 embedded template and changed it as following:

    "__version": "2.13.0",
    "__comments": "Sample template to deploy a vCenter Server Appliance with an embedded Platform Services Controller on a vCenter Server instance.",
    "new_vcsa": {
        "vc": {
            "__comments": [
                "'datacenter' must end with a datacenter name, and only with a datacenter name. ",
                "'target' must end with an ESXi hostname, a cluster name, or a resource pool name. ",
                "The item 'Resources' must precede the resource pool name. ",
                "All names are case-sensitive. ",
                "For details and examples, refer to template help, i.e. vcsa-deploy {install|upgrade|migrate} --template-help"
            "hostname": "${vc_fqdn}",
            "username": "${vc_user}",
            "password": "${vc_user_pass}",
            "deployment_network": "${vm_network}",
            "datacenter": [
            "datastore": "${datastore}",
            "target": [
        "appliance": {
            "__comments": [
                "You must provide the 'deployment_option' key with a value, which will affect the vCenter Server Appliance's configuration parameters, such as the vCenter Server Appliance's number of vCPUs, the memory size, the storage size, and the maximum numbers of ESXi hosts and VMs which can be managed. For a list of acceptable values, run the supported deployment sizes help, i.e. vcsa-deploy --supported-deployment-sizes"
            "thin_disk_mode": true,
            "deployment_option": "small",
            "name": "${vcsa_name}"
        "network": {
            "ip_family": "ipv4",
            "mode": "static",
            "system_name": "${vcsa_fqdn}",
            "ip": "${vcsa_ip}",
            "prefix": "${prefix}",
            "gateway": "${gateway}",
            "dns_servers": [
        "os": {
            "password": "${vcsa_root_pass}",
            "ntp_servers": "${ntp_servers}",
            "ssh_enable": false
        "sso": {
            "password": "${sso_password}",
            "domain_name": "vsphere.local"
    "ceip": {
        "description": {
            "__comments": [
                "++++VMware Customer Experience Improvement Program (CEIP)++++",
                "VMware's Customer Experience Improvement Program (CEIP) ",
                "provides VMware with information that enables VMware to ",
                "improve its products and services, to fix problems, ",
                "and to advise you on how best to deploy and use our ",
                "products. As part of CEIP, VMware collects technical ",
                "information about your organization's use of VMware ",
                "products and services on a regular basis in association ",
                "with your organization's VMware license key(s). This ",
                "information does not personally identify any individual. ",
                "Additional information regarding the data collected ",
                "through CEIP and the purposes for which it is used by ",
                "VMware is set forth in the Trust & Assurance Center at ",
                " . If you ",
                "prefer not to participate in VMware's CEIP for this ",
                "product, you should disable CEIP by setting ",
                "'ceip_enabled': false. You may join or leave VMware's ",
                "CEIP for this product at any time. Please confirm your ",
                "acknowledgement by passing in the parameter ",
                "--acknowledge-ceip in the command line.",
        "settings": {
            "ceip_enabled": false

If you look at resource definition you will see the same keys from JSON file between {}.

Now all the code is written down and it's a simple matter of running terraform plan and terraform apply.