vSphere with Tanzu - Create custom VM template for VM Operator

 We've seen in the previous post how to enable and use VM Operator. We've also noticed that currently there are only 2 VM images that are supported to be deployed using VM Operator. What if we need to create our own image? 

There is a way, but the way is not supported by VMware. So once going this path, you have to understand the risks. 

What is so special about the VM image deployed using VM Operator? It is using cloud-init and OVF environment variables to initialize the VM. 

Let's start with a new Linux VM template. We will install VMware Tools. Then we need to install cloud-init. Once cloud init is installed update the configuration as following:

• in  /etc/cloud/cloud.cfg check the following value: disable_vmware_customization: true 
• setting it to true invokes traditional Guest Operating System Customization script based workflow (GOSC); in case it is set to false, cloud-init customization will be used. 

• create a new file /etc/cloud/cloud.cfg.d/99_vmservice.cfg and add the following line to it network: {config: disabled};
• this will actually prevent cloud-init to configure the network; you guessed, VMware Tools will be used to configure the network

Before exporting the VM as OVF template, run cloud-init to simulate a clean instance installation. It should be run on subsequent template updates too. 

Next we'll customize the OVF file itself. We need to enable OVF environment variables to be used to transport data to cloud-init. For this to work, I just copied the configuration from VMware CentOS VM service image ovf file and updated several sections: 

In <VirtualSystem ovf:id="vm">, add the following ovf properties. Please note that you could/should change the labels and descriptions to match your template

<ProductSection ovf:required="false">
  <Info>Cloud-Init customization</Info>
  <Product>Linux distribution for VMware VM Service</Product>
  <Property ovf:key="instance-id" ovf:type="string" ovf:userConfigurable="true" ovf:value="id-ovf">
      <Label>A Unique Instance ID for this instance</Label>
      <Description>Specifies the instance id.  This is required and used to determine if the machine should take "first boot" actions</Description>
  </Property>
  <Property ovf:key="hostname" ovf:type="string" ovf:userConfigurable="true" ovf:value="centosguest">
      <Description>Specifies the hostname for the appliance</Description>
  </Property>
  <Property ovf:key="seedfrom" ovf:type="string" ovf:userConfigurable="true">
      <Label>Url to seed instance data from</Label>
      <Description>This field is optional, but indicates that the instance should 'seed' user-data and meta-data from the given url.  If set to 'http://tinyurl.com/sm-' is given, meta-data will be pulled from http://tinyurl.com/sm-meta-data and user-data from http://tinyurl.com/sm-user-data.  Leave this empty if you do not want to seed from a url.</Description>
  </Property>
  <Property ovf:key="public-keys" ovf:type="string" ovf:userConfigurable="true" ovf:value="">
      <Label>ssh public keys</Label>
      <Description>This field is optional, but indicates that the instance should populate the default user's 'authorized_keys' with this value</Description>
  </Property>
  <Property ovf:key="user-data" ovf:type="string" ovf:userConfigurable="true" ovf:value="">
      <Label>Encoded user-data</Label>
      <Description>In order to fit into a xml attribute, this value is base64 encoded . It will be decoded, and then processed normally as user-data.</Description>
  </Property>
  <Property ovf:key="password" ovf:type="string" ovf:userConfigurable="true" ovf:value="">
      <Label>Default User's password</Label>
      <Description>If set, the default user's password will be set to this value to allow password based login.  The password will be good for only a single login.  If set to the string 'RANDOM' then a random password will be generated, and written to the console.</Description>
  </Property>
</ProductSection>

In <VirtualHardwareSection ovf:transport="iso">, add the following:

<vmw:ExtraConfig ovf:required="false" vmw:key="guestinfo.vmservice.defer-cloud-init" vmw:value="ready"/>

Save the OVF file and export it to the content library. The name must by DNS compliant and must not contain any capital letters. 

Lastly, in the YAML manifest file add disable checks done by VM Operator:

metadata:
  name: my-vm-name
  labels:
    app: db-server
  annotations:
    vmoperator.vmware.com/image-supported-check: disable

vSphere with Tanzu - VM Operator

VM Operator is an extension to Kubernetes that implements VM management through Kubernetes. It was released officially at end of April 2021 with vCenter Server 7.0 U2a. This is a small feature pushed through a vCenter Server patch that is bringing a huge shift in the paradigm of VM management. It changes the way we are looking at VMs and at the way we are using virtualization. One could argue that Kubernetes already did that. I would say that unifying resource consumption through VMs and pods is actually a huge step forward. VM Operator brings to play not only Infrastructure as Code (IaC), but it also enables GitOps for VMs.

Let's look briefly at the two concepts. IaC represents the capability to define your infrastructure in a human readable language. A lot of tools exist that enable IaC -  Puppet, Chef, Ansible, Terraform and so on. They are complex and powerful tools, some of them used in conjunction with others. All these tools have a particularity: they have their own language - Ruby, Python, HCL. GitOps expands the IaC concept. In this case, Git repository is the only source of truth. Manifests (configuration files that describe the resource to be provisioned) are pushed to a Git repository monitored by a continuous deployment (CD) tool that ensures that changes in the repository are applied in the real world. Kubernetes enables GitOps. Kubernetes manifests are written in YAML. With introduction of VM Operator the two concepts can be used in conjunction. For example you could have a GitOps pipeline that deploys the VMs using Kubernetes manifests and then configuration management tools could actually make sure the VMs are customized to suit  their purpose - deploying an application server, monitoring agents and so on. 

In the current post we will only look at the basics of deploying a VM through VM Operator. Once these concepts are clear then you can add other tools such as Git repositories, CD tools, configuration management. 

So, what do we need to be able to provision a VM through VM Operator? 

We need vCenter Server updated to U2a and a running Supervisor cluster. 

At namespace level a storage policies needs to be configured. It is needed for both VM deployment and persistent volumes 

We need a content library uploaded with a supported VMware template (we will follow soon with a post on how to create unsupported VMware templates for VM operator). At the time of writing CentOS 8 and Ubuntu images are distributed through VMware Marketplace (https://marketplace.cloud.vmware.com/ search for "VM service Image")

The images are installed with cloud-init and configured to transport user data using OVF environment variables to cloud-init process which in turn customizes the VM operating system. 

In Workload Management, VM Service allows the configuration of additional VM classes and content libraries. VM classes and content library are assigned to the namespace to be able to provision the VMs. 

VM classes selected for a particular namespace:

Content library selected for a particular namespace:

Once or the prerequisites are in place, connect to supervisor cluster, and select the namespace you want to deploy the VM

kubectl vsphere login --verbose 5 --server=https://192.168.2.1 --insecure-skip-tls-verify  -u cloudadmin@my.lab

kubectl config use-context my-app-namespace

Check that the VM images in the content library are available 

kubectl get virtualmachineimages

Create the VM manifest file - centos-db-2.yaml 

apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachine
metadata:
 name: centos-db-2
 labels:
  app: my-app-db
spec:
 imageName: centos-stream-8-vmservice-v1alpha1.20210222.8
 className: best-effort-xsmall
 powerState: poweredOn
 storageClass: tanzu-gold
 networkInterfaces:
  - networkType: nsx-t
 vmMetadata:
  configMapName: my-app-db-config
  transport: OvfEnv
---
apiVersion: v1
kind: ConfigMap
metadata:
 name: my-app-db-config
data:
 user-data: |
  I2Nsb3VkL6CiAgICBlbnMxNjA6CiAgICAgIGRoY3A0OiB0cnVlCg==
 hostname: centos-db-2

In the manifest file we've added 2 resources:

- VirtualMachine: where we specify the VM template to use, the VM class. storage policy, network type and also how to send variables to the cloud-init inside the VM (using a config map resource to keep the date in Kubernetes and OVF environment variables to transport it to the VM)

- ConfigMap: contains in our case user data (Base64 encoded user data - this is a SSH key) and the hostname of the VM; Base64 output in this post is trunked 

 To create the VM, apply the manifest. Then check its state.

kubectl apply -f centos-db-2.yaml 

kubectl get virtualmachine

Once the VM has been provisioned, it has been assigned an IP from the POD CIDR 

POD CIDRs are private subnets used for inter-pod communication. To access the VM, it needs an Ingress CIDR IP. This is a routable IP and it is implemented in NSX-T as a VIP on the load balancer. The Egress CIDR is used for communication from VM to outside world and it is implemented as SNAT rule. To define an ingress IP, we need to create a virtual machiner service resource of type load balancer:

Create the manifest file - service-ssh-centos-db-2.yaml 

apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachineService
metadata:
 name: lb-centos-db-2
spec:
 selector:
  app: my-app-db
 type: LoadBalancer
 ports:
  - name: ssh
    port: 22
    protocol: TCP
    targetPort: 22

We are using the selector app: my-app-db to match the VM resource for this service. The service will be assigned an IP from Ingress network and it will forward all requests coming to that IP on SSH port to the VM IP on SSH port. 

 To create the service, apply the manifest. Then check its state.

kubectl apply -f service-ssh-centos-db-2.yaml 

kubectl get service lb-centos-db-2

The External IP displayed in the above listing is the ingress IP that you can use now to ssh to the VM:

ssh cloud-user@external_ip

Please note the user used to SSH. From it, you can then sudo and gain root privileges. 

A VM provisioned via VM Operator can only be managed through the Supervisor cluster API (Kubernetes API). In this regard, the VM cannot be any longer managed directly from the UI or other management tools. Looking at the picture below you will notice that the VM is marked in UI as "Developer Managed" and that there are no actions that can be taken on the VM

If you are this far, then well done, you've just provisioned you first VM using Kubernetes API. Now put those manifests in a git repo, install and configure a CD tool (such as ArgoCD) to monitor the repo and apply the manifests on the Supervisor cluster and you don't even need to touch kubectl command line or vCenter Server :-) 

vSphere with Tanzu - Enable Supervisor Cluster using PowerCLI

In previous post we looked at how to manually enable Supervisor cluster on a vSphere cluster. Now we'll reproduce the same steps from GUI in a small script using PowerCLI. 

PowerCLI 12.1.0 brought new cmdlets for VMware.VimAutomation.WorkloadManagement module and one of this is Enable-WMCluster. We will be using this cmdlet to enable Tanzu supervisor cluster. In the following example we'll be using NSX-T, but the cmdlet can be used with distributed switches. 

The following script is very simple .First we need to connect to vCenter Server and NSX manager

Connect-VIServer -Server vc11.my.lab

Connect-NsxtServer -Server nsxt11.my.lab

Next we define the variables (all variable that were in the UI wizard).

The cluster where we enable Tanzu, the content library and the storage policies:

$vsphereCluster = Get-Cluster "MYCLUSTER"

$contentLibrary = "Tanzu subscribed"

$ephemeralStoragePolicy = "Tanzu gold"

$imageStoragePolicy = "Tanzu silver"

$masterStoragePolicy = "Tanzu gold"

Management network info for Supervisor Cluster VMs

$mgmtNetwork = Get-VirtualNetwork "Mgmt-Network"

$mgmtNetworkMode = "StaticRange"

$mgtmNetworkStartIPAddress = "192.168.100.160"

$mgtmNetworkRangeSize = "5"

$mgtmNetworkGateway = "192.168.100.1"

$mgtmNetworkSubnet = "255.255.255.0"

$distributedSwitch = Get-VDSwitch -Name "Distributed-Switch"

DNS and NTP servers

$masterDnsSearchDomain = "my.lab"

$masterDnsServer = "192.168.100.2"

$masterNtpServer = "192.168.100.5"

$workerDnsServer = "192.168.100.2"

Tanzu details - size and external and internal IP subnets

$size = "Tiny" 

$egressCIDR = "10.10.100.0/24"

$ingressCIDR = "10.10.200.0/24"

$serviceCIDR = "10.244.0.0/23"

$podCIDR = "10.96.0.0/23"

One more parameter needs to be provided: Edge cluster ID. For this we use NSX-T manager connectivity and 

$edgeClusterSvc = Get-NsxtService -Name com.vmware.nsx.edge_clusters

$results = $edgeClusterSvc.list().results

$edgeClusterId = ($results | Where {$_.display_name -eq "tanzu-edge-cluster"}).id

Last thing is to put all the parameters together in the cmdlet and run it against the vSphere cluster object

$vsphereCluster | Enable-WMCluster `

-SizeHint $size `

-ManagementVirtualNetwork $mgmtNetwork `

-ManagementNetworkMode $mgmtNetworkMode `

-ManagementNetworkStartIPAddress $mgtmNetworkStartIPAddress `

-ManagementNetworkAddressRangeSize $mgtmNetworkRangeSize `

-ManagementNetworkGateway $mgtmNetworkGateway `

-ManagementNetworkSubnetMask $mgtmNetworkSubnet `

-MasterDnsServerIPAddress $masterDnsServer `

-MasterNtpServer $masterNtpServer `

-MasterDnsSearchDomain $masterDnsSearchDomain `

-DistributedSwitch $distributedSwitch `

-NsxEdgeClusterId $edgeClusterId `

-ExternalEgressCIDRs $egressCIDR `

-ExternalIngressCIDRs $ingressCIDR `

-ServiceCIDR $serviceCIDR `

-PodCIDRs $podCIDR `

-WorkerDnsServer $workerDnsServer `

-EphemeralStoragePolicy $ephemeralStoragePolicy `

-ImageStoragePolicy $imageStoragePolicy `

-MasterStoragePolicy $masterStoragePolicy `

-ContentLibrary $contentLibrary

And as simple as that, the cluster will be enabled (in a scripted and repeatable way). 

vSphere with Tanzu - Enable Supervisor Cluster

Before diving head first into how to enable supervisor cluster it's important to clarify a few aspects. There are several great posts (here and here) on how to deploy automatically Tanzu on vSphere. The reason I choose to present a step by step guide is because going through the manual steps helped me clarifying some aspects. I will not be covering the networking part. There are two ways of enabling Tanzu on vSphere - using NSX-T or using vSphere networking and a load balancer. 

The Supervisor Cluster is a cluster enabled for vSphere with Tanzu. There is a one to one mapping between the Supervisor Cluster and the vSphere cluster. It is important because there features that are defined at Supervisor Cluster level only and inherited at Namespace level. A vSphere Namespace represents a set of resources where vSphere Pods, Tanzu Kubernetes clusters and VMs can run. It is similar to a resource pool in the sense that it brings together the compute and storage resources that can be consumed. A Supervisor Cluster can have many Namespaces, however at the time of writing there is a limit of 500 namespaces per vCenter Server. Depending on how you map namespaces to internal organizational units this can also be important. The high level architecture and components of a supervisor cluster can seen here. 

Requirements

• Configure NSX-T. Tanzu workloads need a T0 router configured on a edge cluster. All other objects (T1's, LB's, segments) are configured automatically during pod deployment. Edge recommended size is large, but it works with medium for lab deployments. Also for lab only, the edge cluster can run with a single edge node. Deploying and configuring NSX-T is not in the scope of this article

• vCenter Server level
• vSphere cluster with DRS and HA enabled
• content library for Tanzu Kubernetes cluster images subscribed to https://wp-content.vmware.com/v2/latest/lib.json. In case you don't have Internet connectivity from vCenter Server you will need to download them offline and upload to the library. Check if you can have access to Internet via  a proxy and you can add the proxy in vCenter Server VAMI interface (https://vcs_fqdn:5480) 
• storage policies - for lab purpose one policy can be created and used for all types of storage. Go to Policies and Profiles and create a new VM Storage Profile - Enable host based rules and select Storage I/O Control

• IP's - for ingress and egress traffic (routed), pod and service (internal traffic) 

• latest version of  vCenter Server  - 7.0 U2a (required for some of the new functionalities - vm operator and namespace self service)

• NTP working and configured for vCenter Server  and NSX-T manager (and the rest of components) 

Enabling the Supervisor Cluster is pretty straight forward - go to workload management, clusters and add cluster. The wizard will take you through the following steps. 

First select vCenter Server and the type of networking. If you don't have NSX-T configured, then you can use vSphere Distributed Switch but first a load balancer needs to be installed (HAproxy or AVI)

Then you select the vSphere cluster where to enable the Supervisor cluster. 

Choose the size of the control plane VMs - the smaller they are the smaller the Kubernetes environment.

Map storage policies to types of storage in the Supervisor cluster

Add management network details. It is important to clarify that the supervisor VMs have 2 NIC's - one connected to vSphere distributed portgroup that has access to vCenter Server and NSX-T manager and another one connected to Kubernetes service network. Please check the "View Network Topology" in the step to have a clear picture of the configuration of the Supervisor VM. Also supervisor VMs need a range of 5 IPs free that will be use - in my case I am selecting a range from the management network.  

Next add the network details for ingress and egress networks and also for internal cluster networks (service and pod). Ingress and egress networks are used to access services inside the Kubernetes cluster  via DNAT (ingress) and by internal services to access outside world via SNAT (egress). 

In case you use the same DNS server for management and service networks, the server must be reachable over both interfaces. Service network will use the IP of the egress network to reach DNS. 

Lastly, add the content library, review the configuration and give it a run. 

 Once the cluster is deployed successfully you will see it in the ready state:

You can now create namespaces and Kubernetes guest clusters. To access the cluster you will need to connect to https://cluster_ip and download kubectl vSphere plugin. 

Since we got through all the manual steps, we can look next at automating the configuration using PowerCLI in the next post.

Surprises at VMworld 2019

It's been an interesting year where VMware announced a ton of new and exciting stuff from projects Pacific and Tanzu to integrating Carbon Black to almost all of the products. Hence this VMworld has been one of the best so far: kubernetes, containers, integrated security, even more networking, infrastructure as a code, terraform, ansible, machine learning... almost forgot, vSphere on ARM. A lot of words and talks not in the traditional virtualization admin's vocabulary. Things are changing and doing it fast. Looking at how VMware redefined itself, caught up with missed trains and how it will lead some of those trains, I think we are going through some of most radical shifts in IT and I personally feel like I've missed at least one train.

I am an old style IT guy: started installing servers in racks and learning by heart RJ45 cross and direct cabling schemes. When cloud came along, it was not very difficult to apply what I had learnt in datacenters. But I stayed somehow focused on the private and sometimes hybrid cloud. Never went fully cloud native, never gotten knee deep into deploying and operating applications in the public cloud. Naturally, DevOps went from hype to mainstream. And as time passed by, machine learning came along and AI started doing crazy stuff like beating humans at League of Legends (I did play a lot of DotA back in the days and not a single game of Go :-) ) A lot happened in the last 10 years.

I've been going to VMworlds since 2013 and kind of grew up with it. Still, this year I felt like I've been taken a bit by surprise. VMworld is, as always, a place to meet old and new friends and a place to learn about new trends. Barcelona adds to that whole vibe of the event. It was a blast to see people I've worked with on different projects years and years ago or to meet my fellow VMUG leaders.

As for the technical part:
- vSphere on ARM looks promising - comparing to last year there were several devices installed with ESXi  from tablets, to network cards to high performance ARM 2U rackable servers
- machine learning and Watson applied to log monitoring and analysis
- vRA 8 redesign - container based, no Windows
- NSX-T everywhere - in your DC, in the cloud and  with a single pane of glass
- can't wait for the next vSphere release

Solution exchange walk was also interesting:
- if AWS presence was expected, having Azure and Oracle (although the latter was a bit hidden) was cool
- most of the traditional networking/security vendors were not present anymore
- LG and Samsung had pretty big stands
- a lot of backup vendors old and new
- ComputerWeekly was also there with a stand
- Penetration Testing as a Service - a platform from where you could hire pen testers to do some... pen testing
- VMware's booth is getting bigger and bigger

My take away from last week is I have to do a lot of reading and learning, way more than the usual, but this is what makes IT interesting.