TCP ports "greater than 65535"

Let me tell this nice story. And yes, there are no TCP ports above 65535, but earlier today after receiving the credentials, I connected without a second thought to A.B.C.D:73382. I was even happy it worked, because the first time I tried "by mistake" port 63382 and I had problems with the login. And this time it worked. But there is a problem - the port number should have been smaller than 65535. Still, take a look at the following picture and notice the strange TCP/IP connection:

It is TCP port 73382. And the connection works (more or less optimal, but it works). But what actually happens is the interesting part. The connection in the image is a port translation from 73382 to 3389. This means there is a device doing the translation - in our case a linux box. And there is a program listening on port 73382. Well, not exactly...

 

TCP header uses 16 bits for representing the source port and 16 bits for representing the destination port. 

The 16 bits limit the maximum number of ports to 65536 (2^16) having the numbers between 0 and 65535. Number 73382 needs actually 17 bits for its binary form - 10001111010100110. The only way to fit 17 bits into 16 bits is to truncate the 17 bits. The only question is what part to truncate - left bit or the right bit. To find out the answer I used good old wireshark:

The communication takes place on port 7846 - which on 16 bits looks like this 0001111010100110. So, it truncates the left most bit.What happens is that a small piece of code (perl or python) will successfully  accept to open a socket using numbers bigger 65535 - only it  fits the number on 16 bits. Remember, on a linux system "everything is a file; if it is not file, it is a process".
There are client programs that using a simple input check do not allow using strange numbers, however RDC is not one of them. And after all, it was actually a nice debug.

Command line upgrade of ESXi 5.1 to 5.1 U1

The reason I`ve chosen this method is not having VUM installed and also wanting to remember a bit of esxcli. The process is pretty simple:

• download the zip bundle (update-from-esxi5.1-5.1_update01.zip) and put it on a datastore accessible by all hosts (I am using a NFS share)
• check to see if the host has the datastore mounted, if not - add the datastore

~ # esxcli storage nfs list

~ # esxcli storage nfs add -H 192.168.1.1 -s /mnt/vol1-nfs -v shared1-nfs-sata

• check the host version

~ # esxcli system version get
   Product: VMware ESXi
   Version: 5.1.0
   Build: Releasebuild-799733
   Update: 0

• check the update file and see if it requires the host in maintenance mode (in this case, it does not - however since the update will request a system reboot, I think it is better to place the host in maintenance mode and have it cleaned up of VMs)

~ # esxcli software sources vib get --depot=/vmfs/volumes/shared1-nfs-sata/update-from-esxi5.1-5.1_update01.zip
...
   Maintenance Mode Required: False

• install the update

~ # esxcli software vib update --depot=/vmfs/volumes/shared1-nfs-sata/update-from-esxi5.1-5.1_update01.zip

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true

• reboot the system (after all VMs have been migrated to other hosts) and check the new host version

~ # reboot
...
~ # esxcli system version get
   Product: VMware ESXi
   Version: 5.1.0
   Build: Releasebuild-1065491
   Update: 1

vCloud Director 5.1 upgrade and missing sysadmin permissions

I`ve recently managed to crash my test lab. One of the rebuild tasks was to reinstall vCloud Director - the hard way: dropping the DB and recreating it. Next I was running the vCloud Director installation script (5.1.1), recreating the DB schema and so on. After a week I`ve decided to upgrade the vCD installation from 5.1.1 to 5.1.2.

After downloading the bin file (vmware-vcloud-director-5.1.2-1068441.bin) from VMware site and uploading it on my vCD cell , I`ve executed the bin and a long list of "Yes" ends up with an error:

Unable to update database statistics. Database user has insufficient privileges to update database statistics. To complete this step manually, run 'EXEC sp_updatestats' as the DBO or a member of the sysadmin fixed server role.

Next, I`ve been looking in the DB server (MSSQL) - logging in with vCD DB admin user (vcddbadmin) and checking if the rights are ok. And they were, since the user was dbowner. I`ve ran 'EXEC sp_updatestats' in query editor and ended up with another error:

Msg 15247, Level 16, State 1, Procedure sp_updatestats, Line 15
User does not have permission to perform this action.

It was stored procedure time: vcddb01 -> Programmability ->Stored Procedures -> System Store Procedures and after a few pages of scrolling down: sys.sp_updatestats. I`ve opened the stored procedure (Context menu - Modify) and on line 15 there was a check to see if the current user was sysadmin. Which, obviously it was not.

Below, a small SQL script that will display if the current user is sysadmin or not:

if is_srvrolemember('sysadmin') = 1
print  ' It is sys admin'
else if is_srvrolemember('sysadmin') = 0
print  ' It is NOT sys admin'

To solve the problem, I`ve added sysadmin role to the vCloud Director DB admin user (vcddbadmin) and ran again the stored procedure:

This time it worked ok and updated database statistics. I`ve checked back the installation procedure for vCloud DB and I`ve taken a look at KB 2034540 to see if I`veforgotten anything while creating the user for vCD DB. But it seems I did not. The only reference is to db_owner role. Since I am not a DB guy, I`ll just leave it as is, happy to have solved my problem. 

Command line installation of vSphere Auto Deploy

After a long absence, partially motivated by changing my job while organizing a VMUG meeting, I`ve found a bit of time to restart working at the blog.

In  this post I am presenting command line installation of Auto Deploy. It is assumed that vCenter Server exists (in this case called vcsrv01.domain.test). The installation file is also needed - it can be found in vCenter ISO at vctools\VMware-autodeploy.exe. Before running any command, copy the exe file on the server where it will be installed or on a centralized repository (file share). Open a command prompt and run the command line installation:

start /wait D:\VC\vctools\VMware-autodeploy.exe /L1033 /v" /qn VM_PATH_DATA=D:\DATA VM_DATA_SIZE=4 VM_PORT=6501 VM_PORTADD=6502 VC_SERVER_ADDRESS=vcsrv01.domain.test VC_SERVER_PORT=80  VC_SERVER_USER=VC-Admin VC_SERVER_PASSWORD=AdminPass" 

Two types of arguments are passed in the command line:

• first arguments for windows installer
• /L1033 - english language
• /qn - no user interface 
• /v" " - passes arguments to msi
•  second - software specific arguments:
• VM_PATH_DATA - root destination data directory for the installation 
• VM_DATA_SIZE - space for VM_PATH_DATA in GB
• VM_PORT - Auto Deploy service TCP port number
• VM_PORTADD - Auto Deploy management TCP port number
• VC_SERVER_ADDRESS - IP or FQDN for vCenter Server
• VC_SERVER_PORT - vCenter Server port number
• VC_SERVER_USER - vCenter Server user name
• VC_SERVER_PASSWORD - vCenter Server user password

Several VMware components can be installed or upgraded using command line parameters. For more information see the following technical paper from VMware.

VMware Distributed Power Management (DPM)

One of the features Enterprise license brings to VMware vSphere is Distributed Power Management (DPM). This feature saves power by dynamically adjusting cluster capacity according to existing workloads. DPM will power off and power on hosts in the cluster based on load average of each host. During power off, VMs are automatically consolidated on remaining hosts. 

For powering on ESXi hosts, DPM uses one of the following technologies: iLO, IPMI or WoL. WoL packets (Magic packets) are sent over vMotion interface by another host in the cluster. DPM puts hosts into so called "standby" by actually entering the host in ACPI S5 power state. In this state power consumption is minimal, no user mode or system code is run and the systems`s context is not preserved. Some components are still running to ensure starting of the host. This state is also called "soft off".

DPM evaluates CPU and memory utilization for each host in the cluster and tries to keep the host within a specific range. By default utilization range is 45% to 81%. The range is computed from two settings:
DemandCapacityRatioTarget = utilization target of the host - by default it is 63%
DemandCapacityRatioToleranceHost = variation around utilization target - by default it is 18%
Utilization range = DemandCapacityRatioTarget +/- DemandCapacityRatioToleranceHost
Default settings can be changed in DRS - Advanced settings and can be set between 40% and 90% for DemandCapacityRatioTarget and between 10% and 40% for DemandCapacityRatioToleranceHost.

DPM uses two time intervals for evaluating power on and power off recommendations. For power on the period is 300 seconds (5  minutes), while for power off the period is 2400 seconds (40 minutes). This means DPM considers more important responses to increased load. However, this also means that a sudden increase in the load will be be considered by DPM after 5 minutes and will be resolved after the host boots up, which may add another 5 to 10 minutes. The values can be changed by setting parameters VmDemandHistorySecsHostOn (default 300 seconds) and VmDemandHistorySecsHostOff (default 2400 seconds) to a value between 0 and 3600 seconds.


DPM will ensure that at least one host will be left running. The settings MinPoweredOnCpuCapacity (default 1 MHz) and MinPoweredOnMemCapacity (default 1 MB) are used to control how many hosts will be left running. Default values ensure minimum one host is up and can be changed. For example if the cluster has hosts configured with 24 GHz and 128 GB setting the parameters 24001 MHz and 131073 MB will reserve 2 hosts running at all time. Even with default values, when HA cluster is enabled, DPM will leave 2 hosts powered on to provide fail over resources in case of one host failing.

For more details about how DPM selects hosts for power off and cost/benefit analysis, I strongly recommend the book VMware vSphere Clustering Deepdive and the white paper VMware Distributed Power Management: Concepts and Usage

Configuring DPM

Before enabling DPM on a cluster it is best to test that hosts can be powered off and started. This can be done from vSphere client by putting a host in standby. Check first that the network adapter supports WoL:

Put the host in standby and confirm moving all powered off and suspended VMs to other hosts in the cluster:

After the host is powered off, bring it back from vSphere client:

After all hosts in the cluster have been tested, one last check can be done by going to Cluster - vSphere DRS - Host Options and view if hosts succeeded to exit standby:

Next, enable DPM and choose one of the 3 power management policies:

If DPM behavior needs to be controlled, a very good idea is to have it enabled during NBH and disabled during BH. Set power management to Automatic and add 2 scheduled tasks of type Change cluster power settings in vSphere - one to enable DPM in the evening (19.00) and another one to disable DPM in the morning ( 7.00).  This way during BH all hosts are running at full capacity and during night the data center takes full advantage of power savings. 

PowerCLI - Batch move templates

While preparing for configuring DPM on one of the clusters, I noticed that templates are scattered around all the hosts. A design best practice for DPM states to put all your templates on one host and disable DPM on that host - this way you will not get the surprise of not being able to access some of your templates because the host was powered off. Remembering that I decided to bring all templates in one place. First I took a look at PowerCLI cmdlets - get-template and move-template and hoped for an easy job. However the cmdlets are a bit tricky:

• get-template - documentation states that it takes as input for Location "vSphere container objects (such as folders, data centers, and clusters) you want to search for templates". It is partialy true, since it does not accept clusters, but it accepts hosts;
• move-template -  Destination parameter used to "specify a container object where you want to move the templates" accepts only Datacenters and Folders.

My use case is input a cluster name and consolidate all templates on a single host. The next best thing was to use the cmdlets above and make my little script. The script takes as input 2 parameters: cluster and destination host. It gets all templates from cluster, converts them to VMs, migrates the VMs to destination host and reconverts them to template.

Function MoveTemplate()

{

Param(

[Parameter(Mandatory=$True,Position=0)][string]$cluster,

[Parameter(Mandatory=$True,Position=1)][string]$dsthost

)

get-cluster $cluster | get-vmhost | foreach {

get-template -Location $_ | foreach {

Write-Host "moving $_"

get-vm (set-template -Template $_ -ToVM).Name | move-vm -Destination $dsthost | set-vm -ToTemplate -Confirm:$false 

}

}

}

MoveTemplate

vShield Edge configuration and operational issues

In this post I will cover some of the aspects of using vShield Edge (vSE) devices at vApp level, mostly things I have learnt the hard way. I am currently using vCloud 1.5 and the aspects herein are related to this version. However, I will specify changes that appear in vCD 5.1. 

In vApp diagram, vSE is represented interposed between the vApp network and the organization network:

The symbol displays a firewall, if firewall services are active. vSE offers the following services:

• DHCP
• firewall
• routing 
• NAT

All configuration is done using vCD portal. DHCP services can be created for vApp network and the configuration is basic:

It is not possible to create MAC reservations, but basic service functions are available.

Firewall configuration has by default Deny policy enabled and allows any traffic going outbound:

vSE can be configured to send syslog messages to a central server, and it is highly recommended to do it if you want to debug anything. But the syslog server can only be configured globally at vCD level, not separately for each organization. If you are an organization admin, it will be difficult, if not impossible to get access to syslog server.. Other issue is that the graphical interface does not allow copy or clone rules.Which means a lot of manual work. To make things worse, you cannot add two different hosts or subnets in the source or destination field - no possibility to group more objects. More manual work. I do not know why they have provided such a bad interface to vSE.

Next service is NAT - by default vSE is configured in IP translation mode. VMs are assigned IPs from Organization network and vSE does the rest: source and destination NAT. 

Org network IPs can be statically assigned or automatically. But, they cannot be unassigned from a VM. If there are 10 VMs connected to the vApp network, then all 10 will be assigned IPs from the Organization network even if only 1 VM needs such an IP (simple use case - one arm load balancing topology). Adding the 1 IP used by vSE, a total of 11 IPs will be used. This is a problem since you might get into situations where you run out of IPs at Organization level (I did).

The only solution is to change NAT mode of vSE to Port forwarding. In which case you are limited to the IP of vSE. All requests are sent to vSE`s IP and it vSE does basic port forwarding:

Do not think of publishing in the vApp two services on the same port. You will not be able to do it.

A difference in 5.1 is that you can specify source and destination NAT separately.

Last feature that vSE provides is static routing. There is not much to talk about it - it allows adding of static routes:

One more limitation of vSE in 1.5 is that it has only 2 interfaces: one external and one internal. In 5.1 there are multiple network interfaces provided to external networks.

vCloud Director 5.1 brings other changes from 1.5:

• IP masquerading setting was removed
• there is no default rule on the firewall

More details about these two changes can be found here

Unfortunately, vSE brings too many technical limitations to be a useful firewall and in many cases architectural decisions have to be made to not use vSE. Changing vSE with other technologies that provide better firewall, NAT and routing functions also brings its own technological challenges.