Showing posts with label NSX. Show all posts
Showing posts with label NSX. Show all posts

Wednesday, December 2, 2020

Using NSX load balancer as a monitoring tool for RESTful APIs

We are going to look at a different use case for NSX load balancer - monitoring tool for external API's. 

Our core platform is integrating with other systems using RESTful API's. These systems even though they are built with high availability in mind, they are sometimes highly unavailable. They are also part of the critical path for our core platform. Not being able to reach the systems creates troubles  in the form of incident tickets because we fail to deliver services to our customers. So we needed a way to monitor those API's.

We know that the systems are monitored, but we don't have access to those tools. We have our own tools, but they do not offer a simple and efficient way to check the status of a RESTful API. Ideally we don't want to introduce another monitoring tool. However, the core platform is running on top of NSX and it actually uses NSX load balancers for its internal service. So why not use load balancers to monitor the external services? 

We created a service monitor and a pool in the load balancer for each of the external systems. This way NSX monitors the status of the RESTful API of the system and generates alerts whenever it is down. The status of the pool is then checked by the core platform. All communication between core platform and APIs goes directly. It does not use the load balancer.

The pool contains the RESTful API endpoint of the system that we use to connect directly from the core platform. 

The service monitor uses GET requests to check the availability of the RESTful API. 


Nothing fancy, basic configuration for a load balancer. Half-configuration actually because here we stop as no traffic goes through the load balancer to these pools. But whenever the external system is not reachable, the load balancer knows it because now the external system is a member in on of its pools: 


The status of the member in the pool is accessible through the RESTful API of NSX Manager. 

GET /api/4.0/edges/{edge-id}/loadbalancer/statistics

<status>DOWN</status>
<failureCause>layer 7 response error, code:400 Bad Request</failureCause>
<lastStateChangeTime>2020-12-02 18:20:43</lastStateChangeTime>

This way the core platform knows the status of its external systems before doing anything. More important core platform can now act on that status. In this case it will wait a specific period of time until trying again to use the system. 

It is a pretty simple solution. It is also pretty obvious that the APIs should have been monitored. We actually relied too much on the availability of those API's and used a fire and forget approach. The approach was far from optimal and it impacted our KPIs and created additional operational workload. 

Thursday, June 14, 2018

Automating NSX with PowerShell and RESTful API

Often, there are situations when one needs to do the same actions over and over again. System admins solved this repetitive tasks by scripting them. Scripting languages were however limited to the operating system or in some cases to a middleware application. New applications provide RESTful APIs that extend the power of scripting languages from the OS level to virtually every application available in the infrastructure. Using RESTful APIs one can create complex workflows that execute orchestrated actions on different applications. This brings a new word into the vocabulary: automation.

Let's take a simple example - new developer is hired and needs a dedicated dev and test environment. We are using VMware vSphere and NSX to provide that environment. Because we don't want to interfere with other environments, we want to isolate it. For isolating environments at network level we can use Edge Services Gateway (ESG) and dedicated VXLAN. At vSphere level we need to control resource consumption. If there is vRealize Automation or vCloud Director, these tasks can be easily automated from the GUI. What is there is no Cloud Management Portal, or the one the exists is not integrated with NSX and vSphere. Then we need to go back to scripting language.

In this post we'll take a look at how to call NSX RESTful API using PowerShell. There are languages that can be used: ruby, perl, python, JavaScript, depends on the preference. There is also a PowerShell extension for NSX called PowerNSX. The scope of the post is to see how we can use PowerShell to consume RESTful APIs (as the principles will apply to any API) using NSX.


Before we begin, a few words about the environment. I am using PowerShell 6.0.1, downloaded latest stable version from GitHub. The reason for doing this is that in version prior to 6.x there are issues with getting PowerShell methods to trust self signed SSL certificates. In 6.x you can add  -SkipCertificateCheck parameter to both Invoke-RestMethod and Invoke-WebRequest to accept self signed certificates.



Any RESTful API connection needs to be authenticated. Let's create the authentication header that will be sent with the request. We need to get the username and password and encode it to Base 64 string. The password can be simply put as a string in code, but I would like to complicate things and request the password from the user as a secure string (characters masked with star symbols)

1
2
$username = "nsxAdmin"
$securedValue = Read-Host "Enter password" -AsSecureString

After we get the password, we need to decrypt the secure string from unmanaged memory and write it in a string variable. We use the following methods: PtrToStringAuto and SecureStringToBSTR.

1
2
$password = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($securedValue))
$userpass  = $username + ":" + $password

We encode the username and password to Base64 and create the header.

1
2
3
4
5
$bytes= [System.Text.Encoding]::UTF8.GetBytes($userpass)
$encodedlogin=[Convert]::ToBase64String($bytes)
$authheader = "Basic " + $encodedlogin
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization",$authheader)

Lastly we create the URI (will get existing edges) and run the query. The output will be sent to a XML file.


1
2
3
$uri = "https://nsx-manager/api/4.0/edges"
$outXml = $baseFilePath + 'allEsg.xml'
Invoke-RestMethod -Uri $uri -Headers $headers -Method 'GET' -OutFile $outXml -SkipCertificateCheck

We have the XML file with all existing edges. Let's get vnic configuration for each edge. First we load the XML file and look for edge child nodes - pagedEdgeList.edgePage.edgeSummary. For each edge we find, we use the objectId to query the API for its vnics and put that result into a file.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
$esgObjectArray = @()
[xml]$config = Get-Content $outXml
foreach ($esg in $config.pagedEdgeList.edgePage.edgeSummary) {
  $esgObject = New-Object PSObject -property @{Id=$esg.objectId;Name=$esg.name}

  $uri = 'https://nsx-manager/api/4.0/edges/' + $esg.objectId + '/vnics'
  $outXml = $baseFilePath + $esg.objectId + '-vnics.xml'
  try {
    Invoke-RestMethod -Uri $uri -Headers $headers -Method 'GET' -OutFile $outXml -ea stop -SkipCertificateCheck
  }
  catch {
    Write-Host $_ -foreground green
  }

  $esgObjectArray += $esgObject
}

Within the foreach loop we are creating an array ($esgObjectArray) that contains the edge id and the edge name. $esgObject object is used to get the parameters for each iteration of the loop and add them to the array. This will be used to get the IP addresses for each edge and export all in a CSV file.

The variable $baseFilePath can be any destination on your local system where you want to place output files (e.g "D:\scripts\myEsgConfig\").

Finally we'll export edge ID, edge name and its IP addresses to a csv file. We'll loop through $esgObjectArray and get the information from the saved xml configuration files for each edge (-vnics.xml files):


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$esgVnics = @()
foreach ($esg in $esgObjectArray) {
  $outXml = $baseFilePath + $esg.Id + '-vnics.xml'
  if (Test-Path -Path $outXml) {
    $vnics = ""
    $item = ""
    [xml]$config = Get-Content $outXml
    foreach ($interface in $config.vnics.vnic) {
      $vnics += "," + $interface.addressGroups.addressGroup.primaryAddress
    }
    $item = $esg.Id + "," + $esg.Name + $vnics
    Write-Host $item
  } else { continue}
  $esgVnics += $item
}

$esgVnics | foreach { Add-Content -Path  $csvFile -Value $_ }

In the end we'll get a CSV file with the following format:

As you've seen, we used Invoke-RestMethod to query the API and saved the results in XML files for later use. You can use other REST methods like PUT, POST or DELETE to manage NSX environment. In the end it is all about manipulating creating and manipulating XML/JSON content to do the operations from command line .

Tuesday, May 15, 2018

NSX integration with vRealize Automation 7.4 - part 2

In part 1 of this post we presented the configuration at vRA level. In this post we'll see how to create a service in the Service Catalog for  programmatic NSX consumption.

First let's remember the main concepts of vRealize Automation service catalog:

  • catalog items are published in the service catalog for user consumption e.g Linux VM or 3 tier web app
  • catalog items can be grouped under different services: QA, Test&Dev, Web Apps, Linux Servers
  • a user is allowed to request a service item based on his entitlements; Entitlements define who has access to catalog items and what actions he can do 

For start, we'll create a service called Linux VMs and a new Entitlement called Allow Linux VMs. We'll entitle all users of the business group to the Linux VMs service. Using services in the entitlement instead of individual items we make sure that every new item mapped to this service will be automatically accessible to the users.

Administration > Catalog Management > Services

Administration > Catalog Management > Entitlements




Next we'll create a blueprint that deploys vSphere VMs. There are several ways to provision vSphere VMs, we will use linked clones because they are very fast and use deltas to keep the changes (which is good in labs). To use linked clones we need to create a golden image: a VM configured to the desired state.

First create the VM: deploy it from an existing template or create it from scratch. VM hostname and networking details will be configured at deployment during guest OS customization. For this to work we need VMware tools installed in the VM and a customization specification created in vCenter Server. 


No other special configuration is needed for the VM.

Optional step (vRA agent installation): if you don't plan to run scripts inside the guest OS of the vRA manged VM, you can skip this step. The installation should be pretty easy since VMware already provides a script that can handle it. Go to vRA appliance URL and download the script on your Linux VM:

 wget  https://vra_app_fqdn/software/download/prepare_vra_template_linux.tar.gz --no-check-certificate

Then extract the script from the archive and run it:

tar -xvzf prepare_vra_template_linux.tar.gz
cd prepare_vra_template_linux
./prepare_vra_template.sh  

Choose default agent type (vSphere), add the address of vRealize Appliance, manager service, accept the key fingerprints for the certificates, set the download timeout, and install JRE (if not already in the VM)




Now we have a VM with all the utils inside (VMware tools and optionally vRA agent) and we create the snapshot that will be the base for linked clones.

At this point we login to vRA portal and we start working on our service creation. Go to Design > Blueprints. Start creating a New blueprint. Type the name of the blueprint, assign an unique ID or leave the automatically generated one, limit the number of deployments per request (if you want). Add lease days to control the spawn of deployed VMs (especially for temporary environments) and add a period of time you want the item to be archived before deleting (when lease expires).

Since this is for demo, I've added a default lease of 1 day and no archival (automatic deletion after lease expires). On NSX Settings tab, choose the NSX transport zone and if you want to isolate the VMs deployed from this blueprint (allow only internal traffic between VMs).







Pressing OK button will take you to the canvas. From Machine Types category, drag and drop vSphere (vCenter) Machine


From Network&Security category, drag and drop On-Demand Routed Network


Select the vSphere__vCenter__Machine_1 components form the canvas and fill in the configuration details. Add the number of instances that can be deployed in a request 


Add build information: how the VM will be created (linked clone), where to clone it from, what customization specification to use:


Type the VM consumed resources: number of CPUs, memory, storage. Take care when configuring these values: if you allow 10 instances in a deployment, each instance with a maximum of 8 vCPU and 32 GB of RAM you may end up with a deployment using 80 vCPU and 320 GB of RAM. This is a good moment when approval workflows come into place.



Finally we need to connect the VM to the network. But first we'll configure the network component. On canvas select the On-Demand_Routed_Network_1 components and choose the parent network profile (profile that has been created in part 1) 


Go back to vSphere component, go to Network tab and click New. From drop down box select the network name

Lastly, add a custom property for the VM to define the operating system that is being used


At this moment we've configured how to create the VM, how to create the network and linked the VM to the network. Press Finish and then Publish the blueprint:


Once the blueprint has been published, it will appear under Administration > Catalog Management > Catalog Items. Select the new catalog item, press Configure and map it to the service created earlier at the beginning of the post. 


The service will appear in the Catalog tab and you can press Request to deploy a new instance of it. To see what is happening, go to Requests tab, select the request, press View Details and when the request details open press Execution Information


Here you will see that the vxlan has been created on demand and DLR reconfigured. Also the VM has been created and mapped to the new vxlan. The process can also be monitored in vCenter Server


After the provisioning finished successfully, the components will be displayed in Items tab from where they can be managed using day 2 operations.

Friday, May 4, 2018

NSX integration with vRealize Automation 7.4 - part 1

From time to time I find myself facing a new configuration of vRealize Automation. And since I don't do it very often I also find myself having forgotten some steps. For this reason I will write down my typical integration of NSX with vRA.

The post has two parts:

  • part 1 - describes the vRA configuration at infrastructure/tenant level
  • part 2 - focuses on creation of the service that consumes NSX


Requirements:

  • NSX  ( > 6.3.x) is installed and configured (VXLAN, distributed logical router, edge services gateways)
  • vRA 7.4 is deployed, tenant created, user directory integrated (optional) 
  • familiarity with vRA 


Goal: 

  • all vRA workloads are deployed to on-demand created networks and we do not worry about routing or virtual network creation
  • security for workloads is ensured using distributed firewall and security tags (but more on this in another post). 


First thing first, drawing a small diagram (sometimes my creative side kicks in :-)) of the desired state:


To solve L2/L3 requirements, we need the following:

  • vRA uses on demand created VXLAN's that use Distributed Logical Router (DLR) as default gateway
  • DLR is connected to Edge Services Gateway (ESG) via a transit network
  • dynamic routing protocol is running between DLR and ESG
  • ESG connects VRA workloads to the rest of the world via the External portgroup (which is a distributed portgroup)
  • ESG may also run dynamic routing protocols
(please don't judge the usage of /24 in the diagram, in real life I am subnetting)

As stated earlier, NSX is already configured, DLR and ESG deployed. Let's see how to configure vRA. 

Logon to the tenant as IaaS Administrator. Ideally, you would have IaaS Administrator + Tenant Administrator roles assigned to your account so you don't need to change between roles. We  need to create vSphere endpoint, fabric group, business group, NSX endpoint, reservation. 



Create vSphere (vCenter) Endpoint:

  • go to Infrastructure > Endpoints > Endpoints
  • New > Virtual > vSphere (vCenter)
  • on General tab: give it a name (vcenter1), add the URL of vCenter Server API (https://vcenter1.mydomain.local/sdk, add credentials and Test Connection. 
  • If all good, press OK and you have created your endpoint.  

Create NSX Endpoint:

  • go to Infrastructure > Endpoints > Endpoints
  • New > Network and Security > NSX
  • on General tab: type the name of the endpoint, add the  URL to the NSX manager and the credentials
  • on Associations tab: map NSX endpoint with the previously created vSphere endpoint (this is a step that appeared in vRA 7.3 due to changes on how NSX is integrated) 

  • press Test Connection, and then OK


Create Fabric Group:
  • go to Infrastructure > Endpoints > Fabric Group > New
  • give it a name, add the Fabric administrators (users, user groups) and select the compute resources available (the list of compute resources is based on the permissions the user has in vCenter Server)


Now the compute resources are available (Infrastructure > Compute Resources > Compute Resources). Check on the compute resource that data collection has run successfully - hover on the compute resource and from the menu choose Data Collection. 

Create Business Group:

  • go to Infrastructure > Users & Groups > Business Groups > New
  • on General tab: type in the name, add an e-mail for alerts, test
  • on Members tab: add the users/user groups for the following roles - Group manager role, Support role, Shared access role,User role and press ok
Starting with vRA 7.3 there is a new role for Business group - Shared access role which can use and run actions on resources deployed by other users in the business group. It is a good addition since I remember a client wanting this back in 2015.  

Create Network Profiles
  • go to Infrastructure > Reservations > Network Profiles
  • New > External 
  • on General tab: for Transit VXLAN: add the name, subnet mask and gateway IP address 
  • on DNS tab: add DNS details
  • on Network Ranges tab: add the IP range that is usable (do not forget that there are 2 IPs already used by ESG and DLR) 
  • New > Routed 
  • on General tab: use the external network profile created previously, and add some subnetting details:
    • subnet mask (could be /24 - 254 IPs) - the whole range given to vRA for its workloads
    • range subnet mask (can be /29 - 6 IPs ) - for each application/group of application deployed
  • on Network Ranges tab: press Generate Ranges
At least the following two network profiles will be displayed:




Create Reservation:
  • go to Infrastructure > Reservations 
  • New >  vSphere (vCenter)
  • on General tab: type in the Name of the reservation, tenant name (if multiple tenants exist), business group for which the reservation is created, priority (in case multiple reservations exist for the same business group)
  • on Resources tab: select the compute resource, put in a machine quota (if needed), select the size of the memory, select the datastores and the storage quota and select the Resource pool (if one has been defined in vCenter Server)
  • on Network tab: select network adapter (transit VXLAN), transport zone, DLR and network profile
  • finalize the task by pressing OK
And we are set: we have compute, storage and network resource available for consumption. In the next post we will create a service and see how we consume NSX on demand.