A Quick Look At Terraform Provider for Ansible

Terraform Provider for Ansible v1.0.0 has been release recently and while reading a couple of articles about it I actually wanted to see how it work end to end. 

We're going to look in this article at an use case where we provision cloud infrastructure with Terraform and then use Ansible to configure that infrastructure.

To be more specific, in our scenario we are looking at achieving the following

1. use Terraform to deploy an infrastructure in Google Cloud: VPC, VM instance with an external IP address and firewall rule to allow access to the instance 

2. automatically and transparently update Ansible inventory file 

3. automatically configure the newly provisioned VM instance with Ansible  

We use Terraform provider for Ansible and Ansible Terraform collection. From the collection we will be using the inventory plugin. Everything is run from a management machine installed with Terraform, Ansible and the Ansible collection (for installation please see the GitHub project linked above).

We will orchestrate everything from Terraform. We'll use Ansible provider to place the newly created VM instance to a specific Ansible group called "nginx_hosts" and execute Ansible commands to update the inventory and run the playbook that installs nginx. 

For simplicity we use a flat structure with a single Terraform configuration file, an Ansible inventory file and an Ansible playbook. 

We start by looking at the Ansible files.

inventory.yml contains only one line that references the collection inventory plugin:

plugin: cloud.terraform.terraform_provider

This way we make sure the inventory file is actually created dynamically based on the Terraform state file. 

nginx_install.yml is the playbook that installs nginx on the VM instance. It's a very simple playbook that checks the latest version is installed and that it is started. We will be using Ubuntu for our Linux distribution. 

---
- hosts: nginx_hosts
  tasks:
    - name: ensure nginx is at the latest version
      apt: name=nginx state=latest update_cache=true
      become: true
    - name: start nginx
      service:
          name: nginx
          state: started

Based on the code written so far, if we add any host to the group named "nginx_hosts" running the playbook will ensure latest version of nginx is installed. We have no knowledge of IP addresses or the hostnames of those hosts. We actually have no idea if there are any hosts in the group. 

The Ansible hosts that we want to configure are created using Terraform. For simplicity there is only one flat file - main.tf file. We start by defining the Ansible provider.

terraform {
  required_providers {
    ansible = {
      source  = "ansible/ansible"
      version = "1.0.0"
    }
  }
}

Next we define the variables. We are using Google Cloud provider and we need some variables to configure it and deploy the resources. We are using a user_id to generate unique resource name for each deployment. We add GCP provider variables (region, AZ, project) and variables for the network.

variable "user_id" {
  type = string
  description = "unique id used to create resources"  
  default = "tfansible001"
}

variable "gcp_region" {
  type = string
  description = "Google Cloud region where to deploy the resources"  
  default = "europe-west4"
}

variable "gcp_zone" {
  type = string
  description = "Google Cloud availability zone where to deploy resources"  
  default = "europe-west4-a"
}

variable "gcp_project" {
  type = string
  description = "Google Cloud project name where to deploy resources" 
  default = "your-project"
}

variable "networks" {
  description = "list of VPC names and subnets"
  type        = map(any)
  default = {
    web = "192.168.0.1/24"
  }
}

variable "fwl_allowed_tcp_ports" {
  type        = map(any)
  description = "list of firewall ports to open for each VPC"
  default = {
    web = ["22", "80", "443"]
  }
}

We need also variables for Ansible provider resources: ansible user that can connect and configure the instance, the path the the ssh key file and the path to python executable. In case you use just want to test this, you can use your Google Cloud user. 

variable "ansible_user" {
  type = string
  description = "Ansible user used to connect to the instance"
  default = "ansible_user"
}

variable "ansible_ssh_key" {
  type = string
  description = "ssh key file to use for ansible_user"
  default = "path_to_ssh_key_for_ansible_user"
}

variable "ansible_python" {
  type = string
  description = "path to python executable"
  default = "/usr/bin/python3"
}

Then we configure the Google Cloud provider. Note that in Terraform it is not mandatory to define a provider with requried_provider block. Also note that for Ansible provider there is no configuration. 

provider "google" {
  region  = var.gcp_region
  zone    = var.gcp_zone
  project = var.gcp_project
}

Time to create the resources. We start with the VPC, subnet and firewall rules. The code iterates through the map object defined in variables section:

resource "google_compute_network" "main" {
  for_each                = var.networks
  name                    = "vpc-${each.key}-${var.user_id}"
  auto_create_subnetworks = "false"
}

resource "google_compute_subnetwork" "main" {
  for_each                 = var.networks
  name                     = "subnet-${each.key}-${var.user_id}"
  ip_cidr_range            = each.value
  network                  = google_compute_network.main[each.key].id
  private_ip_google_access = "true"
}

resource "google_compute_firewall" "allow" {
  for_each = var.fwl_allowed_tcp_ports
  name     = "allow-${each.key}"
  network  = google_compute_network.main[each.key].name

  allow {
    protocol = "tcp"
    ports    = each.value
  }

  source_ranges = ["0.0.0.0/0"]

  depends_on = [
    google_compute_network.main
  ]
}

Then we deploy the VM instance and we inject the ssh key using VM metadata. Again, ansible_user could be your Google user if you are using this for a quick test.

resource "google_compute_instance" "web" {
  name         = "web-vm-${var.user_id}"
  machine_type = "e2-medium"

  boot_disk {
    initialize_params {
      image = "projects/ubuntu-os-cloud/global/images/ubuntu-2210-kinetic-amd64-v20230125"
    }
  }

  network_interface {
    network    = google_compute_network.main["web"].self_link
    subnetwork = google_compute_subnetwork.main["web"].self_link
    access_config {}
  }

  metadata = {
    "ssh-keys" = <<EOT
      ansible_user:ssh-rsa AAAAB3NzaC1y...
     EOT
  }

}

So far we have the infrastructure deployed. We now need to configure the VM instance. We will configure a resource of type ansible_host. The resource will be used to dynamically update the Ansible inventory. 

resource "time_sleep" "wait_20_seconds" {
  depends_on      = [google_compute_instance.web]
  create_duration = "20s"
}

resource "ansible_host" "gcp_instance" {
  name   = google_compute_instance.web.network_interface.0.access_config.0.nat_ip
  groups = ["nginx_hosts"]
  variables = {
    ansible_user                 = "${var.ansible_user}",
    ansible_ssh_private_key_file = "${var.ansible_ssh_key}",
    ansible_python_interpreter   = "${var.ansible_python}"
  }

  depends_on = [time_sleep.wait_20_seconds]
}

We've added a sleep time to make sure the VM instance is powered on and services are running. Please note that we add the public IP of the VM instance, whatever that is, as the host name in Ansible. The host is added to "nginx_hosts" group. We also let Ansible know what user, ssh key and python version to use. 

Last thing to do is to update Ansible inventory and run the playbook. We will use terraform_data resources to execute Ansible command line.

resource "terraform_data" "ansible_inventory" {
  provisioner "local-exec" {
    command = "ansible-inventory -i inventory.yml --graph --vars"
  }
  depends_on = [ansible_host.gcp_instance]
}

resource "terraform_data" "ansible_playbook" {
  provisioner "local-exec" {
    command = "ansible-playbook -i inventory.yml nginx_install.yml"
  }
  depends_on = [terraform_data.ansible_inventory]
}

And that's it. Once you update the code above with your information and run terraform apply, it will automatically deploy a Google Cloud VM instance and configure it with Ansible. All transparent and dynamic, all driven from Terraform. 

In this article you've seen how to use Terraform to deploy a cloud VM instance and automatically and transparently configure it with Ansible.

Moving Backups to Hardened Linux Repositories

It's not enough to have a backup of your data. You need to make sure that you will be able to recover from that backup when the time comes. And one of the best ways to make sure you can do it, is to make protect your backups from being modified intentionally or unintentionally. 

In Veeam Backup & Replication, a hardened repository is using a Linux server to provide immutability for your backups. The feature was first released in version 11. Let's see what makes the hardened repository special, how it protects your backups from changes and how easy is to actually start using it 

Immutable file attribute

Linux file system allows setting special attributes to its files. One of these attributes is immutable attribute. As long as it is set on a file, that file cannot be modified by any user, not even root. More, root user is the only user that can actually set and unset the immutable attribute on a specific file. You can do it using lsattr and chattr commands in Linux as seen in the below screenshot:

Veeam hardened repo uses exactly the same mechanism of making backup files immutable.  

Isolate Linux processes 

To run a successful repository, Veeam needs several functionalities: to receive data from proxies, to open and close firewall ports, to set and unset immutability as per retention policy. In order to harden the repository, Veeam implements these functionalities as separate Linux processes.

The processes that sets and unsets immutable attribute on the backup files is called veeamimmureposvc and it needs to run with root privileges, as root user is the only user that can modify immutable attribute.

veeamtransport --run-service is the Data Mover service performing data receiving, processing and storing.  Because it is a service exposed on the network, it is running under a standard Linux user. In case of a  breach, the service will give access only to a standard user with limited privileges. The Linux user under which this service runs must not be allowed to elevate its privileges. 

A third process takes care of dynamically opening and closing  firewall ports: veeamtransport --run-environmentsvc and this one is also running with elevated privileges. 

The following screen shot shows the three main services that are part of a hardened repository. 

Single use credentials

Another layer of protection is added through the way the credentials are handled within the backup server.

To add the Linux repo to the backup server you need to specify Linux credentials. These credentials are only used during the initial configuration process and they are never stored in backup server's credential manager. Temporary privilege elevation may be needed during the repository configuration  for deployment and installation of Veeam processes. After the configuration process finishes, all elevated privileges must be revoked from the user. 

 

Additional repository features - fast clone

This one is not a security related feature, but it comes in as a great add on to the hardened repository.

In case you formatted your file system with XFS file system and you have a supported Linux distribution (see this user guide page for more details), Veeam will use fast clone to reduce used disk space on the repository and increase the speed of synthetic backups and transformations. Fast clone works by referencing existing data blocks on the repository instead of copying the data blocks between files. 

Using the hardened repository

For new backup jobs, just point them to your hardened repository. In case you have existing backups then you need to migrate those to your new repo. With v12 comes a new feature that allows to move any backup from an existing repository to another one. Simply select your backup, right click it and you will see that now you can "move backup"

Let's look at moving backups from a Windows NTFS repository to our hardened Linux repo. We start with an empty repository configured with a service account called veeambackup

The first backup chain is for an unencrypted backup job. The backup job is configured to use a standard Windows repository. There are 2 full restore points in the backup chain. Each restore point is 960 MB and the total size on disk is 1.87 GB. We use "move backup" to send the the backup chain to the hardened repository:

Once the move processes finished, the backup job has been updated to point to the new repository. Let's check what happened on the Linux hardened repo. 

Find the backup chain in our repo:

Check the immutability flag:

The restore points are set as immutable. The metadata file is not since this file is modified during each backup operation. Trying to delete any of the restore points will fail:
 

We can also check that XFS fast clone is working by looking at the used space on the repo which is less that the sum of the 2 full backups:

In this post we've looked at the features of hardened repository and how they work. To implement a hardened repository in your environment follow the steps in the user guide  

Clear DNS Cache on VCSA after ESXi IP Address Update

I've recently had to do some changes and modify the ESXi management IP address. Once the ESXi host has been put to maintenance mode and removed from vCenter Server inventory I've updated the DNS server records: A and PTR records. Checking DNS resolution works, I've tried to re-add servers to vCenter Server using their FQDN, but it errored with no route to host. This is because of the DNS client cache on VCSA. 

To solve it fast and not wait too long you need to ssh to VCSA appliance and run the following commands: 

systemctl restart dnsmasq

systemctl restart systemd-resolved

Once the services are restarted you can add again the ESXi hosts using the FQDN.

A Look at Veeam Backup & Replication v12 NAS Backup - Creating file share backup job

In the previous post "A Look at Veeam Backup & Replication v12 NAS Backup - Initial Configuration" we discussed about NAS backup architecture and the initial configuration needed to setup infrastructure. It is time to continue with the creation of the backup  job and its options

Give the backup job a name

Select the shares to protect

If  you want to exclude files from being backed up, go to Advanced

Select the repository (we use here S3 compatible on premises minio)

Advanced settings for the backup job will let you specify how many versions to protect

Specify how to process ACLs for files and folders

Define compression and encryption 

If you want to plan periodic backup file maintenance, you can do it here

You can run scripts pre and post job execution, for example if you want to create a snapshot of the file share before the job runs

In case you would like to get warnings in the job about skipped files, configure here

Because we use S3 compatible, we are asked about helper appliance. We will skip it.

On archival page we selected another S3 compatible storage, this time in AWS. We will AWS S3 bucket to hold a copy of the primary storage and also to move there any files older than 3 months

Here you can filter out files that are sent to acrhive

Finally, schedule the backup job

And run it

A Look at Veeam Backup & Replication v12 NAS Backup - Initial Configuration

NAS backup was introduced back in v10. At the time I published a small article about how to backup a VSAN based file share using Veeam Backup & Replication (VBR) that you can read here. A lot of things changed since v10 in terms of features added to NAS backup. The new release will add support for direct to object storage backup and immutability just to name a couple.

The architecture stayed fundamentally the same since the release and its main components can be seen in the following diagram : 

File share - storage device where our protected data (files) resides. 

File proxy - backup proxy that runs the data mover service, reads data from the file share and sends it to the backup repository.

Cache proxy - dedicated component in NAS backup architecture to keep metadata about the protected sources. The cache holds temporary data and is used to reduce the load on the source during backups. 

Backup repository  - storage location for the backups 

Backup server - controls and coordinates jobs and resource allocation

In a large environment the components will be sized and distributed accordingly on different machines. The lab setup is a bit different: All Veeam components on the same machine. As source we have installed a True NAS VM  serving 2 NFS file shares 

To populate the file shares we've used a simple bash script similar to the code below. To make it faster, we've used multiple scripts that we ran in parallel. 

#!/bin/bash

for i in {1..100000}
do
    head -c 2K /dev/urandom > /mnt/filer/lots_of_files/2K_file_$i
done

The lab setup has limitations coming from ethernet connectivity between hosts, Intel NUC resources (SCSI interfaces of the local datastores, CPU and RAM) and resources allocated to the VMs. 

VBR Configuration

We will start preparing VBR for NAS backups. Add the required Veeam roles: NAS backup proxy and cache repo. Then we will connect the NFS shares to the server. For brevity of the post, steps in the wizard where no changes are done will be skipped.  

NAS Backup Proxy

In our scenario with all in one VBR deployment, the NAS backup proxy role will be installed by default on the backup server.

In case you would add the proxy role(s) on different machines, then you would need to start the "Add proxy" wizard from "Backup Infrastructure"

Select "General purpose backup proxy" 

Then choose your server to which you want to assign NAS proxy role. If the server you want is not in the list, press Add new to start adding it as a managed server. We've only lowered the number of max concurrent tasks due to lab resources availability. Normally you keep default settings. 

Next follow the wizard's screens and install the proxy role. 

Add Cache Repository

Cache repo role should be as close as possible to proxy and source. It holds metadata in memory, hence having a fast disk is not mandatory. In lab we install it on backup server VM, but in a prod environment you should move it to another machine to avoid resource competition (or size you backup server to accommodate the cache repo). To create a cache repo, go to Backup Repositories and start Add Backup Repository wizard and follow the wizard. No special configuration is needed. 

Select direct attached storage

Select repo OS

Give the repo a name

Select the managed server where to install the cache repo and disk location for the repo (press Populate button). It needs only a few GB of free space. If the server you want is not in the list, press Add new to start adding it as a managed server. 

Choose the folder for the repo

Use the default settings until the end of the wizard. It will install the transport service, 

Add file share

We are using NFS shares served from a TrueNAS VM. We'll go to "Inventory" and start "Add File  Share"

Enter the NFS path to the share. Here you can also select advanced settings on how to process the file share - directly from the filer or using a snapshot of the share

Next select to use any available proxy or specific one. In case you have a distributed architecture with multiple sites, you would select here the proxies closer to you share. Also, the cache repo and how aggressive you want the backup job to be are configured on the same step.

Finally, install Veeam components and finish adding the share.

Backup repository 

From the point of view of a backup repository there is no difference in v12 between a NAS job and a VM image one. Any available repository can be used now including direct to object storage and hardened repository. Just pick your favorite.  

Now you are ready to create the backup job - in this post A Look at Veeam Backup & Replication v12 NAS Backup - Creating file share backup job

Five Reasons to Monitor Your Infrastructure with Veeam ONE

 Veeam ONE (VONE) is the monitoring tool for your backup environment and not only. You can use it as well to gain visibility into your virtual infrastructure. And when you think at the features it provides out of the box such as proactive alerting, monitoring, reporting, capacity planning, chargeback, intelligent automation and diagnostics you understand the value it brings to your environment. But what got me so excited about Veeam ONE? Well, it was this:

It is a screen shot from VONE monitoring my lab environment. It's not the fact that there are alarms that got my attention. It is actually what each of those alarms is describing.

Let's look closer. Veeam ONE is monitoring in this case Veeam Backup & Replication server from my personal lab. Upon a quick look at the screenshot you will notice there are 5 different issue displayed. Some are errors, others just warnings. In both cases, these would prove to be critical if ignored in a production environment.

1. Backup repository connection failure 

My scale out backup repository has a capacity tier in Google Cloud Storage. The S3 bucket is not accessible anymore. This alarm triggers by default when the repo is not accessible for more than 5 minutes. 

2. Backup job state 

This alarm is looking at the state of all backup jobs. The backup job has ended with a error because the vSphere tags where deleted from vCenter Server. So no more backups for those VMs. 

3. Suspicious incremental backup size

This is one of the out of the box alarms that can help you in case of a ransomware attacks. It looks at changes in size between incremental backups and it triggers to let you know you should further investigate what's happening. 

4. Job disabled

There are disabled backup jobs. This can be on purpose or by mistake. In any situation, as a backup admin you would like to know if there are any and which they are. The predefined time before the alarm is triggered is 12 hours. More, this alarm has remediation actions. You just need to press "Approve Action" enter a comment and Veeam ONE will enable back the job. How cool is that!

Once the action is executed successfully and the job enabled in VBR, the status changes to let you know

5. Immutability state

Seems that even if I am using S3 compatible storage, immutability flag has not been set. In today's cybersecurity context, this is one small configuration that should be applied to all your repositories. Keep your backups protected from any type of modification. 

Conclusion 

These are only 5 alarms out of hundreds in Veeam ONE that help you keep your IT infrastructure operating securely. The alarms are all coming out of the box, but you can customize them and create your own. As I stated above, any of the issues highlighted by the alarms will prove critical in case of a situation arising. So it's better to catch and solve them pro-actively. 

Managing vSphere VM Templates with Packer

Packer is an open source tool developed by HashiCorp that lets you create identical images using the same source. It helps in implementing and managing golden images across your organization. I will be using Packer in a vSphere environment only and not be using its multi platform support. The use case I am looking at is managing VM templates applying infrastructure as code concepts. 

The workflow I am implementing is using base VM templates made of basic OS installation, VMware tools and networking connectivity. These base templates do not need any management except for periodic updates/patches. The base VMs then are customized into project specific templates using Packer. The process installs any given project customization such as additional users, software packages, devices and creates a new template to be used as the source for prod deployment. Packer will not replace a configuration management tool, but it will reduce the time to deploy and configure the prod (or running) instances. It is faster to have a prepped template than to wait for packages to install on each of your instances during prod deployment. The diagram below exemplifies the intended process:  

In this workflow, Packer plays a crucial role allowing for fast and repeatable automation of the VM templates based on specific requirements. All credentials are kept in a dedicated secrets manager, called Vault. I will not enter into details about Vault, just keep it in mind as it is used to store any credentials used by Packer. A new set of templates results at the end of the customization process and these are used to run the prod instances.

Packer will also ensure that any changes to the base VM template are tracked and can be repeated in any other infrastructure while they are written in a human readable format. Let's look at a simple example where we modify a CentOS 7 base template. For our project we will use the the following folder structure: 

There are 3 files:

• variables.pkr.hcl - keeps all variable definitions
• tmpl-linux.auto.pkrvars.hcl - keeps the initialized input variables and it will be loaded during run; this allows to only change this file when moving to another environment
• tmpl-linux.pkr.hcl - main Packer file 

Packer  uses HashiCorp Configuration Language (HCL). Let's look at variables.pkr.hcl file contents:

variable "vcenter_server" {
  type        = string
  description = "FQDN or IP address of the vCenter Server instance"
}

variable "build_user" {
  type        = string
  description = "user name for build account"
}

locals {
    timestamp = regex_replace(timestamp(), "[- TZ:]", "") 
}

local "linux_user_pass" {
  expression = vault("/kv/data/linux_workshop", "${var.ssh_user}")
  sensitive  = true
}

local "build_user_pass" {
  expression = vault("/kv/data/build_user", "${var.build_user}")
  sensitive  = true
}

There are 2 types of variables - input variables and local variables. Input variables need to be initialized from a default value, command line, environment or variable files (we are using auto.pkvras.hcl file for this). Local variables cannot be overridden at run time and can viewed as some kind of constants. In the example above the real number of variables has been truncated to keep it readable. You can see the input variables such as "vcenter_server" and "build_user". There are also 2 local variables - "timestamp" which is calculated from a function and used in our case in the note field of the VM and "build_user_pass" which keeps the password for our build user and it takes this value from Vault secrets manager. The "build_user_pass" is marked as sensitive which will hide it from the output.

Next, let's look the variable initialization file tmpl-linux.auto.pkrvars.hcl

vcenter_server = "vcsa.mylab.local"
build_user = "build_user@vsphere.local"

We chose to initialize the variables from a separate file. In it, we just assign values to our input variables. If we need to modify any variable this is the only place where we make the changes which makes it easier to manage. Again, for ease of reading the file has been truncated.

Time to see what the tmpl-linux.pkr.hcl file contains. In the customization we'll apply to our template we are looking at two things:

• add a new disk to the target image
• install software packages in the target image

We'll look at each section in the packer fil. First we define the required plugins - in our case vsphere. You can make sure that a certain version is loaded. 

packer {
  required_version = ">= 1.8.5"
  required_plugins {
    vsphere = {
      version = ">= v1.1.1"
      source  = "github.com/hashicorp/vsphere"
    }
  }
}

Next we define the source block which has the configuration needed by the builder plugin (vsphere plugin loaded above).

source "vsphere-clone" "linux-vm-1" {
  
  # vcenter server connection
  vcenter_server      = "${var.vcenter_server}"
  insecure_connection = "true"
  username            = "${var.build_user}"
  password            = local.build_user_pass

  # virtual infrastructure where we build the templates
  datacenter          = "${var.datacenter}"
  host                = "${var.vsphere_host}"
  datastore           = "${var.datastore}"
  folder              = "Templates/${var.lab_name}"

  # source template name 
  template            = "${var.src_vm_template}"

  # build process connectivity 
  communicator        = "ssh"
  ssh_username        = "${var.ssh_user}"
  ssh_password        = local.linux_user_pass

  # target image name and VM notes  
  vm_name             = "tmpl-${var.lab_name}-${var.new_vm_template}"
  notes               = "build with packer \n version ${local.timestamp} "  

  # target image hardware changes 
  disk_controller_type = ["pvscsi"]
  storage {
      disk_size = var.extra_disk_size
      disk_thin_provisioned = true
      disk_controller_index = 0
  }

  convert_to_template = true
}

In the source we let the build plugin know how to connect to vCenter Server, what virtual infrastructure to use (datastores, hosts), what is the source template that we will use, how to connect to it and what is the configuration for the target template that we build. At the end we instruct the plugin to convert the newly created image to a VM template. Notice the communicator defined as "ssh". Communicators instruct Packer how to upload and execute scripts in the target image. It supports: none, ssh and winrm. Please mind some builders have their own communicators, such as Docker builder. 

With the current configuration we can actually define our build process. We've already accomplished half of our customization - adding the new disk is defined in the source block. In the build block we place the configuration that is needed by our build plugin. We will actually use a shell provisioner to install two packages - htop and tree. In my example, shell provisioner is sufficient to do the job, basically run in the target image "yum install". However I would recommend using a proper configuration management tool such as Ansible instead of directly running commands. 

build {
  sources = ["source.vsphere-clone.linux-vm-1"]

  provisioner "shell" {
    execute_command = "echo '${local.linux_user_pass}' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
    inline = ["yum install tree htop -y"]
  }

}

Notice execute_command - this is a customization of the command we want to run (yum) and we use it to send the sudo password. The password itself is take from the local variable which is initialized with the value from kept in Vault secrets manager (as defined in variables.pkr.hcl).

The only thing left to do is to validate your configuration and run the build process.

packer validate .

packer build .

Please note that variable files in this post have been truncated for ease of reading. If you intend to use this example, you would need to fill in the missing variables and initialize them according to your environment.