Archive June 2021

Customize an Ubuntu VM with Cloud-Init on vSphere

Cloud-Init seems to be the favorite customization tool for major OSs aiming to be installed on different cloud environments (AWS, Azure, GCP, vSphere, …), it is very powerful, heterogeneous but at the beginning, it is difficult to understand how it works.

Personalization information are stored in a file called user-data. This file is transmitted to Cloud-Init using a mechanism specific to each cloud. In our case, it is the VMtools that will transmit the user-data file, once received, Cloud-Init will execute it.

I wasted a tremendous amount of time finding the minimum steps to customize Ubuntu OS with Cloud-Init in a vSphere environment. I was looking for the possibility of customizing the OS when cloning from an OVF template.

Below is the procedure I use for an Ubuntu 20.04.2 LTS, freshly installed and after its first reboot. I kept the default values ​​except for the French keyboard and the installation of the OpenServer SSH option.

Cloud-Init must be told to retrieve the user-data customization file via the OVF parameters of the VM, there is a step to be done on the VM side and an OS side.

OS side:

  • Delete the file that sets the Datasource to none instead of OVF:
    • sudo rm /etc/cloud/cloud.cfg.d/99-installer.cfg
  • If you want the network configuration to be done, delete the file that disables it:
    • sudo rm /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
  • If you are in DHCP, the VM will always retrieve the same IP because it will keep the same machine ID. To avoid this, you must reset the identity of the OS:
    • sudo truncate -s 0 /etc/machine-id

VM side:

  • Get the contents of your cloud-init user-data file and encoded it with base64:
    • Base64 user-data.yaml
  • From the vSphere client, when the VM is stopped, activate the properties of the OVF :
    • Select the VM => Configure => vApp Options => vApp Options are disabled … Edit => Click on “Enable vApp options”
    • In the OVF details tab => => check the VMware Tools case
  • From the vSphere client, when the VM is stopped, add the user-data field in the properties of the OVF:
    • => Select the VM => Configure => vApp Options => Properties => ADD enter the Label and the Key id with the user-data value.
  • From the vSphere client, when the VM is stopped, add the value to the user-data field in the properties of the OVF:

    • Select the VM => Configure => vApp Options => Properties => SET VALUE, a popup appear and set it with the base64 key comming from the user-data file retrieved in the step early

Now from this VM, directly make a clone to make another VM or to make a template. If you want to change the user-data file, when deploying the VM, change only the base64 key to the new key

Deploy VM in and via Kubernetes

Applications are often made up of Kubernetes PODs and VMs. The most common example that we find is, a database in the form of a VM and the rest of the application in the form of PODs. By reflex, rightly or wrongly, what requires data persistence is put in the form of VMs.

The vSphere with Tanzu platform is also a platform that allows simultaneous and native hosting of Kubernetes PODs and VMs.

Until now, VMs and PODs were deployed using different methods and connected to different networks, which could cause developers to delay development environment provisioning and the risk of connection failures. Indeed, developers had to ask the team that manages the infrastructure, the deployment of a VM with an expression of need.

To reduce the time impact and the risk of errors, the infrastructure teams have implemented automation tools via a ticketing system or via a self-service portal to give a certain autonomy. Deployment is much simpler but it is not yet sufficient because it involves the developer learning and using additional tools and retrieving the details of connections to the deployed VM. The self-service portal is not obsolete though, it has many other values ​​such as governance management, I hope I will have the opportunity to write an article on it for details.


Diagram showing a developer who clicks on his portal to deploy a VM that will be connected to a network.
This same developer uses the Kubernetes kubectl command to deploy their PODs. Kubernetes uses its own network.


Since vSphere 7U2a it is now possible to provision VMs in the same way as one deploys PODs, using the Kubernetes kubectl command. To be more precise, since the beginning of vSphere with Tanzu (originally it was called Project Pacific) it was possible to deploy Virtual Machines from Kubernetes, they were however reserved for internal Kubernetes use as for creation by Tanzu Kubernetes Cluster.

Now the developer can also deploy his own virtual machines, they will also be connected to the same network as the pods. The waste of time and the risk of error are thus eliminated. I did the test on my demo environment which is shared with my other colleagues, it takes less than 3 minutes to have a freshly installed MongoDB database from a completely virgin Linux Ubuntu.


Diagram showing a developer who uses both the Kubernetes kubectl command to deploy their PODs and VMs.
Everything will be connected to the same Kubernetes network.
What are the perimeters of each persona?
There are two, the resource provider and the consumer. The resource provider is the infrastructure administrator who will present the resources to be consumed and, if necessary, cap them. The consumer is the developer who will use these resources through Kubernetes to develop their application.
The person of the infrastructure with his usual tool (vSphere client), creates a namespace of resources, grants access rights to the developer, defines the classes of service (number of CPU, amount of RAM) to which the developer will have the right to use and the VM image library that he will be able to use.
The developer connects via his account to the Namespace provided and thus creates his YAML files in order to define his resource needs for his virtual machine (s) and if they wish, he can customize it or them in order to install his tools and the services he needs.
In summary, vSphere with Tanzu leaves the choice to the developer to have its application components developed and hosted on PODs or on VMs using the same tool, the same network and the same platform. This saves time for deployment, development and offer more agility.
If you want to lift the hood, I invite you to read this article: Steps for creating VM through Kubectl

Steps for creating VM through Kubectl


To create a virtual machine with vSphere with Tanzu via the kubectl command, there are steps to follow for the administrator and for the developer, they are very simple but that did not prevent me from wasting a little time on the customization side regarding the OS part.

I recommend this article to you to understand the interest of deploying VMs through Kubernetes: Deploy VM in and via Kubernetes. My colleague’s blog: Introducing Virtual Machine Provisioning, via Kubernetes with VM service | VMware is also very well detailed.

In the last part of this article, I will provide some details on the Content Library part and on the YAML part. But first, let’s review the parts to be done on the administrator side and the developer side.



Regarding the administrator

The first step is to download the VMs images that are different from those used for TKC (Tanzu Kubernetes Cluster aka Guest Cluster). The images are available in the VMware marketplace, at the time of writing this article there are 2 (Ubuntu and Centos), the current Ubuntu version does not allow the use of persistent volume (PVC) because it is based on a virtual hardware version 10 and at least version 12 is required, this problem will be soon corrected.

You have to go to the marketplace and do a search with the keyword “vm service”, this allows you to filter (a little) the compatible images => VMware Marketplace.

Then click on the desired image, connect with your MyVMware account.

You have two options, download it then upload it to a local content library

or retrieve the subscription url to create a content library that will synchronize with the one hosted by VMware.

Once the image is loaded or the link is filled in, you should have a content library like this:

Still from the vSphere interface, we must now create a namespace, grant the rights to the users so that they can connect to it, assign the VM class, the content library and the storage class, which should give this

The example above shows once the namespace has been created, how to assign a VM class, a content library, authorize the developers who can consume this namespace, which storage class to use and finally if necessary cap the resources, CPU, memory and storage.

That’s all there is to it on the infrastructure administrator side.

Regarding the developer

You need a YAML description for:

  • The configmap which contains the customization of the VM
  • The creation of the VM
  • The Network service if you want to connect to it from an outside network (optional) PVC if you want to use persistent volumes (optional)

Via the Kubernetes command, the developer connects with his account to the Namespace provided, he will be able to list the classes of services that he can use as well as the images that he can deploy.

kubectl get virtualmachineclass
NAME                  CPU   MEMORY   AGE
best-effort-2xlarge   8     64Gi     22d
best-effort-4xlarge   16    128Gi    22d
best-effort-8xlarge   32    128Gi    22d
best-effort-large     4     16Gi     22d
best-effort-medium    2     8Gi      31d
best-effort-small     2     4Gi      31d
best-effort-xlarge    4     32Gi     22d
best-effort-xsmall    2     2Gi      22d
guaranteed-xsmall     2     2Gi      22d


kubectl get virtualmachineimage
NAME                                                         VERSION                           OSTYPE                FORMAT   AGE

centos-stream-8-vmservice-v1alpha1-1619529007339                                               centos8_64Guest       ovf      4h8m
ob-15957779-photon-3-k8s-v1.16.8—vmware.1-tkg.3.60d2ffd    v1.16.8+vmware.1-tkg.3.60d2ffd    vmwarePhoton64Guest   ovf      2d19h
ob-16466772-photon-3-k8s-v1.17.7—vmware.1-tkg.1.154236c    v1.17.7+vmware.1-tkg.1.154236c    vmwarePhoton64Guest   ovf      2d19h
ob-16545581-photon-3-k8s-v1.16.12—vmware.1-tkg.1.da7afe7   v1.16.12+vmware.1-tkg.1.da7afe7   vmwarePhoton64Guest   ovf      2d19h
ubuntu-20-1621373774638                                                                        ubuntu64Guest         ovf      4h8m

He can thus create his YAML descriptive files in order to define his resource requirements for his virtual machine (s) and if they wish, he can customize it or them in order to install his tools.

The configmap descriptive file, includes the customization of the VM. The 3 important fields to fill in for personalization are:

  • The hostname which contains the OS hostname
  • The public-keys, which contains the public key of a computer from which we will connect to the OS in ssh.
  • The user-data part is, if you wish, the place where we put the contents of the Cloud Init configuration file, it will have to be encrypted with the base64 command

cat loeil-du-se-vm-configmap.yaml
apiVersion: v1
kind: ConfigMap
    name: loeil-du-se-vm-configmap # The name of the ConfigMap, must be the same as in the VirtualMachine
    namespace: loeil-du-se
  # OVF Keys values required by the VM at provision time
  hostname: loeil-du-se
  public-keys: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC4Cclh3rN/l70lBNlwQyK6ZtugxqT/7HeerHZKPSO0hcl5ZWLvz2+7QG5FqvYbkPP6EomvyDhE2MPnQ0kWaIrumVxYXAbVdpBBKKdTI3xJpewWB2syxgVOXP2ZOrw4cRLFv18rnESGHfsohedyaSB1qvubPWAqBFa+PSS4xh3zKalUknwc7Bs14fci8tEwEg8cpvNsqvrPZliJ6qTYFGqKuG6Ct+y449JNW6k6itTepgSYvUdJfjBTxk5tDzBdWz28km5N7lxgUB0rIWgSDl1XLCBrmm+H6bkHtD59MxAuxwLjih4tS4PzspcVjwWiJhd0HH7u2wbsPLCrrAX7am4EP40zphu9IR+fVxk+2jp7eD2uXPS6p9sDPEWHl6wGclI7pnfuoyvcn+CIwCtMweLuUw5MPj2eIIXcBhqUffeVAXVHrx8+e7+yHvqfyhqm2J9Ay3yt3zvAcXW0VqDxfvnfmv8sc9VNUW+8fUeyoo4b4uZRLLSf2DHM8= root@fbenrejdal-z01 # the public key to be able to do ssh without password from my laptop
  user-data: | # optional, enter the base64 cloud init encodded file, the result key could be a one line key or a multiple lines key. watch out the indentation, the line should be start under the “r” of user-data

The base64 is obtained as follows:

base64  loeil-du-se-vm-cloud-init.yaml

Its content in clear:

cat  loeil-du-se-vm-cloud-init.yaml

# WATCHOUT the first line must start with #cloud-config
  – devops
  – default # Create the default user for the OS
  – name: fbe
  ssh-authorized-keys: # the public key of my laptop, it could also be filled in the OVF property
    – ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC4Cclh3rN/l70lBNlwQyK6ZtugxqTG7HeerHZKPSO0hcl5ZWLvz2+7QG5FqvYbkPP6EomvyDhE2MPnQ0kWaIrumVxYXAbVdpBBKKdTI3xJpewWB2syxgVOXP2ZOrw4cRLFv18rnESGHf+sohedyaSB1qvubPWAqBFa+PSS4xh6D3zKalUknwc7Bs14fci8tEwEg8cpvNsqvrPZliJ6qTYFGqKuG6Ct+y449JNW6k6itTepgSYvUdJfjBTxk5tDzBdWz28km5N7lxgUB0rIWgSDl1XLCBrmm+H6bkHtD59MxAuxwLjih4tS4PzspcVjwWiJhd0HH7u2wbsPLCrrAX7am4EP40zphu9IR+fVxk+2jp7eD2uXPS6p9sDPEWHl6wGclI7pnfuoyvcn+CIwCtMweLuUw5MPj2eIIXcBhqUffeVAXVHrx8+e7+yHvqfyhqm2J9Ay3yt3zvAcXW0VqDxfvnfmv8sc9VNUW+8fUeyoo4b4uZRLLSf2DHM8= root@fbenrejdal-z01
      groups: sudo, devops
    shell: /bin/bash
    passwd: VMware1!
    sudo: [‘ALL=(ALL) NOPASSWD:ALL’] # the user fbe will not need to enter password when using sudo
  ssh_pwauth: true
  list: |
    fbe:VMware1! # in case you want to change password of users
    expire: false  # if you don’t want your password to expire
runcmd: # Example of runcmd to install MongoDB. Cloud Init has also APT keyword to do installation
  – echo “deb [ arch=amd64,arm64 ] focal/mongodb-org/4.4 multiverse” | tee /etc/apt/sources.list.d/mongodb-org-4.4.list
  – wget -qO – | apt-key add –
  – echo “deb [ arch=amd64,arm64 ] focal/mongodb-org/4.4 multiverse” | tee /etc/apt/sources.list.d/mongodb-org-4.4.list
  – apt-get update
  – apt-get install -y mongodb-org
  – echo “mongodb-org hold” | dpkg –set-selections
  – echo “mongodb-org-server hold” | dpkg –set-selections
  – echo “mongodb-org-shell hold” | dpkg –set-selections
  – echo “mongodb-org-mongos hold” | dpkg –set-selections
  – echo “mongodb-org-tools hold” | dpkg –set-selections
  – sed -i ‘s/’ /etc/mongod.conf
  – ufw allow from any to any port 27017 proto tcp
  – sleep 2
  – systemctl start mongod

Very very important, the file must absolutely start with # cloud-config and nothing else. It’s still a classic Cloud Init file. If you’re not too familiar with Cloud Init, I’ve put some comments there to make it a bit more readable.

The VM description file

cat loeil-du-se-vm-deployment.yaml

kind: VirtualMachine
  name: loeil-du-se-vm
  namespace: loeil-du-se
  vm: loeil-du-se-vm
  imageName: ubuntu-20-1621373774638 #the image must exist on the content library and must be listed with the command kubectl get virtualmachineimage
  className: best-effort-xsmall
  powerState: poweredOn
  storageClass: silver-storage-policy
  – networkType: nsx-t #must be nsx-t or vsphere-distributed depending on your install.
    # networkName: if vsphere-distributed you must specify the name of the network portgroup
    configMapName: loeil-du-se-vm-configmap # The K8s configmap where personalization is stored
    transport: OvfEnv
#  when writing this article, the available image ubuntu (ubuntu-20-1621373774638) is not able to use volume because is using virtual hardware version 10
#  Instead you can use the centos image (centos-stream-8-vmservice-v1alpha1-1619529007339)
#  volumes: #when writing this article, the volume mount parameter is not use, the volume is seen in the guest but should be formated and mounted manualy
#    – name: loeil-du-se-volume
#      persistentVolumeClaim:
#      claimName: loeil-du-se-pvc
#      readOnly: false

Optional, the description file of the network service In my example, I created a service of type LoadBalancer to connect in ssh from an external network to that of the PODs.

Please note, the kind is not Service as usual but VirtualMachineService

cat loeil-du-se-vm-service.yaml

kind: VirtualMachineService
  name: loeil-du-se-vm
    vm: loeil-du-se-vm
  type: LoadBalancer
    – name: ssh
      port: 22
      protocol: TCP
      targetPort: 22

Once the YAML files are created, all that remains is to have them taken into account by Kubernetes.

Kubectl create -f loeil-du-se-vm-configmap.yaml

Kubectl create -f loeil-du-se-vm-deployment.yaml

Kubectl create -f loeil-du-se-vm-service.yaml

To verify the creation of the VM:

Kubectl get vm

To know more about it:

Kubectl describe vm loeil-du-se-vm

It remains a classic VM, so it will benefit from HA and vMotion (via DRS or host maintenance mode). On the other hand, it is “Developer Managed”, that is to say that it cannot be managed via vCenter, you will not see the contents of the console for example.

One tip though, check which ESXi the VM is running on, then connect directly to the ESXi through a browser and there you will have access to the console.

To connect in ssh, if you have access via a loadbalancer like me, you can connect to it directly, otherwise you will have to go through a bounce POD (like busybox, alpine or other) and do an ssh with the IP address on the POD network. You can find it as follows:

kubectl get vm loeil-du-se-vm -o jsonpath='{.status.vmIp}’; echo

The ssh must be done via the user entered in the Cloud Init, I had set fbe, it looks like this:

kubectl get svc loeil-du-se-vm
NAME             TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
loeil-du-se-vm   LoadBalancer   22:32148/TCP   2d22h

ssh fbe@

To run a command as administrator (user “root”), use “sudo <command>”.
See “man sudo_root” for details.

If the ssh does not work, it is because the user was not taken into account by Cloud Init, try with root to obtain the default user, generally ubuntu for Ubuntu and cloud-user for CentOS:

ssh roo@
Please login as the user “ubuntu” rather than the user “root”.

If you have the error below, it is because the laptop from which you are connecting does not have the public ssh key entered or there is an error in it, so you must check the key appearing in the configmap file:

fbe@ Permission denied (publickey,password).

To debug Cloud Init, you must connect to the vm os via ssh or via the console and look at the log /var/log/cloud-init-output.log

There you go, feel free to ping me if you need more information.