Category Technical tips

If you want to deploy pods on a Kubernetes cluster that does not know the certificate of the registry containing the images (this is generally the case for labs with self-signed certificates that are not known by an authority), you risk to not be able to deploy your images, let’s see an example by deploying a Kuard image from my private Harbor registry:

(Warning, WordPress replaces the two dashes with one: cry:)

# kubectl run kuard –image=harbor.cpod-velocity.az-fkd.cloud-garage.net/library/kuard-amd64:blue
# kubectl get pods
NAME  READY STATUS   RESTARTS  AGE
kuard    0/1         ImagePullBackOff  0                     7s

# kubectl describe pods kuard
….
Type Reason Age From Message
—- —— —- —- ——-
Normal Scheduled 33s default-scheduler Successfully assigned default/kuard to test-md-0-5d6756b7fd-b9kwl
Normal Pulling 19s (x2 over 32s) kubelet Pulling image “harbor.cpod-velocity.az-fkd.cloud-garage.net/library/kuard-amd64:blue”
Warning Failed 19s (x2 over 32s) kubelet Failed to pull image “harbor.cpod-velocity.az-fkd.cloud-garage.net/library/kuard-amd64:blue”: rpc error: code = Unknown desc = failed to pull and unpack image “harbor.cpod-velocity.az-fkd.cloud-garage.net/library/kuard-amd64:blue”: failed to resolve reference “harbor.cpod-velocity.az-fkd.cloud-garage.net/library/kuard-amd64:blue”: failed to do request: Head “https://harbor.cpod-velocity.az-fkd.cloud-garage.net/v2/library/kuard-amd64/manifests/blue”: x509: certificate signed by unknown authority
Warning Failed 19s (x2 over 32s) kubelet Error: ErrImagePull
Normal BackOff 4s (x2 over 31s) kubelet Back-off pulling image “harbor.cpod-velocity.az-fkd.cloud-garage.net/library/kuard-amd64:blue”
Warning Failed 4s (x2 over 31s) kubelet Error: ImagePullBackOff

In this example, I tried from my Tanzu Kubernetes Grid (TKG) cluster to deploy the Kuard image which is located on my private Harbor registry hosted on my lab. This self-signed certificate is not recognized by an authority. For this to work anyway, each worker node in my cluster would need to know this certificate. I can copy it to each of them but the principle of TKG is to have a cluster whose life cycle can evolve easily and automatically, add worker nodes, remove them, replace them during the update , …. This would require each time a node is added to include the certificate.

Jesse Hu wrote a tkg-ytt-overlay-additional-ca-certs · GitHub procedure that I tested in TKG 1.4. It works great, it makes sense for those who use Kubernetes every day but may seem complicated to others. I will try to clarify it. This consists of obtaining the certificate, encoded in base64, of executing the commands to have the certificate taken into account by all existing and future Kubernetes nodes.

Encrypt the certificate with base64 and copy it

# base64 -w0 ca.crt (le résultat a volontairement été modifié)
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lRREthWmp6NmF4MitZS3RxS0YrUUNKekFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsb1lYSmliM0l0WTJFd0hoY05NakV3T0RJek1UVXdNREEwV2hjTk1qSXdPREl6TVRVdwpNREEwV2pBVU1SSXdFQVlEVlFRREV3bG9ZWEppYjNJdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRREl0MEhDUEF0ZWxrSVZhOXZtbjJnQWR4VXNaL3lTd2psNi95SzV0eTZ5OW1PbU5peDgKOHAvVFk2SG9MVHlJNUhtNytUYTJ4RGpvUUpmNllWMURsc3Y3d2E2R2pTUXE0WWxQWG1hUUNvVkp5eno5OVRvUgpDbW80VjhRUnJLbE5WL3NFbStrVXhseGFNZTZOMlA3UjB1MHVCV0Q5NW9Ra3RqWC8rS01uT0ErUlZoWkJ1dEFmCjJXSzhIMmRYeWo4bFV3Vk4rWWJqeW83dkdnNERNZlVHMWtDa0hYYURTMGcvWlhyNU1LRTNDWEo4YUVPZFhMNjMKb3BHTXVMQWp4WUZIdng0SnBMN0lNQ2VZMGFCcjcybUUzcy9SMENCMW1zTU5nYWhXTkhNZjhINktsUy9qUVlnUgpQMjV6WmYyOUZ0VFdnOWhZNVJZQngxalcyeXZERGJsMmFGNXBBZ01CQUFHallUQmZNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNESUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QWRCZ05WSFE0RUZnUVVVbTBvVUtiSXBHbTcxL3JzUlBReUJJbkZYYmt3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFE cVJBamwzWTNqVk5JK1JUbHBYdmwvdUYxbXNZUTNzTnFwVXR6eVNCVlNDWlpDMkRSSnpwYWZGCjhPdXN5QjBMTTNlS3VTd0t4STgrT0o5OTlhZkdOazRWTnpySVhOQURaZ1BxbnRFSWRucXNReGg4eFBuOVY0T2QKQUtsTVJycVI4R3g4ejdRM2EvN01uR0sra1l3VmorZ3BBNkFGUEJxSVJrU3Jscmo5b2dXVzBqWTFzL2tNU21ydgpaVEFZWTJqcFhBaGZrdzcrVDN4OHYwa0NRai9NREo5L3dNTnhxeVNGMEhzNXd6THVvbVJOM0VEME03eUNhWjg0CmJuOFZTN1VUSjBaWnhBRmx3TWxySlRYWmFpQmNOeDRNdm4wNXN4RG5KZktCdFloSkZwbGRwR3hLMDRUSmNXWm0KU0dDemZhc2FIK2M1NklNT0IvRllMdlJlelh4cE5mMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo =

 

Find the context of your TKG management cluster and the name of the TKG workload cluster on which you will copy the certificate:

# kubectl get-contexts

CURRENT                                     NAME CLUSTER      AUTHINFO                  NAMESPACE
….
my-cluster-admin@my-cluster                 my-cluster        my-cluster-admin
* test-admin@test                           test              test-admin
tkg-mgmt-vsphere-admin@tkg-mgmt-vsphere     tkg-mgmt-vsphere  tkg-mgmt-vsphere-admin
tool-admin@tool 

Edit the configuration of the controller and worker template to add the content of the certificate and the command to take this certificate into account. So each time a node is added, the certificate will be taken into account. The content of the file is to be added in the files: part and the command in the preKubeadmCommands: part as below (I have a configuration based on Photon OS, if you have another OS you must use another command. Copied / pasted does not work very well it is better to type the words again):

To edit the control plane template : kubectl edit KubeadmControlPlane test-control-plane –context tkg-mgmt-vsphere-admin@tkg-mgmt-vsphere

(the syntax just after  content shloud be like thist : – content: <base64 certificate>)

……

  files:
  – content: S0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lRREthWmp6NmF4MitZS3RxS0YrUUNKekFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsb1lYSmliM0l0WTJFd0hoY05NakV3T0RJek1UVXdNREEwV2hjTk1qSXdPREl6TVRVdwpNREEwV2pBVU1SSXdFQVlEVlFRREV3bG9ZWEppYjNJdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRREl0MEhDUEF0ZWxrSVZhOXZtbjJnQWR4VXNaL3lTd2psNi95SzV0eTZ5OW1PbU5peDgKOHAvVFk2SG9MVHlJNUhtNytUYTJ4RGpvUUpmNllWMURsc3Y3d2E2R2pTUXE0WWxQWG1hUUNvVkp5eno5OVRvUgpDbW80VjhRUnJLbE5WL3NFbStrVXhseGFNZTZOMlA3UjB1MHVCV0Q5NW9Ra3RqWC8rS01uT0ErUlZoWkJ1dEFmCjJXSzhIMmRYeWo4bFV3Vk4rWWJqeW83dkdnNERNZlVHMWtDa0hYYURTMGcvWlhyNU1LRTNDWEo4YUVPZFhMNjMKb3BHTXVMQWp4WUZIdng0SnBMN0lNQ2VZMGFCcjcybUUzcy9SMENCMW1zTU5nYWhXTkhNZjhINktsUy9qUVlnUgpQMjV6WmYyOUZ0VFdnOWhZNVJZQngxalcyeXZERGJsMmFGNXBBZ01CQUFHallUQmZNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNESUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QWRCZ05WSFE0RUZnUVVVbTBvVUtiSXBHbTcxL3JzUlBReUJJbkZYYmt3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFEcVJBamwzWTNqVk5JK1JUbHBYdmwvdUYxbXNZUTNzTnFwVXR6eVNCVlNDWlpDMkRSSnpwYWZGCjhPdXN5QjBMTTNlS3VTd0t4STgrT0o5OTlhZkdOazRWTnpySVhOQURaZ1BxbnRFSWRucXNReGg4eFBuOVY0T2QKQUtsTVJycVI4R3g4ejdRM2EvN01uR0sra1l3VmorZ3BBNkFGUEJxSVJrU3Jscmo5b2dXVzBqWTFzL2tNU21ydgpaVEFZWTJqcFhBaGZrdzcrVDN4OHYwa0NRai9NREo5L3dNTnhxeVNGMEhzNXd6THVvbVJOM0VEME03eUNhWjg0CmJuOFZTN1VUSjBaWnhBRmx3TWxySlRYWmFpQmNOeDRNdm4wNXN4RG5KZktCdFloSkZwbGRwR3hLMDRUSmNXWm0KU0dDemZhc2FIK2M1NklNT0IvRllMdlJlelh4cE5mMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    encoding: base64
    owner: root:root
    permissions: “0644”
    path: /etc/ssl/certs/tkg-custom-ca.pem

……

  preKubeadmCommands:

  – ‘! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh‘

……….

It is taken into account immediately and a new controller is deployed to replace the old one. (or several depending on the plan chosen at the time of creation). Wait while the new controller is deploying and replaces the old one. The same for the worker: kubectl edit KubeadmConfigTemplate test-md-0 –context tkg-mgmt-vsphere-admin@tkg-mgmt-vsphere. The consideration for workers is not immediate, you have to run the following command:

# kubectl patch machinedeployment test-md-0 –type merge -p “{\”spec\”:{\”template\”:{\”metadata\”:{\”annotations\”:{\”date\”:\”`date +’%s’`\”}}}}}” –context tkg-mgmt-vsphere-admin@tkg-mgmt
patched Wait a bit for the worker nodes to be replaced by new ones then we can retry deploying a new kuard image

# kubectl get node NAME STATUS ROLES AGE VERSION test-control-plane-gsmqz Ready control-plane,master 34m v1.21.2+vmware.1 test-md-0-698857566f-8pvt7 Ready <none> 118s v1.21.2+vmware.1 Now we can redeploy the Kuard image to check that it will run # kubectl run kuard –image=harbor.cpod-velocity.az-fkd.cloud-garage.net/library/kuard-amd64:blue pod/kuard created

# kg pods
NAME READY STATUS RESTARTS AGE
kuard 1/1 Running 0 17s

This procedure is valid for taking certificates into account if the cluster is already deployed. if it has not yet been created, it is preferable to have the certificate taken into account from the start using the procedure described in the installation documentation.

In order to simplify the authentication of Kubernetes clusters operating on different clouds, VMware has developed the Pinniped project accessible in Opensource. Pinniped has been integrated by default into the VMware Tanzu Kubernetes Grid (TKG) offering since version 1.3, replacing the Gangway. inniped allows authentication from OIDC or LDAP sources. In the case of LDAP source, Pinniped does not connect directly to LDAP but currently relies on the Dex component as Gangway already did.

When a user runs a Kubernetes command for the first time or after a certain period of inactivity, they are prompted to authenticate only once with their corporate credentials and can then consume multiple Kubernetes clusters.

I wanted to test this functionality in my lab with an LDAPS / Active Directory server running Windows 2019 and I quickly encountered the eternal problem of certificates not signed by a known authority. So I had to create a certificate that is recognized by the Active Directory server. Searching for hours on the internet, I ended up finding an article by Peter Mescalchin that worked on the first try: Enable LDAP over SSL (LDAPS) for Microsoft Active Directory servers. – bl.ocks.org.

However, when I wanted to use this procedure with Pinniped, it did not work because the SAN (Subject Alternative Name) information was not present in the certificate. By crossing several articles on the subject, I was able to adapt Peter Mescalchin’s solution so that the certificates integrate the SAN information. It gives this:

Creation of the Root certificate

Via OpenSSL (I used a Linux Ubuntu) create a private key (ca.key in my example) to be able to then create the root certificate (ca.crt in my example). The first command will ask you for a password and the second for your organization information.

$ openssl genrsa -aes256 -out ca.key 4096
$ openssl req -new -x509 -days 3650 -key ca.key -out ca.crt

 

Import the Root certificate on the AD server

From the AD server, type the command certlm or via Control Pannel, type computer certificates in the search bar:

Be careful to choose “Manage computer certificates” and not “Manage user certificates”

Import the previously generated ca.crt in the “Trusted Root Certification Authorities \ Certificates” section

 

Creation of the Client certificate

Still from the Active Directory server, create a file, in our example it has the name request.inf. In red, I have made changes from the original procedure to add the SAN information. Be careful to put the FQDN of the AD server in CN. The values ​​of _continue_ = “dns” and _continue_ = “ip-address” correspond to the SAN values, the other possible values ​​to reference the AD server.

[Version]
Signature=”$Windows NT$”

[NewRequest]
Subject = “CN=ad-server.cpod-velocity.az-fkd.cloud-garage.net“
KeySpec = 1
KeyLength = 2048
Exportable = TRUE
MachineKeySet = TRUE
SMIME = FALSE
PrivateKeyArchive = FALSE
UserProtected = FALSE
UseExistingKeySet = FALSE
ProviderName = “Microsoft RSA SChannel Cryptographic Provider”
ProviderType = 12
RequestType = PKCS10
KeyUsage = 0xa0

[EnhancedKeyUsageExtension]
OID = 1.3.6.1.5.5.7.3.1 ; Server Authentication

[Extensions]
; SANs can be included in the Extensions section by using the following text format. Note 2.5.29.17 is the OID for a SAN extension.
2.5.29.17 = “{text}”
_continue_ = “dns=ad-server&”
_continue_ = “dns=ad-server.cpod-velocity.az-fkd.cloud-garage.net&”
_continue_ = “dns=cloud-garage.net&”
_continue_ = “ipaddress=172.17.13.9&”

Generate the client.csr file with the command below:

c:\> certreq -new request.inf client.csr

From the Linux machine:

Create an extension file, in our example it has the name v3ext.txt. In red, I have made the changes from the initial procedure to add the SAN information under the heading v3_ca which will be referenced in the next order.

keyUsage=digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectKeyIdentifier=hash

# These extensions are added when ‘ca’ signs a request.
[ v3_ca ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = ad-server.cpod-velocity.az-fkd.cloud-garage.net
DNS.2 = ad-server
IP.1 = 172.17.13.9

Still from the Linux machine, create the client.crt certificate from the files generated in the previous steps ca.crt, ca.key, client.csr and v3ext.txt, in red which has been added compared to the command outcome of the initial procedure

$ openssl x509 -req -days 3650 -in client.csr -CA ca.crt -CAkey ca.key -extfile v3ext.txt -set_serial 01 -out client.crt -extensions v3_ca

 

To verify the presence of SAN information

$ openssl x509 -in client.crt -text
 …..
X509v3 extensions:
      X509v3 Subject Alternative Name:
          DNS:ad-server.cpod-velocity.az-fkd.cloud-garage.net, DNS:ad-server, IP Address:172.17.13.9

 

Import the Client certificate

From the AD server

C:\> certreq -accept client.crt

The certificate should appear in “Personal \ Certificates”

For the certificate to be taken into account, you must either restart the AD server or force LDAPS to load the certificate with the procedure below:

Still from the AD server, create a text file, in our example it is called ldap-renewservercert.txt with the content below (note the end of the file includes a line with a – (a dash):

dn:
changetype: modify
add: renewServerCertificate
renewServerCertificate: 1
–

Then type the command below:

c:\> ldifde -i -f ldap-renewservercert.txt

To test the taking into account, use the ldp.exe utility by selecting port 636 (or another if specific) and checking the SSL box.

Once all the procedure is done, you have to recover the ca.crt generated in the first step to give it to Pinniped. This can be done either when the TKG management cluster is created or subsequently.

If the management cluster has not yet been created:

$ tanzu management-cluster create –ui
(there are two dashes before the ui argument but WordPress only displays one)

In my test I chose vSphere as the platform, at the identity manager step it will be necessary to copy the certificate in the ROOT CA part

If the TKG management cluster has already been created and you want to update it:

From the Kubernetes context of the manager cluster, encrypt the Root certificate of the AD server with the base64 command and get the result:

$ base64 -w 0 ca.crt

Modify the certificate in the dex configmap by the result of the previous command:

$ kubectl edit configmap -n tanzu-system-auth dex
# Please edit the object below. Lines beginning with a ‘#’ will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  config.yaml: |
    issuer: https://172.17.13.10:30167
    frontend:
        theme: tkg
    web:
        https: 0.0.0.0:5556
        tlsCert: /etc/dex/tls/tls.crt
        tlsKey: /etc/dex/tls/tls.key
    expiry:
        signingKeys: 90m
        idTokens: 5m
    logger:
        level: info
        format: json
    staticClients:
        – id: pinniped-client-id
          name: pinniped-client-id
          redirectURIs:
            – https://172.17.13.10:31234/callback
          secret: 089db7e23b19cb628ba841b17cc32ea4
    connectors:
        – type: ldap
          id: ldap
          name: LDAP
          config:
            host: ad-server.cpod-velocity.az-fkd.cloud-garage.net:636
            insecureSkipVerify: false
bindDN: cn=administrator,cn=Users,dc=velocity,dc=local
            bindPW: $BIND_PW_ENV_VAR
            usernamePrompt: LDAP Username
            rootCAData:
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdQekNDQkNlZ0F3SUJBZ0lVU2tQd0JPazVYRVFLRlpydXdwZXBoeTlINndzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2dhNHhDek
KQmdOVkJBWVRBa1pTTVF3d0NnWURWUVFJREFOSlJFWXhEakFNQmdOVkJBY01CVkJoY21segpNUlF3RWdZRFZRUUtEQXRNYjJWcGJDMWtkUzFUUlRFTE1Ba0dBMVVFQ3d3Q1UwVXhPRE
EyQmdOVkJBTU1MMkZrCkxYTmxjblpsY2k1amNHOWtMWFpsYkc5amFYUjVMbUY2TFdaclpDNWpiRzkxWkMxbllYSmhaMlV1Ym1WME1TUXcKSWdZSktvWklodmNOQVFrQkZoVm1ZbVZ1Y
21WcVpHRnNRSFp0ZDJGeVpTNWpiMjB3SGhjTk1qRXdOekEzTURjeQpNekl5V2hjTk16RXdOekExTURjeU16SXlXakNCcmpFTE1Ba0dBMVVFQmhNQ1JsSXhEREFLQmdOVkJBZ01BMGxFClJqRU9NQXdHQTFVRUJ3d0ZVR0Z5YVhNeEZEQVNCZ05WQkFvTUMweHZaV2xzTFdSMUxWTkZNUXN3Q1FZRFZRUUwKREFKVFJURTRNRFlHQTFVRUF3d3ZZV1F0YzJWeWRtVnlMbU53YjJ
RdGRtVnNiMk5wZEhrdVlYb3RabXRrTG1OcwpiM1ZrTFdkaGNtRm5aUzV1WlhReEpEQWlCZ2txaGtpRzl3MEJDUUVXRldaaVpXNXlaV3BrWVd4QWRtMTNZWEpsCkxtTnZiVENDQWlJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dJUEFEQ0NBZ29DZ2dJQkFMaWx0WE81WkxNOFRzZ0YKMXFxRFFURi9xV1EzTGkvalU2ZFJqM1VMLys2YitRL0VUVjdIb2VLMi9hK09UdHlRbzY4c
k9ySTVJRjNNWlJqKwpzU0JuWC9SejczczYvVjArWXhJTFozSmNNenlWUzZtR1ZhNTZBTmFZRFRqSkErUzF5enZJczFpZXdMWDR2YlFzCnJRdFE2NVphb3NYbFlMSWpxdzZCY01TZUlX
NUlUWitlUVF3emlkN2t5ZFBYNDdTBSSm1vR…..1
            ….

Relaunch the dex pod in the tanzu-system-auth namespace to take the change into account.

 

Once the management cluster has the right certificate

From there, create a workload cluster:

$ tanzu cluster create my-cluster -f <fichier-environnement>

Import the administration Kubeconfig of the created workload cluster:

$ tanzu cluster kubeconfig get my-cluster –admin
(there are two dashes before the admin argument but WordPress only displays one)

Connect to the workload cluster with the admin context, as admin no need for an account:

$ kubectl use-context my-cluster-admin@my-cluster

Create a cluster role binding with the role that interests you (here cluster-admin) for the desired users, this will allow the user to use this cluster once authenticated:

$ kubectl create clusterrolebinding admin-fbenrejdal  –clusterrole cluster-admin –user fbe@velocity.local
(there are two dashes before the clusterrole and user arguments but WordPress only displays one)

Export the workload cluster kubeconfig, this is the kubeconfig that will need to be passed to users, it has no admin context and will require user authentication. The user will consume this cluster according to the rights defined in clusterrolebinding from the previous step:

$ tanzu cluster kubeconfig get my-cluster –export-file my-cluster-kubeconfig
(there are two dashes before the export argument but WordPress only displays one)

Issue a kubernetes command with the generated kubeconfig file, which will launch the browser for authentication:

$ kubectl get pods -A –kubeconfig my-cluster-kubeconfig
(there are two dashes before the kubeconfig argument but WordPress only displays one)

You should be redirected to a browser with a web page asking for your username and password:

Once entered, you will get the result of your last command:

The result of the previously command should be displayed:

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system antrea-agent-q9xpg 2/2 Running 0 7d15h
kube-system antrea-agent-qlmj8 2/2 Running 0 7d15h
kube-system antrea-controller-6bb57bd84-6cj58 1/1 Running 0 7d15h
kube-system coredns-68d49685bd-bjcps 1/1 Running 0 7d15h
kube-system coredns-68d49685bd-vttdw 1/1 Running 0 7d15h
kube-system etcd-my-cluster-control-plane-48n9f 1/1 Running 0 7d15h
kube-system kube-apiserver-my-cluster-control-plane-48n9f 1/1 Running 0 7d15h
kube-system kube-controller-manager-my-cluster-control-plane-48n9f 1/1 Running 0 7d15h
kube-system kube-proxy-dntrc 1/1 Running 0 7d15h
kube-system kube-proxy-k5m9g 1/1 Running 0 7d15h
kube-system kube-scheduler-my-cluster-control-plane-48n9f 1/1 Running 0 7d15h
kube-system kube-vip-my-cluster-control-plane-48n9f 1/1 Running 0 7d15h
kube-system metrics-server-66cb4fb659-xlprc 1/1 Running 0 7d15h
kube-system vsphere-cloud-controller-manager-vmfwl 1/1 Running 1 7d15h
kube-system vsphere-csi-controller-bd8b6cc8c-8ljl8 6/6 Running 0 7d15h
kube-system vsphere-csi-node-6xqf5 3/3 Running 0 7d15h
kube-system vsphere-csi-node-vmbmq 3/3 Running 0 7d15h
pinniped-concierge pinniped-concierge-dcd587f97-lk9n5 1/1 Running 0 7d15h
pinniped-concierge pinniped-concierge-dcd587f97-zrnb7 1/1 Running 0 7d15h
pinniped-concierge pinniped-concierge-kube-cert-agent-8a8e3e38 1/1 Running 0 7d15h
pinniped-supervisor pinniped-post-deploy-job-4ldt7 0/1 Completed 0 7d15h
pinniped-supervisor pinniped-post-deploy-job-m74gz 0/1 Error 0 7d15h
tkg-system kapp-controller-69c4d4bbb4-kwk5l 1/1 Running 0 7d15h

Cloud-Init seems to be the favorite customization tool for major OSs aiming to be installed on different cloud environments (AWS, Azure, GCP, vSphere, …), it is very powerful, heterogeneous but at the beginning, it is difficult to understand how it works.

Personalization information are stored in a file called user-data. This file is transmitted to Cloud-Init using a mechanism specific to each cloud. In our case, it is the VMtools that will transmit the user-data file, once received, Cloud-Init will execute it.

I wasted a tremendous amount of time finding the minimum steps to customize Ubuntu OS with Cloud-Init in a vSphere environment. I was looking for the possibility of customizing the OS when cloning from an OVF template.

Below is the procedure I use for an Ubuntu 20.04.2 LTS, freshly installed and after its first reboot. I kept the default values ​​except for the French keyboard and the installation of the OpenServer SSH option.

Cloud-Init must be told to retrieve the user-data customization file via the OVF parameters of the VM, there is a step to be done on the VM side and an OS side.

OS side:

• Delete the file that sets the Datasource to none instead of OVF:
  • sudo rm /etc/cloud/cloud.cfg.d/99-installer.cfg
• If you want the network configuration to be done, delete the file that disables it:
  • sudo rm /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
• If you are in DHCP, the VM will always retrieve the same IP because it will keep the same machine ID. To avoid this, you must reset the identity of the OS:
  • sudo truncate -s 0 /etc/machine-id

VM side:

• Get the contents of your cloud-init user-data file and encoded it with base64:
  • Base64 user-data.yaml
• From the vSphere client, when the VM is stopped, activate the properties of the OVF :
  • Select the VM => Configure => vApp Options => vApp Options are disabled … Edit => Click on “Enable vApp options”
  • In the OVF details tab => OVF environment transport => check the VMware Tools case
• From the vSphere client, when the VM is stopped, add the user-data field in the properties of the OVF:
  • => Select the VM => Configure => vApp Options => Properties => ADD enter the Label and the Key id with the user-data value.
• From the vSphere client, when the VM is stopped, add the value to the user-data field in the properties of the OVF:
  
  • Select the VM => Configure => vApp Options => Properties => SET VALUE, a popup appear and set it with the base64 key comming from the user-data file retrieved in the step early
    

Now from this VM, directly make a clone to make another VM or to make a template. If you want to change the user-data file, when deploying the VM, change only the base64 key to the new key

 

To create a virtual machine with vSphere with Tanzu via the kubectl command, there are steps to follow for the administrator and for the developer, they are very simple but that did not prevent me from wasting a little time on the customization side regarding the OS part.

I recommend this article to you to understand the interest of deploying VMs through Kubernetes: Deploy VM in and via Kubernetes. My colleague’s blog: Introducing Virtual Machine Provisioning, via Kubernetes with VM service | VMware is also very well detailed.

In the last part of this article, I will provide some details on the Content Library part and on the YAML part. But first, let’s review the parts to be done on the administrator side and the developer side.

 

 

Regarding the administrator

The first step is to download the VMs images that are different from those used for TKC (Tanzu Kubernetes Cluster aka Guest Cluster). The images are available in the VMware marketplace, at the time of writing this article there are 2 (Ubuntu and Centos), the current Ubuntu version does not allow the use of persistent volume (PVC) because it is based on a virtual hardware version 10 and at least version 12 is required, this problem will be soon corrected.

You have to go to the marketplace and do a search with the keyword “vm service”, this allows you to filter (a little) the compatible images => VMware Marketplace.

Then click on the desired image, connect with your MyVMware account.

You have two options, download it then upload it to a local content library

or retrieve the subscription url to create a content library that will synchronize with the one hosted by VMware.

Once the image is loaded or the link is filled in, you should have a content library like this:

Still from the vSphere interface, we must now create a namespace, grant the rights to the users so that they can connect to it, assign the VM class, the content library and the storage class, which should give this

The example above shows once the namespace has been created, how to assign a VM class, a content library, authorize the developers who can consume this namespace, which storage class to use and finally if necessary cap the resources, CPU, memory and storage.

That’s all there is to it on the infrastructure administrator side.

Regarding the developer

You need a YAML description for:

• The configmap which contains the customization of the VM
• The creation of the VM
• The Network service if you want to connect to it from an outside network (optional) PVC if you want to use persistent volumes (optional)

Via the Kubernetes command, the developer connects with his account to the Namespace provided, he will be able to list the classes of services that he can use as well as the images that he can deploy.

kubectl get virtualmachineclass
NAME                  CPU   MEMORY   AGE
best-effort-2xlarge   8     64Gi     22d
best-effort-4xlarge   16    128Gi    22d
best-effort-8xlarge   32    128Gi    22d
best-effort-large     4     16Gi     22d
best-effort-medium    2     8Gi      31d
best-effort-small     2     4Gi      31d
best-effort-xlarge    4     32Gi     22d
best-effort-xsmall    2     2Gi      22d
..
guaranteed-xsmall     2     2Gi      22d

 

kubectl get virtualmachineimage
NAME                                                         VERSION                           OSTYPE                FORMAT   AGE
centos-stream-8-vmservice-v1alpha1-1619529007339                                               centos8_64Guest       ovf      4h8m
ob-15957779-photon-3-k8s-v1.16.8—vmware.1-tkg.3.60d2ffd    v1.16.8+vmware.1-tkg.3.60d2ffd    vmwarePhoton64Guest   ovf      2d19h
ob-16466772-photon-3-k8s-v1.17.7—vmware.1-tkg.1.154236c    v1.17.7+vmware.1-tkg.1.154236c    vmwarePhoton64Guest   ovf      2d19h
ob-16545581-photon-3-k8s-v1.16.12—vmware.1-tkg.1.da7afe7   v1.16.12+vmware.1-tkg.1.da7afe7   vmwarePhoton64Guest   ovf      2d19h
……
ubuntu-20-1621373774638                                                                        ubuntu64Guest         ovf      4h8m

He can thus create his YAML descriptive files in order to define his resource requirements for his virtual machine (s) and if they wish, he can customize it or them in order to install his tools.

The configmap descriptive file, includes the customization of the VM. The 3 important fields to fill in for personalization are:

• The hostname which contains the OS hostname
• The public-keys, which contains the public key of a computer from which we will connect to the OS in ssh.
• The user-data part is, if you wish, the place where we put the contents of the Cloud Init configuration file, it will have to be encrypted with the base64 command

cat loeil-du-se-vm-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
    name: loeil-du-se-vm-configmap # The name of the ConfigMap, must be the same as in the VirtualMachine
    namespace: loeil-du-se
data:
  # OVF Keys values required by the VM at provision time
  hostname: loeil-du-se
  public-keys: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC4Cclh3rN/l70lBNlwQyK6ZtugxqT/7HeerHZKPSO0hcl5ZWLvz2+7QG5FqvYbkPP6EomvyDhE2MPnQ0kWaIrumVxYXAbVdpBBKKdTI3xJpewWB2syxgVOXP2ZOrw4cRLFv18rnESGHfsohedyaSB1qvubPWAqBFa+PSS4xh3zKalUknwc7Bs14fci8tEwEg8cpvNsqvrPZliJ6qTYFGqKuG6Ct+y449JNW6k6itTepgSYvUdJfjBTxk5tDzBdWz28km5N7lxgUB0rIWgSDl1XLCBrmm+H6bkHtD59MxAuxwLjih4tS4PzspcVjwWiJhd0HH7u2wbsPLCrrAX7am4EP40zphu9IR+fVxk+2jp7eD2uXPS6p9sDPEWHl6wGclI7pnfuoyvcn+CIwCtMweLuUw5MPj2eIIXcBhqUffeVAXVHrx8+e7+yHvqfyhqm2J9Ay3yt3zvAcXW0VqDxfvnfmv8sc9VNUW+8fUeyoo4b4uZRLLSf2DHM8= root@fbenrejdal-z01 # the public key to be able to do ssh without password from my laptop
  user-data: | # optional, enter the base64 cloud init encodded file, the result key could be a one line key or a multiple lines key. watch out the indentation, the line should be start under the “r” of user-data
     I2Nsb3VkLWNvbmZpZwojIFdBVENIT1VUIHRoZSBmaXJzdCBsaW5lIG11c3Qgc3RhcnQgd2l0aCAj
     Y2xvdWQtY29uZmlnCmdyb3VwczoKICAtIGRldm9wcwp1c2VyczoKICAtIGRlZmF1bHQgIyBDcmVh
     dGUgdGhlIGRlZmF1bHQgdXNlciBmb3IgdGhlIE9TCiAgLSBuYW1lOiBmYmUKICAgIHNzaC1hdXRo
     b3JpemVkLWtleXM6ICMgdGhlIHB1YmxpYyBrZXkgb2YgbXkgbGFwdG9wLCBpdCBjb3VsZCBhbHNv
     IGJlIGZpbGxlZCBpbiB0aGUgT1ZGIHByb3BlcnR5CiAgICAgIC0gc3NoLXJzYSBBQUFBQjNOemFD
     MXljMkVBQUFBREFRQUJBQUFCZ1FEQzRDY2xoM3JOL2w3MGxCTmx3UXlLNlp0dWd4cVRHLzdIZWVy
     SFpLUFNPMGhjbDVaV0x2ejIrN1FHNUZxdllia1BQNkVvbXZ5RGhFMk1QblEwa1dhSXJ1bVZ4WVhB
     YlZkcEJCS0tkVEkzeEpwZXdXL0Iyc3l4Z1ZPWFAyWk9ydzRjUkxGdjE4cm5FU0dIZisvc29oZWR5
     YVNCMXF2dWJQV0FxQkZhK1BTUzR4aDZELzN6S2FsVWtud2M3QnMxNGZjaTh0RXdFZzhjcHZOc3F2
     clBabGlKNnFUWUZHcUt1RzZDdCt5NDQ5Sk5XNms2aXRUZXBnU1l2VWRKZmpCVHhrNXREekJkV3oy
     OGttNU43bHhnVUIwcklXZ1NEbDFYTENCcm1tK0g2YmtIdEQ1OU14QXV4d0xqaWg0dFM0UHpzcGNW
     dGFydCBtb25nb2QK

The base64 is obtained as follows:

base64  loeil-du-se-vm-cloud-init.yaml

Its content in clear:

cat  loeil-du-se-vm-cloud-init.yaml
#cloud-config
# WATCHOUT the first line must start with #cloud-config
groups:
  – devops
users:
  – default # Create the default user for the OS
  – name: fbe
  ssh-authorized-keys: # the public key of my laptop, it could also be filled in the OVF property
    – ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC4Cclh3rN/l70lBNlwQyK6ZtugxqTG7HeerHZKPSO0hcl5ZWLvz2+7QG5FqvYbkPP6EomvyDhE2MPnQ0kWaIrumVxYXAbVdpBBKKdTI3xJpewWB2syxgVOXP2ZOrw4cRLFv18rnESGHf+sohedyaSB1qvubPWAqBFa+PSS4xh6D3zKalUknwc7Bs14fci8tEwEg8cpvNsqvrPZliJ6qTYFGqKuG6Ct+y449JNW6k6itTepgSYvUdJfjBTxk5tDzBdWz28km5N7lxgUB0rIWgSDl1XLCBrmm+H6bkHtD59MxAuxwLjih4tS4PzspcVjwWiJhd0HH7u2wbsPLCrrAX7am4EP40zphu9IR+fVxk+2jp7eD2uXPS6p9sDPEWHl6wGclI7pnfuoyvcn+CIwCtMweLuUw5MPj2eIIXcBhqUffeVAXVHrx8+e7+yHvqfyhqm2J9Ay3yt3zvAcXW0VqDxfvnfmv8sc9VNUW+8fUeyoo4b4uZRLLSf2DHM8= root@fbenrejdal-z01
      groups: sudo, devops
    shell: /bin/bash
    passwd: VMware1!
    sudo: [‘ALL=(ALL) NOPASSWD:ALL’] # the user fbe will not need to enter password when using sudo
  ssh_pwauth: true
  chpasswd:
  list: |
    fbe:VMware1! # in case you want to change password of users
    expire: false  # if you don’t want your password to expire
runcmd: # Example of runcmd to install MongoDB. Cloud Init has also APT keyword to do installation
  – echo “deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse” | tee /etc/apt/sources.list.d/mongodb-org-4.4.list
  – wget -qO – https://www.mongodb.org/static/pgp/server-4.4.asc | apt-key add –
  – echo “deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse” | tee /etc/apt/sources.list.d/mongodb-org-4.4.list
  – apt-get update
  – apt-get install -y mongodb-org
  – echo “mongodb-org hold” | dpkg –set-selections
  – echo “mongodb-org-server hold” | dpkg –set-selections
  – echo “mongodb-org-shell hold” | dpkg –set-selections
  – echo “mongodb-org-mongos hold” | dpkg –set-selections
  – echo “mongodb-org-tools hold” | dpkg –set-selections
  – sed -i ‘s/127.0.0.1/0.0.0.0/’ /etc/mongod.conf
  – ufw allow from any to any port 27017 proto tcp
  – sleep 2
  – systemctl start mongod

Very very important, the file must absolutely start with # cloud-config and nothing else. It’s still a classic Cloud Init file. If you’re not too familiar with Cloud Init, I’ve put some comments there to make it a bit more readable.

The VM description file

cat loeil-du-se-vm-deployment.yaml
apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachine
metadata:
  name: loeil-du-se-vm
  namespace: loeil-du-se
labels:
  vm: loeil-du-se-vm
spec:
  imageName: ubuntu-20-1621373774638 #the image must exist on the content library and must be listed with the command kubectl get virtualmachineimage
  className: best-effort-xsmall
  powerState: poweredOn
  storageClass: silver-storage-policy
  networkInterfaces:
  – networkType: nsx-t #must be nsx-t or vsphere-distributed depending on your install.
    # networkName: if vsphere-distributed you must specify the name of the network portgroup
  vmMetadata:
    configMapName: loeil-du-se-vm-configmap # The K8s configmap where personalization is stored
    transport: OvfEnv
#
#  when writing this article, the available image ubuntu (ubuntu-20-1621373774638) is not able to use volume because is using virtual hardware version 10
#  Instead you can use the centos image (centos-stream-8-vmservice-v1alpha1-1619529007339)
#  volumes: #when writing this article, the volume mount parameter is not use, the volume is seen in the guest but should be formated and mounted manualy
#    – name: loeil-du-se-volume
#      persistentVolumeClaim:
#      claimName: loeil-du-se-pvc
#      readOnly: false

Optional, the description file of the network service In my example, I created a service of type LoadBalancer to connect in ssh from an external network to that of the PODs.

Please note, the kind is not Service as usual but VirtualMachineService

cat loeil-du-se-vm-service.yaml
apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachineService
metadata:
  name: loeil-du-se-vm
spec:
  selector:
    vm: loeil-du-se-vm
  type: LoadBalancer
  ports:
    – name: ssh
      port: 22
      protocol: TCP
      targetPort: 22

Once the YAML files are created, all that remains is to have them taken into account by Kubernetes.

Kubectl create -f loeil-du-se-vm-configmap.yaml

Kubectl create -f loeil-du-se-vm-deployment.yaml

Kubectl create -f loeil-du-se-vm-service.yaml

To verify the creation of the VM:

Kubectl get vm

To know more about it:

Kubectl describe vm loeil-du-se-vm

It remains a classic VM, so it will benefit from HA and vMotion (via DRS or host maintenance mode). On the other hand, it is “Developer Managed”, that is to say that it cannot be managed via vCenter, you will not see the contents of the console for example.

One tip though, check which ESXi the VM is running on, then connect directly to the ESXi through a browser and there you will have access to the console.

To connect in ssh, if you have access via a loadbalancer like me, you can connect to it directly, otherwise you will have to go through a bounce POD (like busybox, alpine or other) and do an ssh with the IP address on the POD network. You can find it as follows:

kubectl get vm loeil-du-se-vm -o jsonpath='{.status.vmIp}’; echo
10.244.0.130

The ssh must be done via the user entered in the Cloud Init, I had set fbe, it looks like this:

kubectl get svc loeil-du-se-vm
NAME             TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
loeil-du-se-vm   LoadBalancer   10.96.1.108   172.20.18.71   22:32148/TCP   2d22h

ssh fbe@172.20.18.71
…
To run a command as administrator (user “root”), use “sudo <command>”.
See “man sudo_root” for details.
fbe@loeil-du-se:~$

If the ssh does not work, it is because the user was not taken into account by Cloud Init, try with root to obtain the default user, generally ubuntu for Ubuntu and cloud-user for CentOS:

ssh roo@172.20.18.71
Please login as the user “ubuntu” rather than the user “root”.

If you have the error below, it is because the laptop from which you are connecting does not have the public ssh key entered or there is an error in it, so you must check the key appearing in the configmap file:

fbe@172.20.18.71: Permission denied (publickey,password).

To debug Cloud Init, you must connect to the vm os via ssh or via the console and look at the log /var/log/cloud-init-output.log

There you go, feel free to ping me if you need more information.