Introduction

We have already described how we can use DCI to achieve Zero-Touch Provisioning (ZTP) deployments in GitOps ZTP with DCI blog post. However in that scenario, that assumption is that the spoke cluster is made up of baremetal server(s), not taking into account other, more lightweight alternatives, that can be more suitable for testing and troubleshooting.

That's the reason of writing this blog post, aiming to provide some advices and good practices to have in mind when deploying your ZTP pipeline with DCI but using virtual machines (VMs) instead of baremetal servers, mainly for the deployment of the spoke cluster, where we will see that there are some differences that you may consider for a smooth transition.

Why considering virtualized scenarios?

This is mainly oriented for testing ZTP deployments in labs where we don't have enough compute resources to allocate physical machines to run the full hub-spoke deployment, so relying in VMs would have the same result. An interesting use case for such deployments is the ability to test specific workflows in a CI pipeline, minimizing resource consumption, enabling the deployment of an entire infrastructure seamlessly on just a single server.

Differences between a virtualized and a baremetal scenario

There are some considerations you may have in mind if you want to move from a baremetal to a virtualized environment:

• The target VMs must have a BMC emulator available, such as sushy-tools or Fakefish.
• You need to identify the proper rootDeviceHints to be used to identify the disk VM.
• Network connectivity between physical and virtual servers is required.
• The boot entries on the virtual machine may have an impact on the re-deployment of the lab.

Let's check each requirement one by one.

BMC access

Typically, a physical server is associated with a physical BMC IP address, which, when combined with the appropriate credentials, can be used to perform out-of-band operations. This could be an example of the bmcAddress defined in a SiteConfig manifest, for the case of a Dell server, using the Redfish implementation for Dell Server's BMCs:

    bmcAddress: "idrac-virtualmedia://192.168.16.160/redfish/v1/Systems/System.Embedded.1"

However, this approach cannot be directly applied to virtual machines created with KVM, as Redfish is not implemented on them. For this to work, we need to use Redfish development tools, such as sushy-tools, in the server where the VM is deployed.

Sushy-tools provides a Redfish endpoint (<host-ip>:<host-port>) for each hosted VM, which, when combined with the VM's UUID, can be used to construct the full path to the system.

It could be possible to have dedicated <host-ip>:<host-port> pairs for each hosted VM but, in our scenario, the <host-ip>:<host-port> pair is common to all the virtual BMCs in the same hypervisor, so we rely on the UUID to identify the specific VM BMC to address.

Consequently, having the following output from virsh dumpxml command to retrieve the VM's UUID:

    <domain type='kvm' id='729'>
        <name>sno</name>
        <uuid>96b951bc-4d37-4596-8d41-d6918a0c0423</uuid>
        ...

Then, we can build the bmcAddress in the following way:

    bmcAddress: "redfish-virtualmedia://192.168.16.19:8082/redfish/v1/Systems/96b951bc-4d37-4596-8d41-d6918a0c0423"

So, we use the IP address and port of the host server, together with the VM's UUID.

Identifying the disk of the VM

For the rootDeviceHints field in the SiteConfig, you mainly need to include the path to the proper disk to be used during spoke installation. This needs to be extracted, also from virsh dumpxml output. Just map the PCI address definition to the path of the target storage device as shown below:

    <devices>
        <disk type='file' device='disk'>
            <driver name='qemu' type='qcow2'/>
            <source file='/var/lib/libvirt/images/sno_main.qcow2' index='1'/>
            <backingStore/>
            <target dev='vda' bus='virtio'/>
            <alias name='virtio-disk0'/>
            <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
        </disk>
    </devices>

In this case, we can seet the rootDeviceHints in the following way:

    rootDeviceHints:
        deviceName: "/dev/disk/by-path/pci-0000:04:00.0"

So, mainly, we include the disk address at the end of the file path, using pci type, 0000 domain, 04 bus, 00 slot and 0 function.

Network connectivity

You need to ensure that the virtual machines are reachable from other components and servers of your internal network, so that these virtual machines can interact each other and with these other elements.

In our case, we rely on a bridged network, where we assign a specific MAC address to each virtual machine deployed in the environment, and then, we have some dnsmasq rules in the server jumphost that replies back to all DHCP queries. This ensures that the virtual machines can obtain a specific IP address that remains consistently reachable within the internal network.

For this, in the SiteConfig, we define something like this for the node:

    nodeNetwork:
        interfaces:
          - name: enp1s0
            macAddress: "52:54:00:00:0a:07"
        config:
          interfaces:
            - name: enp1s0
              type: ethernet
              state: up
              ipv4:
                enabled: true
                dhcp: true

You can see that the MAC address is hardcoded, and that we enable DHCP in the network interface. Then, you will need to instruct the server managing DHCP and DNS queries to provide a specific IP address given that MAC address. For example, this can be achieved with this dnsmasq configuration:

    $ cat /etc/dnsmasq.d/<file>.conf
    dhcp-host=52:54:00:00:0a:07,192.168.16.107,sno.server.example.lab

You can have more information regarding this bridged configuration in our previous blog post that reviews the deployment of distributed libvirt clusters. The approach here would be similar.

Boot entries

The boot entries added during the OCP installation may cause issues in cases where the VM needs to be redeployed. Issues like trying to boot from a previous installation, not booting from the proper device, etc. The recommendation is to clean up these boot entries. For this, we have some roles that already performs this for you, e.g. efi_boot_mgr role from ansible-collection-redhatci-cop. This can be called after deploying your spoke cluster with DCI by using the flag variable dci_delete_uefi_boot_entries, setting it to true.

Conclusion

We hope these instructions and advices are useful for your transition towards a virtualized ZTP scenario. The rest of interactions with DCI and related tools (such as dci-pipeline) are mainly the same, so the interface you use to deploy the cluster and have it ready for its usage is not changing.

In case of any doubts or questions, don't hesitate to reach Telco Partner CI team!