Certification tests for OpenShift containers and operators: how to run with DCI

Certification suites required for the operator certification
How to run certification tests with DCI
Debug test results using DCI UI
End-to-end certification of container images with DCI
End-to-end certification of operators with DCI

This post has some practical information about running Preflight certification suites with DCI. You will learn how to run the tests, debug using log files, submit the results for the certification, and publish containers and operators in the catalog. All this is an embedded functionality offered by DCI.

Prerequisites:

DCI is Red Hat Distributed CI, written in Ansible.
Preflight is a command-line interface for validating if OpenShift operator bundles and containers meet minimum requirements for Red Hat OpenShift Certification.
OSCAP-podman tool to scan containers for vulnerabilities. Containers could be published in the catalog only if they are vulnerability-free.

Certification suites required for the operator certification

To fully certify an operator, you must first certify all containers the operator uses and then certify the operator.

cert_steps Fig. 1. Certification steps.

Here is the entire process at a glance:

Certify the container:

a. Run Preflight check container certification suite on the operator image and ensure that they are entirely green.

b. Run OSCAP-podman check to ensure the container is vulnerability-free.

c. Create a "Container image" certification project at connect.redhat.com and push all test results there.

d. Provide official information about your company and software to be displayed in the catalog and press the "publish" button to finish the certification process.
Certify the operator:

a. Run Preflight check operator certification suite on the bundle image and ensure that they are fully green.

b. Create a certification project, "Operator Bundle Image", at connect.redhat.com, push Preflight test results into this project, and provide support and sales information for your operator.

c. Open a pull request in the repository certified-operators, providing the operator's manifests and metadata. Once the formatting checks are fully green, you can merge your pull request. This merge will trigger operator publication in the catalog.

That means you have three test suites and at least two certification projects ahead of you: a container certification project to certify an operator image and an operator certification project to certify a bundle image. The overall certification scenario could become quite complex if you work in a disconnected environment and have to handle the mirroring.

The good news is that DCI handles this complexity for you. All you need is to provide DCI with bundle and index images. DCI could automatically extract all containers, run required tests for containers and the operator and help you to debug by providing detailed logs out of DCI UI. Once everything is green, DCI could automatically create the certification projects in connect.redhat.com and push the test results there.

There are still some remaining manual steps: to provide official information about your software for the catalog and decide when to publish. To publish a container, press the button in the certification UI, and to publish an operator, close and then re-open an automatically created pull request in certified-operators repository.

How to run certification tests with DCI

Let’s run these test suites for the testpmd-operator, often used as an as-is forwarder for the traffic testing. It uses one container: testpmd-operator.

We assume that the OCP cluster is already up and running and the dci-openshift-app-agent is installed with RPM. Please follow the next three steps to execute the required certification suites.

Export kubeconfig:

$ export KUBECONFIG=/var/lib/dci-openshift-app-agent/kubeconfig

Modify settings.yml file in /etc/dci-openshift-app-agent/settings.yml by providing all the information about your certification projects. Let’s consider two standard scenarios here.

a. If you have a connected environment with a private external registry.

$ cat /etc/dci-openshift-app-agent/settings.yml
---
# Job name and tags to be displayed in DCI UI
dci_name: "Testpmd-Operator-Preflight"
dci_tags: ["debug", "testpmd-operator", "testpmd-container"]
dci_topic: "OCP-4.7"
# DCI component for every OCP version
# could be checked here: https://www.distributed-ci.io/topics
dci_component: ['8cef32d9-bb90-465f-9b42-8b058878780a']

# Optional; provide these credentials
# if your registry is private.
partner_creds: "/opt/pull-secrets/partner_config.json"

# List of operators to certify,
# you could provide many operators at once.
preflight_operators_to_certify:
  - bundle_image: "quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9"
    # Mandatory for the connected environments.
    index_image: "quay.io/rh-nfv-int/nfv-example-cnf-catalog:v0.2.9"
...

b. There is a disconnected environment with the self-signed local registry and operator images in the external private registry. In the case of a disconnected environment, DCI would handle all the mirroring and regenerate a catalog image.

$ cat /etc/dci-openshift-app-agent/settings.yml
---
# job name and tags to be displayed in DCI UI
dci_name: "Testpmd-Operator-Preflight"
dci_tags: ["debug", "testpmd-operator", "testpmd-container"]
dci_topic: "OCP-4.7"
# DCI component for every OCP version
# could be checked here: https://www.distributed-ci.io/topics
dci_component: ['8cef32d9-bb90-465f-9b42-8b058878780a']

# Mandatory for the disconnected environment,
# this registry is used for mirrored images
# and to store an index (catalog) image.
provisionhost_registry: registry.local.lab:4443

# Credentials for your private registries.
# You could have several private registries:
# local and another external, to store the operator.
# In this case, please provide all credentials here.
partner_creds: "/opt/pull-secrets/partner_config.json"

# Optional; provide it if your registry is self-signed.
preflight_custom_ca: "/var/lib/dci-openshift-agent/registry/certs/cert.ca"

# List of operators to certify,
# you could provide many operators at once.
preflight_operators_to_certify:
  # In disconnected environments, provide a digest (SHA), not a tag.
  - bundle_image: "quay.io/rh-nfv-int/testpmd-operator-bundle@sha256:5e28f883faacefa847104ebba1a1a22ee897b7576f0af6b8253c68b5c8f42815"
...

Run dci-openshift-app-agent:
```
$ dci-openshift-app-agent-ctl -s -- -v
```

Debug test results using DCI UI

Once the job is executed, the results are displayed in the DCI UI. Let’s go through an example job and figure out how to debug.

It’s convenient to start from the /tests tab to check the overall situation. Since there are three test suites to execute, you would typically see three JUnit files with the results: two for the container and one for the operator.

In our case, Preflight container tests are 100% green and ready to be submitted.

container_res Fig. 2. Check container results for testpmd-operator.

OSCAP-podman tests have some failures; usually, migrating to the latest UBI or another base image from the catalog is enough to fix them.

container_hc Fig. 3. oscap-podman (health-check) results for testpmd-operator.

There is more to fix about operator tests, and tips and suggestions are available right here.

operator_res Fig. 4. Check operator results for testpmd-operator.

Let’s follow the suggestion to check the preflight.log file in the /files tab of the job. The ValidateOperatorBundle test failed because of an invalid service account found in the bundle.

preflight_log Fig. 5. Detailed logs in preflight.log.

In case of tricky errors, more log files and configuration snapshots are available in the same tab to help.

log_files Fig. 6. There are more log files to check.

End-to-end certification of container images with DCI

Once both check container and OSCAP-podman suites are green, you might want to submit test results to connect.redhat.com and certify your container image by publishing it in the catalog.

The first step is to generate an access token that would be shared between all your projects, save it into a file and provide a path to this file in a variable pyxis_apikey_path.

pyxis_token Fig. 7. Generate Pyxis token.

It's time to get a certification project in connect.redhat.com. DCI provides two options: automatically create a certification project or reuse the existing one.

To automatically create a certification project for the container, add configuration create_container_project: true

$ cat /etc/dci-openshift-app-agent/settings.yml
---
# -- snip --

# Optional; provide it when you need to submit test results.
# This token is shared between all your projects.
# To generate it: connect.redhat.com -> Product certification ->
# Container API Keys -> Generate new key
pyxis_apikey_path: "/opt/cache/pyxis-apikey.txt"

# List of operators to certify,
# you could provide many operators at once.
preflight_operators_to_certify:
  - bundle_image: "quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9"
    # Mandatory for the connected environments.
    index_image: "quay.io/rh-nfv-int/nfv-example-cnf-catalog:v0.2.9"
    # Optional; provide it when you need to create
    # a new "Container Image project" and submit test results in it.
    create_container_project: true

Using this config, DCI creates a new project automatically, displays its ID in the logs, and then submits the results of Preflight tests into this project.

container_project Fig. 8. DCI displays an ID of an automatically created project.

The second option is to manually create a "Certify a Container Image" project at connect.redhat.com and provide its ID in DCI settings.yml using the variable pyxis_container_identifier.

id_container Fig. 9. ID for container image project.

$ cat /etc/dci-openshift-app-agent/settings.yml
---
# -- snip --

# Optional; provide it when you need to submit test results.
# This token is shared between all your projects.
# To generate it: connect.redhat.com -> Product certification ->
# Container API Keys -> Generate new key
pyxis_apikey_path: "/opt/cache/pyxis-apikey.txt"

# List of operators to certify,
# you could provide many operators at once.
preflight_operators_to_certify:
  - bundle_image: "quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9"
    # Mandatory for the connected environments.
    index_image: "quay.io/rh-nfv-int/nfv-example-cnf-catalog:v0.2.9"
    # Optional; provide it when you need to submit test results.
    # It is an id of your Container Image Project
    # https://connect.redhat.com/projects/my_nice_container_id
    pyxis_container_identifier: "my_nice_container_id"

In both cases, the DCI job would run the tests and submit their results directly into the certification UI at connect.redhat.com. The "Images -> Certification test" column displays the Preflight tests run by DCI, and the "Health index" column is for OSCAP-podman vulnerability check results.

submitted_results Fig. 10. Submitted results in connect.redhat.com.

The certification is almost done! The last step is to provide a container description for the Red Hat Container catalog and marketing information about your certified product.

submitted_results Fig. 11. The last step - provide container description and marketing information.

Once all requested information is here, click the publish button to add the container to the catalog.

publish_container Fig. 12. Click the publish button to display the container in the catalog.

End-to-end certification of operators with DCI

Once all containers are certified and the check operator is green, you might want to submit the results in the certification UI. Unlike the container certification, there is an additional step to open a pull request in the certified-operators repository, and below, we use options create_pr: true and merge_pr: true to do it automatically with DCI. Do not forget to generate a github token before proceeding futher.

DCI provides two options: automatically create a certification project or reuse the existing one.

To create a certification project automatically, add create_operator_project: true for the operator.

$ cat /etc/dci-openshift-app-agent/settings.yml
---
# -- snip --

# Optional; provide it when you need to submit test results.
# This token is shared between all your projects.
# To generate it: connect.redhat.com -> Product certification ->
# Container API Keys -> Generate new key
pyxis_apikey_path: "/opt/cache/pyxis-apikey.txt"

# Optional; provide this token when using the create_pr option
github_token_path: "/opt/cache/dcicertbot-token.txt"

# List of operators to certify,
# you could provide many operators at once.
preflight_operators_to_certify:
  - bundle_image: "quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9"
    # Mandatory for the connected environments.
    index_image: "quay.io/rh-nfv-int/nfv-example-cnf-catalog:v0.2.9"
    # Optional; provide it when you need to create
    # a new "Operator Bundle Image" and submit test results in it.
    create_operator_project: true
    # Optional; use it to open the certification PR automatically
    # in the certified-operators repository
    create_pr: true
    # Optional; use it to merge the certification PR automatically
    # in the certified-operators repository
    merge_pr: true

Using this config, DCI creates a new project automatically, displays its ID in the logs, and then submits the results of Preflight tests into this project.

operator_project Fig. 13. DCI displays an ID of an automatically created project.

If you already have a certification project, provide an ID for operator project "Operator Bundle Image" as pyxis_operator_identifier.

$ cat /etc/dci-openshift-app-agent/settings.yml
---
# -- snip --

# Optional; provide it when you need to submit test results.
# This token is shared between all your projects.
# To generate it: connect.redhat.com -> Product certification ->
# Container API Keys -> Generate new key
pyxis_apikey_path: "/opt/cache/pyxis-apikey.txt"

# Optional; provide this token when using the create_pr option
github_token_path: "/opt/cache/dcicertbot-token.txt"

# List of operators to certify,
# you could provide many operators at once.
preflight_operators_to_certify:
  - bundle_image: "quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9"
    # Mandatory for the connected environments.
    index_image: "quay.io/rh-nfv-int/nfv-example-cnf-catalog:v0.2.9"
    # Optional; provide it when you need to submit test results.
    # It is an id of your Container Image Project
    # https://connect.redhat.com/projects/my_nice_container_id
    pyxis_container_identifier: "my_nice_container_id"
    # Optional; provide it when you need to submit test results
    # for the operator certification.
    # It is an id of your Operator Bundle Image
    # https://connect.redhat.com/projects/my_nice_container_id
    pyxis_operator_identifier: "my_nice_operator_id"
    # Optional; use it to open the certification PR automatically
    # in the certified-operators repository
    create_pr: true
    # Optional; use it to merge the certification PR automatically
    # in the certified-operators repository
    merge_pr: true

In both cases, the DCI job would run the tests and submit their results directly into the certification UI at connect.redhat.com. The "Test results -> Test result" column displays the Preflight tests run by DCI.

operator_test_results Fig. 14. Submitted results in connect.redhat.com

Here we have an error "Most recent pull request unsuccessful" because marketing information is missing in the overview tab. To fix that, add a repository description and product listing to be displayed in the catalog.

operator_test_results Fig. 15. Provide operator description and marketing information.

The last step is to take care of the automatically opened pull request. Similarly to automatically created certification projects, the link to the automatically created pull request is displayed in the DCI UI.

create_pr Fig. 16. Provide operator description and marketing information.

Some formatting tests displayed in the pull requests comments will be in the failed state because the marketing information was not provided when DCI created pull request. To re-run the formatting tests, close and re-open the pull request. That should fix the issue, make the tests 100% green, and automatically merge the pull request.

Fig. 17. Fully-green formatting tests.

This merge will finish the certification by triggering operator publication in the catalog.