Using the OpenShift CLI

iPython notebooks make it easy to run and work with virtual experiments. However there is often a need to go deeper, especially for developers. In this case using the OpenShift CLI via a terminal is invaluable.

The basic requirements are:

By default, files created within Pods on a Kubernetes or OpenShift cluster are ephemeral - are lost once the Pod terminates. To persist files, you can mount a Persistent Volume Claim (PVC) to a path inside the Pod and write files to that location.

In ST4SD, any files generated by your virtual experiments are, by default, stored in the workflow-instances PVC within the namespace where ST4SD is deployed.

Transferring files into or out of your cluster is quite simple. Create a Pod and use kubectl cp to copy files to/from a path inside the Pod. If the Pod mounts a PVC, you can also use this method to access files stored in the volume.

Components of your virtual experiments that use the kubernetes backend run as Pods within your cluster. The ST4SD experiment orchestrator also runs as a Pod and mounts the Persistent Volume Claim (PVC) containing your experiment files at the path /tmp/workdir/.

In general, you can debug Pods using the following techniques:

In the following we use

To run from a terminal you first need to login to the OpenShift cluster you want to run on. You can use user/password e.g.

or copy the login command from the OpenShift console of the cluster you want to run in:

Running from a terminal involves two steps

Here is an example yaml for launching the sum-numbers toy workflow. You can use this as a template for your writing your own workflow yaml. This document gives a detailed description of all the available fields.

You can also try running the sum-numbers workflow via this yaml

This currently assumes that various objects in the target ST4SD instance you want to run on have been named following the conventions outlined in the ST4SD installation guide. We hope to remove this requirement shortly.

If the above works it will create a workflow object called sn - this is determined by the value of the name field in the workflow yaml.

This means someone already has run a workflow with the same name in the same instance of ST4SD. Execute

to remove the old version and try the apply command again.

Every kubernetes object has a name given by its name field. This name must be unique as it identifies a specific object. Here will refer to the name of a workflow as WORKFLOW_NAME

A kubernetes object can also have arbitrary labels associated with it. These are given as key-values under the metadata.labels section in a workflow’s yaml. For example in the sum-numbers.yaml file we have:

labels are useful for grouping of objects as they allow you to list all objects with particular label key-value pair.

All labels defined in the workflows yaml are propagated to the pods the workflow creates. This allows us to find all pods created by a workflow instance, for example, by asking for all pods who also have a label workflow:sn. In this document we call the value of the workflow field WORKFLOW_ID

From the example yaml above WORKFLOW_NAME and WORKFLOW_ID can be set independently. This is why we differentiate them. However by convention we assume they will be the same, as this makes life much easier!

Once you’ve started a workflow try the following commands

When you list the pods of a workflow instance you will see one that is also called WORKFLOW_NAME - this is the primary pod of the instance. Pods for other workflow components that use the Kubernetes backend are called flow-$STAGE-$COMPONENT_NAME-$UNIQUEID

Getting the logs of a task is straight-forward:

All components in a workflow write their output under a directory tree on the target cluster which you can login to and browse around with a terminal. The key piece of information you need here is the INSTANCE_NAME of the executing instance of the workflow, which is also the name of the root of the directory tree. To get this run:

This will give output like:

sum-numbers-2021-02-09T133047.732342.instance is the INSTANCE_DIR_NAME

Then you can do:

Your first step in checking if there is an issue is to run oc describe e.g. oc describe workflow WORKFLOW_NAME or oc describe pod WORKFLOW_NAME. This allows you to check for example:

If you want to debug deeper you can use

If you start a shell in the primary pod, via exec or debug you can run the following tools. First cd to the workflow instance directory at INSTANCE_NAME

When you use the RESTApi via notebook to start a workflow it automates the creation of a workflow yaml as described above. Hence, you can still use the command line to work with it.

However note the reverse is not fully true. If you launch from the terminal:

To work with the command line all you need is to REST UID returned when you submitted the workflow from the notebook. This will be the WORKFLOW_NAME and the WORKFLOW_ID of the workflow instance.

The number of Workflows and associated Job and Pod objects in a namespace can easily reach O(1000) and higher. This section describes how to examine, sort and delete these objects.

By default oc get wf only lists workflows name and experiment state in alphabetical order of Name. However this can easily be customized

You can create a column for any data in the workflow objects YAML using the appropriate key-path.

Similarly you can sort workflow by any data in the workflow objects YAML using the appropriate key-path.

Combining the above you can list workflows showing creation data, exist-status and state sorted by date

The following commands illustrate how you can delete workflow objects This also deletes all Job and Pod objects associated with the workflow

The oneliner below will not delete the Pod objects that contain the elaunch-primary container (i.e. the entrypoint Pod of the virtual experiment instance). The code just deletes all Job and Pod objects that elaunch-primary created to run the tasks of components. After running it, you will still be able to run oc debug $WORKFLOW_NAME.