Skip to main contentIBM ST4SD

Using containers

This page will teach you how to write and run a workflow that uses application containers via a toy-example. We advise users becoming familiar with the component specification before continuing.

Here, we are using the old workflow syntax FlowIR, if you need to understand the new workflow specification syntax check out the DSL 2.0 docs.

ST4SD core supports using application containers to execute component tasks. Application containers enable developers to have reproducible experiments that they can share with their community.

Using containers locally

Workflows can use docker-like runtimes (e.g. docker, podman) for components which execute containerized applications.

Here is a minimal example:

components:
- name: hello
  command:
    executable: python3
    arguments: -c "print('Hello world!')"
    expandArguments: none
  resourceManager:
    config:
      backend: docker

Check out the component specification for the full schema of component definition. By default, ST4SD will use docker to execute components whose resourceManager.config.backend is set to docker. Use the elaunch.py parameter --dockerExecutableOverride to use a different docker-like backend. For example, if the $PATH environment variable on your terminal is configured such that you can run the podman executable you can use it for your docker components by setting --dockerExecutableOverride=podman. If the executable is not in $PATH you can still use it, you just need to use the full path to it when you set the value of the --dockerExecutableOverride parameter.

Additionally, ST4SD mounts the instance directory in containers it executes. You may need to configure your container runtime system to have read/write access to the instance directory (e.g. by enabling read/write access to the parent directory of the workflow instance).

Using containers on Kubernetes

Workflows which are running on a Kubernetes environment (e.g. ST4SD Cloud on OpenShift) can use the kubernetes backend to run containers.

Here is a minimal example:

components:
- name: hello
  command:
    executable: python3
    arguments: -c "print('Hello world')"
    expandArguments: none
  resourceManager:
    config:
      backend: kubernetes

To run this, deploy ST4SD Cloud then use the Python client API to execute the experiment.

Using containers in multi-platform workflows

Often, workflows have support for multiple execution environments such as Cloud (e.g. Kubernetes/OpenShift), HPC, or even personal devices like laptops. ST4SD uses the concept of platforms to help workflow developers define how their workflows should execute under different execution environments. Typically developers use a variable as the value of the field resourceManager.config.backend. At execution time, a user configures the workflow to run using one of the available platforms by specifying the --platform parameter to elaunch.py. The components then use the appropriate backend for the desired platform.

For example, here is a single workflow that uses a backend based on the platform that the user decides at execution time:

variables:
 # The "default" platform configures the component to use the "docker" backend
 default:
   global:
     backend: docker
     # All platforms inherit variables of "default" and may override their value
     image: quay.io/fedora/python-311
 # The k8s platform has the component use the "kubernetes" backend
 k8s:

To run this on a laptop, place the above in a file called workflow.yaml.

Next, pip install "st4sd-runtime-core[develop]" and then execute:

elaunch.py -l40 --platform default workflow.yaml
Edit this page on GitHub