FlowIR Specification

• Component
• DataReference
• Environments
• Variables
• Blueprints
• Platforms
• FlowIR Scopes
• FlowIR options/variable inheritance sequence
• Key-outputs
• Interface and Properties
• Application dependencies

Component

stage: integer greater or equal to 0 (optional - defaults to 0)
name: the name of the component (must be unique in the same stage)
command:
  executable: str
  arguments: str
  environment: (null, str)
references:
  - <reference:str>
workflowAttributes:
Copy to clipboard

Override

override:
  <platform name:str>:
    <component-field> #Any top-level field (with the sub-keys to be overriden) except name, stage  command and reference
Copy to clipboard

...
resourceRequest:
  numberThreads: 16
  threadsPerCore: 1
  memory: 100 MBi
override:
   bigmem:
     resourceRequest:
       memory: 1GBi
Copy to clipboard

Description of basic FlowIR component fields

• stage: integer greater or equal to 0 (optional - defaults to 0)
• name: the name of the component (must be unique in the same stage)
• command:
  • executable: path to executable. It can be absolute, relative to the instance directory by prefixing the path with bin/ or <application>/. It can also be just the name of a binary. If the path is not absolute the st4sd-runtime will look for the executable in the folders specified under $PATH.
  • arguments: arguments to binary
  • environment: Name of the environment to use. The definition will be searched in the top level environments field of FlowIR.
  • expandArguments: one of [“double-quote”, “none”] (default is “double-quote”). When set to “double-quote” ST4SD envelops commandline in double-quotes and perform bash expansion by feeding the resulting string to echo before using them to submit the task to the backend.
• references: Each string in this list is a string representation of a DataReference to either a reference to a file, a folder, or a component. References to components and files produced by a component (i.e. under the working directory of a component) indicate a data dependency which the st4sd-runtime respects when scheduling the tasks for components. There are several reference methods, but these are the most commonly used:
  • :output: the st4sd-runtime will replace <component name>:output references with the stdout output of the referenced component
  • :ref: the st4sd-runtime will replace <component name or file>:ref references with the absolute path to the component or file on the filesystem.
  • :copy: the st4sd-runtime will copy the file referenced by this DataReference into the working directory of the component which includes this reference. This DataReference method cannot be part of the command.arguments field.
• workflowAttributes:
  • replicate: If set to a positive number N the st4sd-runtime will replicate this component and its downstream tree N times (see aggregate below before you use this option).
  • aggregate: If this option is set to True and the component belongs in the downstream sub-tree of a replicate component the st4sd-runtime will stop replicating just before the aggregate component. Each reference of the aggregate component to the replicate component will be expanded to N references (one for each upstream replicated component).
• resourceRequest: provides hints to the backend about the resource requirements of this component tasks
  • numberProcesses: integer, defaults to 1
  • numberThreads: float, defaults to 1 (e.g. on kubernetes you can ask for half a thread)
  • ranksPerNode: integer, defaults to 1
  • threadsPerCore: integer, defaults to 1
  • memory: In Bytes or as Mi/Gi (e.g. 128Mi, 16Gi)
  • gpus: Only used by tasks that use the kubernetes or lsf backend
• resourceManager:
  • config:
    • backend: Which backend to use. Valid options are:
      • local (default option)
      • kubernetes
      • docker (also supports other docker-line runtimes like podman)
      • lsf
    • walltime: Maximum execution time of a single task for this component (in minutes). This option is only valid for kubernetes and lsf backends. The defaults is 60 (one hour).
  • kubernetes: Options to use when the kubernetes backend is selected for this component
    • image: which image to use
    • gracePeriod: Kubernetes waits gracePeriod seconds between asking a container to terminate and forcing it to terminate. This applies to tasks that use the Kubernetes backend and their execution time exceeds resourceManager.config.walltime minutes.
    • qos: One of “guaranteed” (default), “burstable”, “besteffort”. See Kubernetes documentation for the definition of Quality Of Service (QoS) classes.
  • docker: Options to use when the docker backend is selected for this component. Supports other docker-like runtimes via the elaunch.py parameter --dockerExecutableOverride
    • image: which image to use
    • imagePullPolicy: one of [“Always”, “Never”, “IfNotPresent”], default is “Always”
  • lsf: Options to use when the lsf backend is selected for this component
    • queue: Name of queue to submit jobs to.
    • resourceString: A LSF request string e.g. "rusage[ngpus_physical=4.00] select[(v100&&infiniband)]"
• variables: A key: value collection of variables; can either override those defined in platform or introduce new ones. In both cases the value specified here is visible to this component only. See FlowIR options/variable inheritance sequence for details on how scope layering/inheritance functions in FlowIR.

Defining components

components:
- stage: int
  name: str
  <component-core>
  override:
    <platform name:str>:
      <component-core>
Copy to clipboard

components:
# ...
- stage: 1
  name: PartialSum
  command:
    executable: "bin/sum.py"
    arguments: "ExtractRow:output"
  references: ["ExtractRow:output"]
Copy to clipboard

DataReference

Absolute representation of DataReference

stage<Index>.<producerName>:/<fileRef>:<method>
Copy to clipboard

• stage<Index>.: is the stage of the producer. This is only valid for DataReferences that point to components. Index should be an integer greater than 0.
• producerName: Either the name of a producer component, or the name of the directory in the root of the instance directory. The directories include all directories that ST4SD an experiment instance directory contains (e.g. input plus directories found in a standalone project such as data, conf, hooks, application-dependencies, etc).
• </fileRef>: Optional path, relative to the root directory of the producer. When omitted defaults to /.
• method: One of ref, output, copy, link. The method determines how ST4SD interprets the DataReference.
  • ref: The DataReference expands to the absolute path of the referenced file/directory
  • output: The DataReference expands to the contents of the referenced file. If the reference is to a component with the fileref ”/” then the DataReference is rewritten to point to the file containing the most recent stdout of the component.
  • copy: The DataReference does not expand to anything. If a component definition contains such a DataReference in its references field, then the runtime will copy the referenced path inside the root directory of the component’s task right before the execution of the task.
  • link: Similar to copy above. The difference is that instead of copying the referenced path, the runtime will create a link to the referenced path.

Relative representation of a DataReference

• If the relative DataReference is in the references field of a component then the Index is the same as the component.stage field.
• If the relative DataReference is in a key-output definition then the key-output should also contain the stages field. See key-output documentation for more details.

Environments

environments:
  <platform-name>:
    myDefinedEnvironment:
      ENV-VAR1: value/for/env-var1
      ENV-VAR2: value/for/env-var2
      DEFAULTS: ENV-VAR3:ENV-VAR4
Copy to clipboard

• ENV-VAR1 whose value is value/for/env-var1
• ENV-VAR2 whose value is value/for/env-var2
• ENV-VAR3 whose value is inherited from the environment variable ENV-VAR3 of the process running the runtime system
• ENV-VAR4 whose value is inherited from the environment variable ENV-VAR4 of the process running the runtime system

components:
  - name: myComponent
    command:
      executable: app.exe
      environment: myDefinedEnvironment
Copy to clipboard

environments:
  default:
    environment:
      ENV-VAR1: sensible/default/for/env-var1
      ENV-VAR2: sensible/default/for/env-var2
      ENV-VAR3: sensible/default/for/env-var3
Copy to clipboard

Variables

variables:
  <platform name:str>:
    Optional(global):
      <variable name:str>: <value: str, int, bool, float>
    Optional(stages):
      <stage index: int>:
        <variable name:str>: <value: str, int, bool, float>
Copy to clipboard

variables:
  default:
    global:
      numberOfPoints: 3
    stages:
      2:
        addToSum: 10

  artifactory:
Copy to clipboard

Using Variables

• values of fields
• values of other variables

variables:
  default:
    global:
      salutation: "hello"
      subject: "world"
      message: "%(salutation)s %(subject)s"

components:
- name: hello-message
Copy to clipboard

Variables can contain space separated arrays

• %(names)s[0]: This resolves to the 1st entry in the names array.
• %(names)s[%(index)s]: Indices may be variables too!

variables:
  default:
    global:
      # All variables are strings in FlowIR
      names: Ann Bob
      # Even those that look like a number
      population: 2

components:
Copy to clipboard

Blueprints

blueprint:
  <platform name:str>:
    Optional(global):
      <component options>
    Optional(stages):
      <stage index:int>:
        <component options>
Copy to clipboard

blueprint:
  default:
    global:
      command:
        environment: environment
  artifactory:
    global:
      resourceRequest:
        memory: 100Mi
Copy to clipboard

Platforms

platforms:
   - bigmem
   - nvidia-gpu
Copy to clipboard

default platform

artifactory platform

FlowIR Scopes

• global (i.e. visible to all components)
• visible to components within a specific stage
• visible to just one component

FlowIR options/variable inheritance sequence

1. Builtin st4sd-runtime blueprint
2. Default global blueprint
3. Default stage blueprint
4. Platform global blueprint
5. Platform stage blueprint
6. Component definition
7. Resolve interpreter option which may affect command.executable and command.arguments

1. If the environment is not set then the environment contains the default environment called “environment”. If the default environment is unset, then the default environment is the active shell environment.
2. If the name is the literal string “none” then the environment contains {}
3. Otherwise the st4sd-runtime uses the definition for the environment name from the selected platform. If there is no definition in the active platform the st4sd-runtime falls back to the default platform.
4. If an environment defines a DEFAULTS key then that key is expected to have the format VAR1:VAR2:VAR3.... Other options in the environment could reference the aforementioned vars using the $VAR and ${VAR} notation and these options will be resolved using their matching keys in the default environment.
   1. Any $VAR and ${VAR} references not matched by DEFAULTS keys will be resolved using the active shell(workflow launch environment).
   2. If a variable is defined in DEFAULTS but there is no value for it in the default environment then treat it as if it was never in the DEFAULTS option in the first place. This means that references to it will remain as is. The system that is executing the component’s task will resolve such environment variables just in time.
5. The runtime injects a couple of variables to the environment (INSTANCE_DIR, FLOW_EXPERIMENT_NAME, and FLOW_RUN_ID).

Default options

Key-outputs

Key-outputs for experiments written in DSL 2.0

output:
  - name: OptimisationResults
    data-in: <entry-instance/ExtractEnergies>/energies.csv:ref
    description: homo/lumo results
    type: csv
Copy to clipboard

name: the unique name of the key output
data-in: "an OutputReference to the output of an instance of a component template"

# Optional fields
description: "A human readable description of the file"
type: "e.g. csv, pdf, etc - this only used to label key-output"
Copy to clipboard

Key-outputs for experiments written in FlowIR

output:
  OptimisationResults:
    data-in: stage1.ExtractEnergies/energies.csv:ref
    description: homo/lumo results
    type: csv
Copy to clipboard

data-in: "a DataReference for FlowIR"

# Optional fields
description: "A human readable description of the file"
type: "e.g. csv, pdf, etc - this only used to label key-output"
stages:
- stage0
- stage1
Copy to clipboard

Interface and Properties

• The specification used to describe input systems it processes e.g. SMILEs for small molecules
• Instructions to extract the input systems from input data
• Instructions to extract the values of properties that the virtual experiment computes

Application dependencies

application-dependencies:
  default:
    - foo    # an application dependency called foo
  custom-platform:
    - bar
Copy to clipboard