Creating Parameterised Experiment Packages

• What is a parameterised virtual experiment package?
• Structure of a parameterised virtual experiment package
• The Base section
• The Parameterisation section
• The Metadata section
• Adding a parameterised package to a registry
• Example

What is a parameterised virtual experiment package?

• How to access a virtual experiment
• What options to allow users to change
• What options have preset values
• Metadata about the package.

Structure of a parameterised virtual experiment package

1. The base packages (i.e. workflow definitions) that the virtual experiment consists of.
   • Where they are located, what version to get, and how to get them. Often there will be just one.
2. The parameterisation information:
   • Presets: options that users cannot change.
   • Execution options: options that users can change potential with some restrictions.
3. Metadata:
   • Various other information about the package e.g. description, license, maintainer and keywords.

definition = {
    "base": {
        # Required: Base package information ...
    }
    "metadata": {
            # Required: Various info about the package  ...
    },
    "parameterisation": {
      # Optional: What values are set and what can be changed ..
Copy to clipboard

The Base section

• Sources
• Specifying image registry dependencies

base:
  packages:
  - $PACKAGE_DEFINITION
Copy to clipboard

packages:
- name: # OPTIONAL - defaults to "main", is required
        # for multi-package experiments
  source:
     #REQUIRED: ONE package source type. See below for options
     $PACKAGE_SOURCETYPE: $PACKAGE_SOURCE_STRUCTURE
  config:
    # How to read the experiment from the given source e.g. manifest etc.
    # config is REQUIRED IF the base virtual experiment uses standard
Copy to clipboard

Sources

Git source

git:
  location:
    url: the http url of the repo
    # Must specify exactly ONE of branch, tag, and commit
    branch: name of branch
    tag: name of tag
    commit: git commit hash
  security:
    oauth:
Copy to clipboard

Datashim source

dataset:
  # No need for a security field because Datashim removes this requirement.
  location:
    dataset: the name of the dataset object
Copy to clipboard

S3 source

s3:
  location:
    region: region (optional)
    endpoint: S3 endpoint url
    bucket: bucket name
  security:
    valueFrom:
        # Must choose exactly ONE of secretS3KeyRef and valuseS3
        # "valuesS3" is automatically converted to "secretS3KeyRef" when you push the package
Copy to clipboard

Specifying image registry dependencies

dependencies:
  #An Optional dictionary of dependency types
  imageRegistries: # An optional list of image registries struct
  - serverUrl: the url to the image registry
    security:
      valueFrom:
        # Must select exactly 1 of secretKeyRef and usernamePassword
        # "usernamePassword" is automatically converted to a "secretKeyRef" when the package is pushed
        secretKeyRef:
Copy to clipboard

The Parameterisation section

• Parameterisation rules
• Presets
• Execution options

parameterisation:
  presets: ...
  executionOptions: ...
Copy to clipboard

Parameterisation rules

• variables (variables): Values for variables used in the experiment
• data-files (data) : Values for data files used by the experiment
• platform (platform): Value for the platform (named set of variables) to use
• runtime arguments (runtime) : elaunch.py command line arguments

1. The value provided by the user
2. The default value provided by the developer in the parameterised package if there is one
3. The first value in the array of options provided by the developer in the parameterised package if there is one
4. If none of the above exist the default value of the parameter in the base-package is used

Presets

parameterisation:
  presets:  # optional
    # Fields defined here *cannot* be overridden by `executionOptions`.
    # All fields are optional
    variables: #A list of preset values for variables in the virtual experiment
    - name: $name of variable
      value: $variableValue
    data:
      - name: name of a file in the "data" directory
Copy to clipboard

Execution options

  executionOptions: # optional
    # users may override values within constraints that workflow developers set
    variables:
    # Variables that the developer allows the user to override.
    # These CANNOT appear in presets.variables
    - name: $variable name
      # .value and .valueFrom are both optional and mutually exclusive
      # if neither fields exist then users can set variable to any value.
      #   at start, if users do not provide a value, the variable receives the
Copy to clipboard

The Metadata section

metadata:
  package: #All the maintainer metadata. Can decide exact structure at implementation time.
    name: the package name
    tags: # Optional
    - latest # On Push, auto insert latest if missing
    maintainer: email (optional)
    license: some string (optional)
    keywords: # optional
    - keyword 1
Copy to clipboard

Adding a parameterised package to a registry

• Pushing the package
• Parameterised Package Identifier
• Package Tags
• Registry metadata

Pushing the package

From a python dictionary

d = {
  "base": ...
}
Copy to clipboard

import mypackage
api.api_experiment_push(mypackage.d)
Copy to clipboard

From YAML

import yaml
with open('mypackage.yaml') as f:
    api.api_experiment_push(yaml.load(f))
Copy to clipboard

From JSON

import json
with open('mypackage.json') as f:
    api.api_experiment_push(json.load(f))
Copy to clipboard

Registry actions when a package is pushed

• Generates a unique Id for the entry see Parameterised Packaged Identifier
• Applies and updates tags - see Package Tags
• Stores any credentials as Kubernetes secrets and converts the relevant fields in the parameterised package to secretKeyRef and secretS3KeyRef types.
• Adds additional data to the parameterised package - see registry metadata

Parameterised Package Identifier

Package Tags

Example

• my-experiment
• my-experiment:latest
• my-experiment@sha256x16092ca4bb13955b1397bf38cfba45ef11c9933bf796454a81de4f8

api.api_experiment_start("my-experiment", payload={})
api.api_experiment_start("my-experiment:latest", payload={})
api.api_experiment_start("my-experiment@sha256x16092ca4bb13955b1397bf38cfba45ef11c9933bf796454a81de4f86", payload={})
Copy to clipboard

Package tag update rules

Registry metadata

registry: #All data added by runtime - developers cannot set anything under registry
    digest: $a string up to 63 characters #The identifier
    createdOn: UTC time that this digest was created, format is %Y-%m-%dT%H%M%S.%f%z
    timesExecuted: int - automatically increased every time a user launches this virtual experiment entry in the ST4SD deployment the registry is attached to
    tags: #A list of tags which point to this digest.
    - $TAG # This is a SUBSET of metadata.package.tags. It can be EMPTY if no tag points to this digest anymore
    interface: {} # ST4SD injects the Virtual experiment interface if it exists
    data: #The list of filenames under the `data` directory (just top-level files, NO directories)
    - name: $DATA_FILE_NAME
Copy to clipboard

Example

definition = {
    "base": {
        # We define the one or more base-packages (here just one)
        "packages": [{
            "source": {
                "git": {
                    "location": {
                        # This one lives on Git, under the "main" branch, we can also use
                        # "tag" and "commit"
Copy to clipboard