Skip to main contentIBM ST4SD

Using graph relationships

Use this page to learn how to use define relationships between virtual experiment graphs. Readers should be familiar with the FlowIR specification of components.

What is a relationship?

A core concept in ST4SD is the virtual experiment. A virtual experiment defines its execution logic in the form of a computational graph. This enables ST4SD to apply graph operations on the graphs of virtual experiments.

For example the runtime support of ST4SD for surrogate models performs a wide range of graph operations on the graphs of virtual experiments. It extracts sub-graphs from one or more virtual experiments, modifies them. Finally, it uses the transformed sub-graphs to update the graphs of different virtual experiment and even generate novel virtual experiments.

Structure of a relationship

ST4SD currently features one type of relationship which we term the transformation relationship.

A transformation explains how to replace the $outputGraph nodes in a target graph with transform($inputGraph) nodes from a source graph. Where transform() produces a graph that produces the same outputs as $outputGraph.

To achieve this, the transformation relationship defines how to:

  1. Map each of $inputGraph to a parameter of $outputGraph (or a default value evaluated in the scope of target)
  2. Map each result of $outputGraph to a result of $inputGraph (or a default value evaluated in the scope of source).

Parameters of the inputGraph are variables and DataReferences that $inputGraph references which are not produced by nodes in the $inputGraph subset of the source graph. The results of the $outputGraph are DataReferences which leftover nodes in the target graph consume which point to $outputGraph nodes.

Notice that parameters and results can be either DataReferences, variables, or strings which contain references to 0 or more variables. This means that the schema enables you to replace a DataReference in one graph with a string containing 0 or more references to variables of the other graph. In some cases this is a valid transformation action, but this is not the general case.

To this end, when you submit/update a transformation to the runtime service, the service will run tests to ensure that transformation can produce a valid experiment. Here are the rules that parameter and result mappings must follow:

  1. It is a valid operation to substitute a variable in one of the graphs with the value of another variable, or a value that contains 0 or more references to variables in the other graph.
  2. It is a valid operation to substitute a DataReference A in one of the graphs with a DataReference B in the other graph provided that B is valid in the scope of the resulting experiment graph.
  3. It is an invalid operation to substitute a variable in one of the graphs which is used in a place other than the command-line arguments of components with a DataReference

The schema of the relationship transformation (in YAML) is:

relationship:
  identifier: name of relationship
  transform: # there is currently only 1 type of relationship
    inputGraph: # required
      components: # required
      - component identifiers to include
      # must set exactly 1 of identifier or source
      identifier: |
        the string identifier of an existing ParameterisedPackage
The runtime inspects the graphs and how they are used to infer obvious mappings between parameters (if inferParameters is true) and results (if inferResults is true).

The runtime validates the transformation relationships by testing that:

  1. All inputGraph parameters have a relationship to an outputGraph parameter
  2. All outputGraph results have a relationship to an inputGraph result

Pushing a relationship to the runtime

You can submit a relationship to the ST4SD runtime-service API using the method experiment.service.db.ExperimentRestAPI.api_relationship_push().

The schema above is in YAML format. If you are using python (i.e. an instance of experiment.service.db.ExperimentRestAPI) you should either create an equivalent python dictionary OR a python string that contains the yaml definition and then convert the string into a dictionary using "yaml.loads(the_string)".

Example: push a relationship

from __future__ import typing
import json
def relationship_push(api: experiment.service.db.ExperimentRestAPI):
    """Creates a Transformation relationship between 2 parameterised
    packages from the ST4SD global registry


    Arguments:
      api: An instance of ExperimentRestAPI that has connected to your
           private ST4SD registry

Automatically synthesize new parameterised packages

You can use a transformation relationship to automatically synthesize a novel virtual parameterised virtual experiment package like so:

Use a transformation relationship to synthesize a new parameterised package

A user can trigger the runtime to synthesize a new parameterised package by submitting a synthesis-from-transformation-relationship payload to the runtime service using the method experiment.service.db.ExperimentRestAPI.api_relationship_synthesize().

The structure (in YAML) of the payload is:

parameterisation:
  presets:  # optional
    # Fields defined here *cannot* be overridden by `executionOptions`.
    # All fields are optional
    variables: #A list of preset values for variables in the virtual experiment
    - name: $name of variable
      value: $variableValue
    data:
      - name: name of a file in the "data" directory
The "parameterisation" field is identical to the "parameterisation" field of parameterised packages. You can find more details about it in our [parameterised package documentation](/creating-a-parameterised-package#the-parameterisation-section).
The schema above is in YAML format. If you are using python (i.e. an instance of experiment.service.db.ExperimentRestAPI) you should either create an equivalent python dictionary OR a python string that contains the yaml definition and then convert the string into a dictionary using "yaml.loads(the_string)".

Side effects of synthesis step:

  1. The runtime service will create a novel parameterised package on the registry.
  2. The new parameterised package:
    1. will contain all components from the virtual experiment that contains outputGraph except for those in the outputGraph. The runtime system will substitute those for the components of the transformed inputGraph.
    2. will have the parameterisation options specified in the parameterisation field of the schema above.
    3. inherits the interface of the virtual experiment that contains the outputGraph

Example: use a transformation relationship to auto-generate a new parameterised package

from __future__ import typing
import json
def relationship_synthesize(api: experiment.service.db.ExperimentRestAPI):
    """Uses a transformation relationship to synthesize a new parameterised
    package.


    Arguments:
      api: An instance of ExperimentRestAPI that has connected to your
           private ST4SD registry
Edit this page on GitHub
Previous
ST4SD Services: OpenShift CLI
Next
ST4SD Services: Using Runtime Policies