New Engine Plugin (--engine)

Important

There is an example of defining a engine plugin in the sample project repo for a fictional JSON-based simulator. It is all but functional, because there is no such JSON-based simulator available :-). This can be used in tandem with this guide to build your own engine plugin.

For the purposes of this tutorial, I will assume you are creating a new Engine Plugin matrix, and the code for that plugin lives in $HOME/git/plugins/engine/matrix.

Before beginning, see the Plugin Development Guide for a general overview of creating a new plugin.

If you are creating a new engine, you have two options.

1. Create a stand-alone engine, providing your own definitions for all of the necessary functions/classes below.

2. Derive from an existing engine by simply calling the "parent" engine's functions/calling in your derived definitions except when you need to actually extend them (e.g., to add support for a new HPC plugin which is a specialization of an existing one).

Before beginning, create the following filesystem structure in $HOME/git/plugins/engine/matrix.

• plugin.py - This file is required, and is where most of the bits for the plugin will go. You don't have to call it this; if you want to use a different name, see Schemas for options.

• cmdline.py This file is optional. If your new engine doesn't need any additional cmdline arguments, you can skip it.

• generators/engine.py - This file is required, and containing bits for generating experiments pertaining to this engine.

These files will be populated as you go through the rest of the tutorial.

Note

For all things that are optional, if you try to use a part of SIERRA requiring functionality you didn't define, you might get an obvious error, or you might get a crash later on, depending. Please help improve this aspect of SIERRA!

Creating The Cmdline Interface

1. Create additional cmdline arguments for the new engine by following Extending the SIERRA Cmdline For Your Plugin for engines.

2. Defining any additional configuration/argument checking beyond what is possible in argparse via cmdline_postparse_configure() in your plugin.py. If your new engine doesn't need any new arguments, you can skip this step.

   def cmdline_postparse_configure(argparse.Namespace) -> argparse.Namespace:
       """
       Additional configuration and/or validation of the passed cmdline
       arguments pertaining to this engine.  Validation should be performed
       with assert(), and the parsed argument object should be returned with
       any modifications/additions.
       """
   

Configuring The Experimental Environment

Define the ExpConfigurer class in plugin.py to configure Experiments in engine-specific ways. This class is required. It should conform to IExpConfigurer. It is used in stage 1 after experiment generation, in case configuration depends on the contents of the experiment. E.g.:

• Creating directories not created automatically by the simulator/project code.

• Copying files which Project or Engine code expects to be found next to the main input file for each Experiment or Experimental Run.

It is also used in stage {1,2} to tell SIERRA the type of execution parallelism that your Engine wants to use via the parallelism_paradigm() function.

Important

parallelism_paradigm() is one of the most important parts of your Engine definition, and will dramatically affect how experiments will be executed during stage 2. Engines can even select different paradigms depending on other configuration if they wish.

See Execution Model for more info.

Generating Experiments

In generators/engine.py, you may define the following functions:

for_all_exp()

This function is required. It is used to generate expdef changes common to all Experiment Runs in an Experiment for your engine.

Note

If your engine supports nested configuration files, this is the place to call flatten() using the selected expdef plugin. See code sample below for suggested implementation.

import pathlib

from sierra.core.experiment import definition, spec
from sierra.core import types
from sierra.core import plugin_manager as pm

def for_all_exp(spec: spec.ExperimentSpec,
                controller: str,
                cmdopts: types.Cmdopts,
                expdef_template_fpath: pathlib.Path) -> definition.BaseExpDef:
    """
    Create an experiment definition from the
    ``--expdef-template`` and generate expdef changes to input files
    that are common to all experiments on the engine. All projects
    using this engine should derive from this class for `their`
    project-specific changes for the engine.

    Arguments:

        spec: The spec for the experimental run.

        controller: The controller used for the experiment, as passed
                    via ``--controller``.

        expdef_template_fpath: The path to ``--expdef-template``.
    """
    # Optional, only needed if your engine supports nested
    # configuration files. Note that this snippet assumes that you have
    # already created the experiment definition object in a variable
    # called 'expdef'.
    expdef.flatten(["pathstring1", "pathstring2"])

for_single_exp_run()

This function is required. It is used to generate expdef changes for a single Experimental Run for your engine.

import pathlib

from sierra.core.experiment import definition
from sierra.core import types
from sierra.core.experiment import spec


def for_single_exp_run(
     exp_def: definition.BaseExpDef,
     run_num: int,
     run_output_path: pathlib.Path,
     launch_stem_path: pathlib.Path,
     random_seed: int,
     cmdopts: types.Cmdopts) -> definition.BaseExpDef:
     """
     Generate expdef changes unique to a experimental run within an
     experiment for the matrix engine.

     Arguments:
         exp_def: The experiment definition after ``--engine`` changes
         common to all experiments have been made.

         run_num: The run # in the experiment.

         run_output_path: Path to run output directory within
                          experiment root (i.e., a leaf).

         launch_stem_path: Path to launch file in the input directory
                           for the experimental run, sans extension
                           or other modifications that the engine
                           can impose.

         random_seed: The random seed for the run.

         cmdopts: Dictionary containing parsed cmdline options.
     """

arena_dims_from_criteria()

This function is optional; only needed if the dimensions are not specified on the cmdline for a scenario where you want to change the size of the arena from what it is in the template file, which can be useful if the batch criteria involves changing them; e.g., evaluating behavior with different arena shapes. See Arena Size for more details.

import typing as tp

from sierra.core import batch_criteria as bc

def arena_dims_from_criteria(criteria: bc.BatchCriteria) -> tp.List[utils.ArenaExtent]:
    """
    Arguments:

       criteria: The batch criteria built from cmdline specification
    """

Note

Neither of these functions is called directly in the SIERRA core; Project generators for experiments must currently call them directly. This behavior may change in the future, hence these functions are required.

1. In plugin.py, you may define the following functions:

   population_size_from_def()

   def population_size_from_def(exp_def: definition.BaseExpDef,
                                main_config: types.YAMLDict,
                                cmdopts: types.Cmdopts) -> int:
       """
       Given an experiment definition, main configuration, and cmdopts,
       get the # agents in the experiment.
       """
       pass
   

   population_size_from_pickle()

   def population_size_from_pickle(chgs: tp.Union[definition.AttrChangeSet,
                                   definition.ElementAddList],
                                   main_config: types.YAMLDict,
                                   cmdopts: types.Cmdopts) -> int:
       """
       Given unpickled experimental changes, main configuration, and
       cmdopts, get the # agents used in the pickled experiment.
       """
       pass
   

   so that SIERRA can extract the # agents used in a given experiment, which some engines need when defining their shell commands for executing an experiment (e.g., ROS). These functions are optional. HOWEVER, if neither is defined, then:

   • You MUST define arena_dims_from_criteria() in your engine plugin.

   • All Batch Criteria that you use must have the arena dimensions extractable when passed to arena_dims_from_criteria().

   See Arena Size for more info.

2. In plugin.py, you may define the following classes which are used in stages {1, 2} to generate the cmdline to execute Experiments and Experimental Runs. SIERRA essentially tries to mimic running experiments using a given engine as close as possible to running them on the cmdline directly; thus, configuring experiments for engine typically involves putting the needed shell commands into a "language" that SIERRA understands.

   BatchShellCmdsGenerator

   This class is optional. If it is defined, it should conform to IBatchShellCmdsGenerator.

   It is used in stage 2 to execute (not generate) shell commands per-batch previously written to a text file using GNU parallel (or some other engine of your choice). This includes sets of cmds for:

   • Pre-batch cmds executed prior to any experiment being executed.

   • Cmds to execute the batch experiment.

   • Post-batch cleanup cmds run after all experiments have been executed.

   This generator corresponds to per-batch parallelism; see Configuring The Experimental Environment for details.

   import typing as tp
   
   import implements
   
   from sierra.core.experiment import bindings
   from sierra.core import types, utils
   
   class BatchShellCmdsGenerator(bindings.IBatchRunShellCmdsGenerator):
       def __init__(self,
                    cmdopts: types.Cmdopts,
                    exp_num: int) -> None:
           pass
   
       def pre_batch_cmds(self) -> tp.List[types.ShellCmdSpec]:
           return []
   
       def post_batch_cmds(self) -> tp.List[types.ShellCmdSpec]:
           return []
   

   ExpShellCmdsGenerator

   This class is optional. If it is defined, it should conform to IExpShellCmdsGenerator.

   It is used in stage 2 to execute (not generate) shell commands per-experiment previously written to a text file using GNU parallel (or some other engine of your choice). This includes sets of cmds for:

   • Pre-experiment cmds executed prior to any experimental run being executed.

   • Post-experiment cleanup cmds before the next experiment is executed.

   Important

   The result of exec_exp_cmds() for engines plugins is ignored, because it doesn't make sense: execution environments execute experiments (DUH), so you don't need to define it.

   This generator corresponds to per-exp parallelism; see Configuring The Experimental Environment for details.

   import typing as tp
   
   import implements
   
   from sierra.core.experiment import bindings
   from sierra.core import types, utils
   
   class ExpShellCmdsGenerator(bindings.IExpRunShellCmdsGenerator):
       def __init__(self,
                    cmdopts: types.Cmdopts,
                    exp_num: int) -> None:
           pass
   
       def pre_exp_cmds(self) -> tp.List[types.ShellCmdSpec]:
           return []
   
       def post_exp_cmds(self) -> tp.List[types.ShellCmdSpec]:
           return []
   

   ExpRunShellCmdsGenerator

   This class is optional. If it is defined, it should conform to IExpRunShellCmdsGenerator.

   It is used in stage 1 to generate (not execute) the shell commands per-experimental run for this engine. These are sets of cmds which:

   • Need to be run before an experimental run.

   • Need to be run to actually execute an experimental run.

   • Need to executed post experimental run to cleanup before the next run is started. The generated cmds are written to a text file that GNU parallel (or some other engine of your choice) will run in stage 2.

   This generator corresponds to per-exp parallelism; see Configuring The Experimental Environment for details.

   import typing as tp
   import pathlib
   
   import implements
   
   from sierra.core.experiment import bindings
   from sierra.core.variables import batch_criteria as bc
   from sierra.core import types, utils
   
   class ExpRunShellCmdsGenerator(bindings.IExpRunShellCmdsGenerator):
       def __init__(self,
            cmdopts: types.Cmdopts,
            criteria: bc.BatchCriteria,
            exp_num: int,
            n_agents: tp.Optional[int]) -> None:
            pass
   
        def pre_run_cmds(self,
                         host: str,
                         input_fpath: pathlib.Path,
                         run_num: int) -> tp.List[types.ShellCmdSpec]:
            return []
   
        def exec_run_cmds(self,
                          host: str,
                          input_fpath: pathlib.Path,
                          run_num: int) -> tp.List[types.ShellCmdSpec]:
            return []
   
    def post_run_cmds(
               self, host: str, run_output_root: pathlib.Path
               ) -> tp.List[types.ShellCmdSpec]:
               return []
   

3. In plugin.py, you may define exec_env_check() to check the software environment (envvars, PATH, etc.) for this engine plugin prior to running anything in stage 2. Since stage 2 can be run in a different invocation than stage 1, this hook is provided so that the correct environment exists prior to executing anything. This function is optional.

   import os
   
   from sierra.core import types
   
   def exec_env_check(cmdopts: types.Cmdopts):
       """
       Check the software environment (envvars, PATH, etc.) for this engine
       plugin prior to running anything in stage 2.
       """
       assert os.environ("MYVAR") != None, "MYVAR must be defined!"
   

4. In plugin.py, you may define pre_exp_diagnostics(), which can be used to emit some useful information via logging at the start of stage 2 before starting execution of the Batch Experiment. This function is optional.

   import logging
   
   from sierra.core import types, batchroot
   
   def pre_exp_diagnostics(cmdopts: types.Cmdopts,
                           pathset: batchroot.PathSet,
                           logger: logging.Logger) -> None:
       """
       Log any INFO-level diagnostics to stdout before a given
       :term:`Experiment` is run. Useful to echo important execution
       environment configuration to the terminal as a sanity check.
       """
       logger.info("Starting batch experiment using MATRIX!")
   

5. In plugin.py, you may define expdef_flatten(), which can be used to flatten nested --expdef-template files before creating experiments, if supported by your chosen expdef plugin. This function is optional.

   def expdef_flatten(exp_def: definition.BaseExpDef) -> definition.BaseExpDef:
       """
       Given an experiment definition, perform engine-specific
       flattening of nested configuration files prior to scaffolding
       the batch experiment.
       """
       pass
   

Generating Products

1. In plugin.py, you may define exp_duration(), which can be used to retrieve the experiment setup information in later pipeline stages for providing nicer X-axis labels for graphs, for example. This function is optional.

   def expsetup_from_def(exp_def: definition.BaseExpDef) -> types.SimpleDict:
       """
       Given an experiment definition, compute the experiment setup
       information. Should contain keys:
   
       - ``duration`` - Duration in seconds.
   
       - ``n_ticks_per_sec`` - Ticks per second for controllers/sim.
       """
       pass
   

A Full Skeleton

cmdline.py

import typing as tp
import argparse

import sierra.core.cmdline as corecmd
from sierra.core import types


class Cmdline(corecmd.BaseCmdline):
    """
    Defines cmdline extensions to the core command line arguments
    defined in :class:`~sierra.core.cmdline.CoreCmdline` for the
    ``matrix`` engine. Any projects using this engine should
    derive their cmdlines from this class.

    Arguments:

        parents: A list of other parsers which are the parents of
                 this parser. This is used to inherit cmdline options
                 from the selected ``--execenv`` at runtime. If
                 None, then we are generating sphinx documentation
                 from cmdline options.

         stages: A list of pipeline stages to add cmdline arguments
                 for (1-5; -1 for multistage arguments). During
                 normal operation, this will be [-1, 1, 2, 3, 4, 5].

    """

    def init_stage1(self) -> None:
        super().init_stage1()

        # Experiment options
        experiment = self.parser.add_argument_group(
            'Stage1: Red pill or blue pill')

        experiment.add_argument("--pill-type",
                                choices=["red", "blue"],
                                help="""Red or blue""",
                                default="red")

    def init_multistage(self) -> None:
        super().init_multistage()

        neo = self.parser.add_argument_group('Neo Options')

        neo.add_argument("--using-powers",
                         help="""Do you believe you're the one or not?""",
                         action='store_true')


def to_cmdopts(args: argparse.Namespace) -> cmdopts: types.Cmdopts:
    return {
        'pill_type': args.pill_type,
        'using_powers': args.using_powers
    }

plugin.py

import typing as tp
import argparse
import logging
import pathlib

import implements

from sierra.core.experiment import bindings, definition
from sierra.core.variables import batch_criteria as bc
from sierra.core import types, utils
from sierra.plugins.execenv import hpc

from engine.matrix import cmdline as cmd


class ExpShellCmdsGenerator(bindings.IExpShellCmdsGenerator):
    """A class that conforms to
    :class:`~sierra.core.experiment.bindings.IExpShellCmdsGenerator`.
    """

    def __init__(self, cmdopts: types.Cmdopts, exp_num: int) -> None:
        pass

    def pre_exp_cmds(self) -> tp.List[types.ShellCmdSpec]:
        return []

    def post_exp_cmds(self) -> tp.List[types.ShellCmdSpec]:
        return []


class ExpRunShellCmdsGenerator(bindings.IExpRunShellCmdsGenerator):
    """A class that conforms to
    :class:`~sierra.core.experiment.bindings.IExpRunShellCmdsGenerator`.
    """

    def __init__(
        self,
        cmdopts: types.Cmdopts,
        criteria: bc.BatchCriteria,
        exp_num: int,
        n_agents: tp.Optional[int],
    ) -> None:
        pass

    def pre_run_cmds(
        self, host: str, input_fpath: pathlib.Path, run_num: int
    ) -> tp.List[types.ShellCmdSpec]:
        return []

    def exec_run_cmds(
        self, host: str, input_fpath: pathlib.Path, run_num: int
    ) -> tp.List[types.ShellCmdSpec]:
        return []

    def post_run_cmds(
        self, host: str, run_output_root: pathlib.Path
    ) -> tp.List[types.ShellCmdSpec]:
        return []


class ExpConfigurer(bindings.IExpConfigurer):
    """A class that conforms to
    :class:`~sierra.core.experiment.bindings.IExpConfigurer`.
    """

    def __init__(self, cmdopts: types.Cmdopts) -> None:
        self.cmdopts = cmdopts

    def for_exp_run(
        self, exp_input_root: pathlib.Path, run_output_root: pathlib.Path
    ) -> None:
        pass

    def for_exp(self, exp_input_root: pathlib.Path) -> None:
        pass

    def parallelism_paradigm(self) -> str:
        return "per-exp"


def cmdline_parser() -> argparse.ArgumentParser:
    """
    Get a cmdline parser supporting the engine. The returned parser
    should extend :class:`~sierra.core.cmdline.BaseCmdline`.

    This example extends :class:`~sierra.core.cmdline.BaseCmdline` with:

    - :class:`~sierra.plugins.hpc.execenv.cmdline.HPCCmdline` (HPC common)
    - :class:`~cmd.EngineCmdline` (engine specifics)

    assuming this engine can run on HPC environments.
    """
    # Initialize all stages and return the initialized
    # parser to SIERRA for use.
    parser = hpc.HPCCmdline([-1, 1, 2, 3, 4, 5]).parser
    return cmd.EngineCmdline(parents=[parser], stages=[-1, 1, 2, 3, 4, 5]).parser


def cmdline_postparse_configure(args: argparse.Namespace) -> argparse.Namespace:
    """
    Additional configuration and/or validation of the passed cmdline
    arguments pertaining to this engine. Validation should be performed
    with assert(), and the parsed argument object should be returned with any
    modifications/additions.
    """


def exec_env_check(cmdopts: types.Cmdopts):
    """
    Check the software environment (envvars, PATH, etc.) for this engine
    plugin prior to running anything in stage 2.
    """


def population_size_from_pickle(
    exp_def: tp.Union[definition.AttrChangeSet, definition.ElementAddList],
    main_config: types.YAMLDict,
    cmdopts: types.Cmdopts,
) -> int:
    """
    Given an experiment definition, main configuration, and cmdopts,
    get the # agents in the experiment.Size can be obtained from added
    tags or changed attributes; engine specific.

    Arguments:

        exp_def: *Part* of the pickled experiment definition object.

        main_config: Main project configuration.

        cmdopts: Dictionary of parsed cmdline options.

    """


def population_size_from_def(
    exp_def: definition.BaseExpDef, main_config: types.YAMLDict, cmdopts: types.Cmdopts
) -> int:
    """
    Arguments:

        exp_def: The *entire* experiment definition object.

        main_config: Main project configuration.

        cmdopts: Dictionary of parsed cmdline options.

    """


def agent_prefix_extract(main_config: types.YAMLDict, cmdopts: types.Cmdopts) -> str:
    """
    Arguments:

        main_config: Parsed dictionary of main YAML configuration.

        cmdopts: Dictionary of parsed command line options.
    """


def pre_exp_diagnostics(cmdopts: types.Cmdopts, logger: logging.Logger) -> None:
    """
    Arguments:

        cmdopts: Dictionary of parsed command line options.

        logger: The logger to log to.

    """


def arena_dims_from_criteria(criteria: bc.BatchCriteria) -> tp.List[utils.ArenaExtent]:
    """
    Arguments:

       criteria: The batch criteria built from cmdline specification.
    """


def expsetup_from_def(exp_def: definition.BaseExpDef) -> types.SimpleDict:
    """
    Given an experiment definition, compute the experiment setup information.
    Should contain keys:

        - ``duration`` - Duration in seconds.

        - ``n_ticks_per_sec`` - Ticks per second for controllers/sim.

    """

generators/engine.py

import pathlib

from sierra.core.experiment import definition
from sierra.core import types
from sierra.core.experiment import spec
from sierra.core import plugin_manager as pm


def for_all_exp(spec: spec.ExperimentSpec,
                controller: str,
                cmdopts: types.Cmdopts,
                expdef_template_fpath: pathlib.Path) -> definition.BaseExpDef:
    """
    Create an experiment definition from the
    ``--expdef-template`` and generate expdef changes to input files
    that are common to all experiments on the engine. All projects
    using this engine should derive from this class for `their`
    project-specific changes for the engine.

    Arguments:

        spec: The spec for the experimental run.

        controller: The controller used for the experiment, as passed
                    via ``--controller``.

        exp_def_template_fpath: The path to ``--expdef-template``.
    """
    # Only needed if your engine supports multiple input formats. Otherwise
    # just hardcode the string identifying the root element.
    fmt = pm.pipeline.get_plugin_module(cmdopts["expdef"])

    # Assuming engine takes a single file as input
    wr_config = definition.WriterConfig([{"src_parent": None,
                                          "src_tag": fmt.root_querypath(),
                                          "opath_leaf": ".myextension",
                                          "new_children": None,
                                          "new_children_parent": None,
                                          "rename_to": None
                                          }])
    module = pm.pipeline.get_plugin_module(cmdopts["expdef"])

    expdef = module.ExpDef(input_fpath=expdef_template_fpath,
                           write_config=wr_config)

    # Optional, only needed if your engine supports nested
    # configuration files.
    expdef.flatten(["pathstring1", "pathstring2"])

    return expdef


def for_single_exp_run(
        exp_def: definition.BaseExpDef,
        run_num: int,
        run_output_path: pathlib.Path,
        launch_stem_path: pathlib.Path,
        random_seed: int,
        cmdopts: types.Cmdopts) -> definition.BaseExpDef:
    """
    Generate expdef changes unique to a experimental run within an
    experiment for the matrix engine.

    Arguments:
        exp_def: The experiment definition after ``--engine`` changes
        common to all experiments have been made.

        run_num: The run # in the experiment.

        run_output_path: Path to run output directory within
                         experiment root (i.e., a leaf).

        launch_stem_path: Path to launch file in the input directory
                          for the experimental run, sans extension
                          or other modifications that the engine
                          can impose.

        random_seed: The random seed for the run.

        cmdopts: Dictionary containing parsed cmdline options.
    """
    pass

Finally--Connect to SIERRA!

After going through all the sections above and creating your plugin, tell SIERRA about it by putting $HOME/git/plugins/ on your SIERRA_PLUGIN_PATH so that your engine can be selected via --engine=engine.matrix.

Note

If your engine supports/requires a new execution environment, head over to New Execution Environment Plugin (--execenv).