Core Command Line Reference#

If an option is given more than once, the last such occurrence is used. If both the shortform and longform variants of an option are passed with different values, shortform wins.

See also Usage By Example.

SIERRA Core#

These options apply to all Experiments, Engines, Batch Criteria, etc.

Bootstrap Options#

sierra [-h] [--project PROJECT] [--version] [--log-level {ERROR,INFO,WARNING,DEBUG,TRACE}]
       [--engine ENGINE] [--skip-pkg-checks] [--execenv EXECENV] [--expdef EXPDEF]
       [--storage STORAGE] [--proc PROC [PROC ...]] [--prod PROD [PROD ...]]
       [--compare COMPARE [COMPARE ...]] [--rcfile RCFILE]

sierra options#

  • -h, --help - show this help message and exit

  • --project PROJECT -

    Specify which Project to load.

    (default: None)

  • --version, -v -

    Display SIERRA version information on stdout and then exit.

  • --log-level LOG_LEVEL -

    The level of logging to use when running SIERRA.

    (default: INFO)

  • --engine ENGINE -

    This argument defines the Engine you want to run experiments on.

    The value of this argument determines the execution environment for experiments; different engines (e.g., simulator, real robots) will have different configuration options.

    Valid values can be any folder name inside a folder on the SIERRA_PLUGIN_PATH (with / replaced with . as you would expect for using path names to address python packages). The engines which come with SIERRA are:

    • engine.argos - This directs SIERRA to run experiments using the ARGoS simulator. Selecting this engine assumes your code has been developed and configured for ARGoS.

    • engine.ros1gazebo - This directs SIERRA to run experiments using the Gazebo simulator and ROS1. Selecting this engine assumes your code has been developed and configured for Gazebo and ROS1.

    • engine.ros1robot - This directs SIERRA to run experiments using ROS1 on a real robot of your choosing. Selecting this engine assumes your code has been developed and configured for ROS1.

    (default: engine.argos)

  • --skip-pkg-checks -

    Skip the usual startup package checks. Only do this if you are SURE you will never use the SIERRA functionality which requires packages you do not have installed or cannot install.

  • --execenv EXECENV -

    This argument defines how experiments are going to be run, using the --engine you have selected.

    Valid values can be any folder name inside a folder on the SIERRA_PLUGIN_PATH (with / replaced with . as you would expect for using path names to address python packages). The execution environments which come with SIERRA are:

    • hpc.local - This directs SIERRA to run experiments on the local machine. See Local HPC Plugin for a detailed description.

    • hpc.pbs - The directs SIERRA to run experiments spread across multiple allocated nodes in an HPC computing environment managed by TORQUE-PBS. See PBS HPC Plugin for a detailed description.

    • hpc.slurm - The directs SIERRA to run experiments spread across multiple allocated nodes in an HPC computing environment managed by SLURM. See SLURM HPC Plugin for a detailed description.

    • hpc.adhoc - This will direct SIERRA to run experiments on an ad-hoc network of computers. See Adhoc HPC Plugin for a detailed description.

    • prefectserver.local - This will direct SIERRA to run experiments locally using a spun-up Prefect job scheduler/server. See Local Prefect Server Plugin for a detailed description.

    • prefectserver.dockerremote - This will direct SIERRA to run experiments remote using a Prefect job scheduler/server in docker containers. See Remote+Docker Prefect Plugin for a detailed description.

    • robot.turtlebot3 - This will direct SIERRA to run experiments on real Turtlebots.

    Not all engines support all execution environments.

    (default: hpc.local)

  • --expdef EXPDEF -

    Specify the experiment definition format, so that SIERRA can select an appropriate plugin to read/write files of that format, and manipulate in-memory representations to create runnable experiments. Any plugin on SIERRA_PLUGIN_PATH can be used, but the ones that come with SIERRA are:

    • expdef.xml - Experimental definitions are created from XML files. This causes --expdef-template to be parsed as XML.

    • expdef.json - Experimental definitions are created from json files. This causes --expdef-template to be parsed as JSON.

    (default: expdef.xml)

  • --storage STORAGE -

    Specify the storage medium for Experimental Run outputs, so that SIERRA can select an appropriate plugin to read them. Any plugin on SIERRA_PLUGIN_PATH, but the ones that come with SIERRA are:

    • storage.csv - Experimental run outputs are stored in a per-run directory as one or more CSV files.

    • storage.arrow - Experimental run outputs are stored in a per-run directory as one or more apache arrow files.

    Regardless of the value of this option, SIERRA always generates CSV files as it runs and averages outputs, generates graphs, etc.

    (default: storage.csv)

  • --proc PROC -

    Specify the set of plugins to run during stage 3 for data processing. The plugins are executed IN ORDER of appearance, so make sure to handle dependencies. Any plugin on SIERRA_PLUGIN_PATH can be used, but the ones that come with SIERRA are:

    (default: ['proc.statistics', 'proc.collate'])

  • --prod PROD -

    Specify the set of plugins to run during stage 4 for product/deliverable generation. The plugins are executed IN ORDER of appearance, so make sure to handle dependencies. Any plugin on SIERRA_PLUGIN_PATH can be used, but the ones that come with SIERRA are:

    (default: ['prod.graphs'])

  • --compare COMPARE -

    Specify the set of plugins to run during stage 5 for product/deliverable comparison. The plugins are executed IN ORDER of appearance, so make sure to handle dependencies. Any plugin on SIERRA_PLUGIN_PATH can be used, but the ones that come with SIERRA are:

    • compare.graphs - Combine previously generated graphs onto a single plot.

    (default: ['compare.graphs'])

  • --rcfile RCFILE -

    Specify the rcfile SIERRA should read additional cmdline arguments from. Any rcfile arguments overriden by cmdline args, if both are present. See also SIERRA_RCFILE.

    (default: None)

Multistage Options#

sierra [--expdef-template filepath] [--exp-overwrite] [--sierra-root dirpath]
       [--batch-criteria [<category>.<definition>,...] [[<category>.<definition>,...] ...]]
       [--pipeline [PIPELINE ...]] [--exp-range EXP_RANGE] [--engine-vc]
       [--processing-parallelism PROCESSING_PARALLELISM]
       [--exec-parallelism-paradigm {per-batch,per-exp,per-run,None}] [--n-runs N_RUNS]
       [-p P [P ...]] [-e E [E ...]] [-x X [X ...]] [-s S [S ...]]

sierra Multi-stage options#

Options which are used in multiple pipeline stages

  • --expdef-template FILEPATH -

    The template .xml input file for the batch experiment. Beyond the requirements for the specific --engine, the content of the file can be anything valid for the format, with the exception of the SIERRA requirements detailed in Experiment Definition (--expdef).

    (default: None)

  • --exp-overwrite -

    When SIERRA calculates the batch experiment root (or any child path in the batch experiment root) during stage {1, 2}, if the calculated path already exists it is treated as a fatal error and no modifications to the filesystem are performed. This flag overwrides the default behavior. Provided to avoid accidentally overwrite input/output files for an experiment, forcing the user to be explicit with potentially dangerous actions.

  • --sierra-root DIRPATH -

    Root directory for all SIERRA generated and created files.

    Subdirectories for controllers, scenarios, experiment/experimental run inputs/outputs will be created in this directory as needed. Can persist between invocations of SIERRA.

    (default: <home directory>/exp)

  • --batch-criteria [<CATEGORY>.<DEFINITION>,...], --bc [<CATEGORY>.<DEFINITION>,...] -

    Definition of criteria(s) to use to define the experiment.

    Specified as a list of 0 or 1 space separated strings, each with the following general structure:

    <category>.<definition>

    <category> must be a filename from the core/variables/ or from a --project <project>/variables/ directory, and <definition> must be a parsable namne (according to the requirements of the criteria defined by the parser for <category>).

    (default: [])

  • --pipeline PIPELINE -

    Define which stages of the experimental pipeline to run:

    • Stage1 - Generate the experiment definition from the template input file, batch criteria, and other command line options. Part of default pipeline.

    • Stage2 - Run a previously generated experiment. Part of default pipeline.

    • Stage3 - Post-process experimental results after running the batch experiment; some parts of this can be done in parallel. Part of default pipeline.

    • Stage4 - Perform deliverable/product generation after processing results for a batch experiment, which can include shiny graphs and videos. Part of default pipeline.

    • Stage5 - Compare generated products.

    (default: [1, 2, 3, 4])

  • --exp-range EXP_RANGE -

    Set the experiment numbers from the batch to run, average, generate intra-experiment graphs from, or generate inter-experiment graphs from (0 based). Specified in the form min_exp_num:max_exp_num (closed interval/inclusive). If omitted, runs, averages, and generates intra-experiment and inter-experiment performance measure graphs for all experiments in the batch (default behavior).

    This is useful to re-run part of a batch experiment in HPC environments if SIERRA gets killed before it finishes running all experiments in the batch, or to redo a single experiment with real robots which failed for some reason.

    (default: None)

  • --engine-vc -

    For applicable --engines, enable visual capturing of run-time data during stage 2. This data can be frames (i.e., .png files), or rendering videos, depending on the engine. If the captured data was frames, then SIERRA can render the captured frames into videos during stage 4. If the selected --engine does not support visual capture, then this option has no effect. See Rendering for more details.

  • --processing-parallelism PROCESSING_PARALLELISM -

    The level of parallelism to use in results processing/graph generation producer-consumer queue. This value is used to allocate consumers and produces in a 3:1 ratio. If you are doing a LOT of processing, you may want to oversubscribe your machine by passing a higher than default value to overcome slowdown with high disk I/O.

    (default: 2)

  • --exec-parallelism-paradigm EXEC_PARALLELISM_PARADIGM -

    The level of parallelism to use when executing experiments in

    stage 2. This should generall be set by parallelism_paradigm() in your selected Engine; this option is provided as an override.

    (default: None)

  • --n-runs N_RUNS -

    The # of experimental runs that will be executed and their results processed to form the result of a single experiment within a batch.

    If --engine is a simulator and --execenv is something other than hpc.local then this may be be used to determine the concurrency of experimental runs.

    (default: 1)

sierra Shortform aliases#

Most cmdline options to SIERRA are longform (i.e., ``--option``), but some families of options have shortforms (i.e., ``-o`` for ``--option``) as well. Shortform arguments behave the same as their longform counterparts. If both are passed, the the shortform overrides.

  • -p P -

    Alias of --plot-; any number of --plot-XX option can be passed with separate -p arguments . If the argument to -p does not map to a valid plot-XX option it is ignored.

    Added in version 1.3.14.

    (default: None)

  • -e E -

    Alias of --exp-; any number of --exp-XX option can be passed with separate -e arguments . If the argument to -e does not map to a valid exp-XX option it is ignored.

    Added in version 1.3.14.

    (default: None)

  • -x X -

    Alias of --exec-; any number of --exec-XX option can be passed with separate -x arguments . If the argument to -x does not map to a valid exec-XX option it is ignored.

    Added in version 1.3.14.

    (default: None)

  • -s S -

    Alias of --skip; any number of --skip-XX option can be passed with separate -s arguments . If the argument to -s does not map to a valid skip-XX option it is ignored.

    Added in version 1.3.14.

    (default: None)

Stage1 Options#

sierra [--preserve-seeds | --no-preserve-seeds]

sierra Stage 1 options#

Options for generating experiments

  • --preserve-seeds, --no-preserve-seeds -

    Preserve previously generated random seeds for experiments (the default). Useful for regenerating experiments when you change parameters/python code that don't affects experimental outputs (e.g., paths). Preserving/overwriting random seeds is not affected by --exp-overwrite.

    (default: %(default)s)

    (default: True)

Stage2 Options#

sierra [--nodefile NODEFILE]

sierra Stage 2 options#

Options for running experiments

  • --nodefile NODEFILE -

    Specify a list of compute nodes which SIERRA will use to run jobs. For simulator Engines, these are HPC resource nodes. For real robot engines, these are robot hostnames/IP addresses. This information can also be supplied via the SIERRA_NODEFILE environment variable; this argument takes priority if both are supplied.

    (default: None)

Stage3 Options#

sierra [--df-verify] [--processing-mem-limit PROCESSING_MEM_LIMIT]
       [--df-homogenize DF_HOMOGENIZE]

sierra Stage 3 options#

Options for processing experiment results

  • --df-verify -

    SIERRA generally assumes/relies on all dataframes with the same name having the same # of columns which are of equivalent length across Experimental Runs (different columns within a dataframe can of course have different lengths). However this is not checked by default.

    If passed, then the verification step not be skipped, and will be executed during experimental results processing, and outputs will be averaged directly.

    If not all the corresponding CSV files in all experiments generated the same # rows, then SIERRA will (probably) crash during experiments exist and/or have the stage4. Verification can take a long time with large # of runs and/or dataframes per experiment.

  • --processing-mem-limit PROCESSING_MEM_LIMIT -

    Specify, as a percent in [0, 100], how much memory SIERRA should try to limit itself to using. This is useful on systems with limited memory, or on systems which are shared with other users without per-user memory restrictions.

    (default: 90)

  • --df-homogenize DF_HOMOGENIZE -

    SIERRA generally assumes/relies on all dataframes with the same name having the same # of columns which are of equivalent length across: term: Experimental Runs < Experimental Run > (different columns within a dataframe can of course have different lengths). This not checked unless --df-verify is passed. If strict verification is skipped, then SIERRA provides the following options when processing dataframes during stage {3, 4} to to homogenize them:

    • none - Don't do anything. This may or may not produce crashes during stage 4, depending on what you are doing.

    • pad - Project last valid value in columns which are too short down the column to make it match those which are longer.

    Note that this may result in invalid data/graphs if the filled columns are intervallic, interval average, or cumulative average data. If the data is a cumulative count of something, then this policy will have no ill effects.

    • zero - Same as pad, but always fill with zeroes.

    Homogenization is performed just before writing dataframes to the specified storage medium. Useful with real robot experiments if the number of datapoints captured per-robot is slightly different, depending on when they started executing relative to when the experiment started.

    (default: none)

Plugins#

See docs for individual Builtin Plugins for details.