These options are for all Platforms.

Bootstrap+Multi-stage Options

usage: __main__.py [-h] [--project PROJECT] [--version]
                   [--log-level {ERROR,INFO,WARNING,DEBUG,TRACE}]
                   [--platform PLATFORM] [--skip-pkg-checks]
                   [--exec-env EXEC_ENV] [--template-input-file filepath]
                   [--exp-overwrite] [--sierra-root dirpath]
                   [--batch-criteria [<category>.<definition>,...]
                   [[<category>.<definition>,...] ...]]
                   [--pipeline [PIPELINE ...]] [--exp-range EXP_RANGE]
                   [--platform-vc] [--processing-serial] [--n-runs N_RUNS]
                   [--skip-collate] [--plot-log-xscale]
                   [--plot-enumerated-xscale] [--plot-log-yscale]
                   [--plot-regression-lines PLOT_REGRESSION_LINES]
                   [--plot-primary-axis PLOT_PRIMARY_AXIS] [--plot-large-text]
                   [--plot-transpose-graphs]

Bootstrap options

Bare-bones options for bootstrapping SIERRA

--project

Specify which Project to load.

Tip

Used by stage {1,2,3,4,5}; can be omitted otherwise. If omitted: N/A.

--version, -v

Display SIERRA version information on stdout and then exit.

Default: False

--log-level

Possible choices: ERROR, INFO, WARNING, DEBUG, TRACE

The level of logging to use when running SIERRA.

Tip

Used by stage {1,2,3,4,5}; can be omitted otherwise. If omitted: N/A.

Default: “INFO”

--platform

This argument defines the Platform you want to run experiments on.

The value of this argument determines the execution environment for experiments; different platforms (e.g., simulator, real robots) will have different configuration options.

Valid values can be any folder name inside a folder on the SIERRA_PLUGIN_PATH (with / replaced with . as you would expect for using path names to address python packages). The platforms which come with SIERRA are:

  • platform.argos - This directs SIERRA to run experiments using the ARGoS simulator. Selecting this platform assumes your code has been developed and configured for ARGoS.

  • platform.ros1gazebo - This directs SIERRA to run experiments using the Gazebo simulator and ROS1. Selecting this platform assumes your code has been developed and configured for Gazebo and ROS1.

Default: “platform.argos”

--skip-pkg-checks

Skip the usual startup package checks. Only do this if you are SURE you will never use the SIERRA functionality which requires packages you don’t have installed/can’t install.

Default: False

--exec-env

This argument defines how experiments are going to be run, using the --platform you have selected.

Valid values can be any folder name inside a folder on the SIERRA_PLUGIN_PATH (with / replaced with . as you would expect for using path names to address python packages). The execution environments which come with SIERRA are:

  • hpc.local - This directs SIERRA to run experiments on the local machine. See Local HPC Plugin for a detailed description.

  • hpc.pbs - The directs SIERRA to run experiments spread across multiple allocated nodes in an HPC computing environment managed by TORQUE-PBS. See PBS HPC Plugin for a detailed description.

  • hpc.slurm - The directs SIERRA to run experiments spread across multiple allocated nodes in an HPC computing environment managed by SLURM. See SLURM HPC Plugin for a detailed description.

  • hpc.adhoc - This will direct SIERRA to run experiments on an ad-hoc network of computers. See Adhoc HPC Plugin for a detailed description.

  • robot.turtlebot3 - This will direct SIERRA to run experiments on real Turtlebots.

Not all platforms support all execution environments.

Tip

Used by stage {1,2}; can be omitted otherwise. If omitted: N/A.

Default: “hpc.local”

Multi-stage options

Options which are used in multiple pipeline stages

--template-input-file

The template .xml input file for the batch experiment. Beyond the requirements for the specific --platform, the content of the file can be any valid XML, with the exception of the SIERRA requirements detailed in Template Input Files.

Tip

Used by stage {1,2,3,4}; can be omitted otherwise. If omitted: N/A.

--exp-overwrite

When SIERRA calculates the batch experiment root (or any child path in the batch experiment root) during stage {1, 2}, if the calculated path already exists it is treated as a fatal error and no modifications to the filesystem are performed. This flag overwrides the default behavior. Provided to avoid accidentally overwrite input/output files for an experiment, forcing the user to be explicit with potentially dangerous actions.

Tip

Used by stage {1,2}; can be omitted otherwise. If omitted: N/A.

Default: False

--sierra-root

Root directory for all SIERRA generated and created files.

Subdirectories for controllers, scenarios, experiment/experimental run inputs/outputs will be created in this directory as needed. Can persist between invocations of SIERRA.

Tip

Used by stage {1,2,3,4,5}; can be omitted otherwise. If omitted: N/A.

Default: “<home directory>/exp”

--batch-criteria

Definition of criteria(s) to use to define the experiment.

Specified as a list of 0 or 1 space separated strings, each with the following general structure:

<category>.<definition>

<category> must be a filename from the core/variables/ or from a --project <project>/variables/ directory, and <definition> must be a parsable namne (according to the requirements of the criteria defined by the parser for <category>).

Tip

Used by stage {1,2,3,4,5}; can be omitted otherwise. If omitted: N/A.

Default: []

--pipeline

Define which stages of the experimental pipeline to run:

  • Stage1 - Generate the experiment definition from the template input file, batch criteria, and other command line options. Part of default pipeline.

  • Stage2 - Run a previously generated experiment. Part of default pipeline.

  • Stage3 - Post-process experimental results after running the batch experiment; some parts of this can be done in parallel. Part of default pipeline.

  • Stage4 - Perform deliverable generation after processing results for a batch experiment, which can include shiny graphs and videos. Part of default pipeline.

  • Stage5 - Perform graph generation for comparing controllers AFTER graph generation for batch experiments has been run. Not part of default pipeline.

Default: [1, 2, 3, 4]

--exp-range

Set the experiment numbers from the batch to run, average, generate intra-experiment graphs from, or generate inter-experiment graphs from (0 based). Specified in the form min_exp_num:max_exp_num (closed interval/inclusive). If omitted, runs, averages, and generates intra-experiment and inter-experiment performance measure graphs for all experiments in the batch (default behavior).

This is useful to re-run part of a batch experiment in HPC environments if SIERRA gets killed before it finishes running all experiments in the batch, or to redo a single experiment with real robots which failed for some reason.

Tip

Used by stage {2,3,4}; can be omitted otherwise. If omitted: N/A.

--platform-vc

For applicable --platforms, enable visual capturing of run-time data during stage 2. This data can be frames (i.e., .png files), or rendering videos, depending on the platform. If the captured data was frames, then SIERRA can render the captured frames into videos during stage 4. If the selected --platform does not support visual capture, then this option has no effect. See Platform Visual Capture for full details.

Tip

Used by stage {1,4}; can be omitted otherwise. If omitted: N/A.

Default: False

--processing-serial

If TRUE, then results processing/graph generation will be performed serially, rather than using parallellism where possible.

Tip

Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.

Default: False

--n-runs

The # of experimental runs that will be executed and their results processed to form the result of a single experiment within a batch.

If --platform is a simulator and --exec-env is something other than hpc.local then this may be be used to determine the concurrency of experimental runs.

Tip

Used by stage {1,2}; can be omitted otherwise. If omitted: N/A.

--skip-collate

Specify that no collation of data across experiments within a batch (stage 4) or across runs within an experiment (stage 3) should be performed. Useful if collation takes a long time and multiple types of stage 4 outputs are desired. Collation is generally idempotent unless you change the stage3 options (YMMV).

Tip

Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.

Default: False

--plot-log-xscale

Place the set of X values used to generate intra- and inter-experiment graphs into the logarithmic space. Mainly useful when the batch criteria involves large system sizes, so that the plots are more readable.

Tip

Applicable graphs:

Tip

Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.

Default: False

--plot-enumerated-xscale

Instead of using the values generated by a given batch criteria for the X values, use an enumerated list[0, …, len(X value) - 1]. Mainly useful when the batch criteria involves large system sizes, so that the plots are more readable.

Tip

Applicable graphs:

Tip

Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.

Default: False

--plot-log-yscale

Place the set of Y values used to generate intra - and inter-experiment graphs into the logarithmic space. Mainly useful when the batch criteria involves large system sizes, so that the plots are more readable.

Tip

Applicable graphs:

Tip

Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.

Default: False

--plot-regression-lines

For all 2D generated scatterplots, plot a linear regression line and the equation of the line to the legend.

Tip

Applicable graphs:

Tip

Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.

--plot-primary-axis

This option allows you to override the primary axis, which is normally is computed based on the batch criteria.

For example, in a bivariate batch criteria composed of

  • Population Size on the X axis (rows)

  • Another batch criteria which does not affect system size (columns)

Metrics will be calculated by computing across .csv rows and projecting down the columns by default, since system size will only vary within a row. Passing a value of 1 to this option will override this calculation, which can be useful in bivariate batch criteria in which you are interested in the effect of the OTHER non-size criteria on various performance measures.

0=criteria of interest varies across rows.

1=criteria of interest varies across columns.

This option only affects generating graphs from bivariate batch criteria.

Tip

Applicable graphs:

Tip

Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.

--plot-large-text

This option specifies that the title, X/Y axis labels/tick labels should be larger than the SIERRA default. This is useful when generating graphs suitable for two column paper format where the default text size for rendered graphs will be too small to see easily. The SIERRA defaults are generally fine for the one column/journal paper format.

Tip

Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.

Default: False

--plot-transpose-graphs

Transpose the X, Y axes in generated graphs. Useful as a general way to tweak graphs for best use of space within a paper.

Changed in version 1.2.20: Renamed from --transpose-graphs to make its relation to other plotting options clearer.

Tip

Applicable graphs:

Tip

Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.

Default: False

Stage1: Generating Experiments

None for the moment.

Stage2: Running Experiments

None for the moment.

Stage3: Processing Experiment Results

usage: SIERRA [--df-skip-verify] [--storage-medium {storage.csv}]
              [--dist-stats {none,all,conf95,bw}]
              [--processing-mem-limit PROCESSING_MEM_LIMIT]
              [--df-homogenize DF_HOMOGENIZE]

Stage3: General options for eprocessing experiment results

--df-skip-verify

SIERRA generally assumes/relies on all dataframes with the same name having the same # of columns which are of equivalent length across Experimental Runs (different columns within a dataframe can of course have different lengths). This is strictly verified during stage 3 by default.

If passed, then the verification step will be skipped during experimental results processing, and outputs will be averaged directly.

If not all the corresponding CSV files in all experiments generated the same # rows, then SIERRA will (probably) crash during experiments exist and/or have the stage4. Verification can take a long time with large # of runs and/or dataframes per experiment.

Tip

Used by stage {3}; can be omitted otherwise. If omitted: N/A.

Default: False

--storage-medium

Possible choices: storage.csv

Specify the storage medium for Experimental Run outputs, so that SIERRA can select an appropriate plugin to read them. Any plugin under plugins/storage can be used, but the ones that come with SIERRA are:

  • storage.csv - Experimental run outputs are stored in a per-run directory as one or more CSV files.

Regardless of the value of this option, SIERRA always generates CSV files as it runs and averages outputs, generates graphs, etc.

Tip

Used by stage {3}; can be omitted otherwise. If omitted: N/A.

Default: “storage.csv”

--dist-stats

Possible choices: none, all, conf95, bw

Specify what kinds of statistics, if any, should be calculated on the distribution of experimental data during stage 3 for inclusion on graphs during stage 4:

  • none - Only calculate and show raw mean on graphs.

  • conf95 - Calculate standard deviation of experimental distribution and show 95% confidence interval on relevant graphs.

  • bw - Calculate statistics necessary to show box and whisker plots around each point in the graph. :class: ~sierra.core.graphs.summary_line_graph.SummaryLineGraph only).

  • all - Generate all possible statistics, and plot all possible statistics on graphs.

Tip

Applicable graphs:

Tip

Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.

Default: “none”

--processing-mem-limit

Specify, as a percent in [0, 100], how much memory SIERRA should try to limit itself to using. This is useful on systems with limited memory, or on systems which are shared with other users without per-user memory restrictions.

Tip

Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.

Default: 90

--df-homogenize

SIERRA generally assumes/relies on all dataframes with the same name having the same # of columns which are of equivalent length across: term: Experimental Runs < Experimental Run > (different columns within a dataframe can of course have different lengths). This is checked during stage 3 unless --df-skip-verify is passed. If strict verification is skipped, then SIERRA provides the following options when processing dataframes during stage {3, 4} to to homogenize them:

  • none - Don’t do anything. This may or may not produce crashes during stage 4, depending on what you are doing.

  • pad - Project last valid value in columns which are too short down the column to make it match those which are longer.

    Note that this may result in invalid data/graphs if the filled columns are intervallic, interval average, or cumulative average data. If the data is a cumulative count of something, then this policy will have no ill effects.

  • zero - Same as pad, but always fill with zeroes.

Homogenization is performed just before writing dataframes to the specified storage medium. Useful with real robot experiments if the number of datapoints captured per-robot is slightly different, depending on when they started executing relative to when the experiment started.

Default: “none”

Stage4: Deliverable Generation

usage: SIERRA [--exp-graphs {intra,inter,all,none}] [--project-no-LN]
              [--project-no-HM] [--models-enable]
              [--render-cmd-opts RENDER_CMD_OPTS] [--project-imagizing]
              [--project-rendering] [--bc-rendering]

Stage4: General options for generating graphs

--exp-graphs

Possible choices: intra, inter, all, none

Specify which types of graphs should be generated from experimental results:

  • intra - Generate intra-experiment graphs from the results of a single experiment within a batch, for each experiment in the batch(this can take a long time with large batch experiments). If any intra-experiment models are defined and enabled, those are run and the results placed on appropriate graphs.

  • inter - Generate inter-experiment graphs _across_ the results of all experiments in a batch. These are very fast to generate, regardless of batch experiment size. If any inter-experiment models are defined and enabled, those are run and the results placed on appropriate graphs.

  • all - Generate all types of graphs.

  • none - Skip graph generation; provided to skip graph generation if only video outputs are desired.

Tip

Used by stage {4}; can be omitted otherwise. If omitted: N/A.

Default: “all”

--project-no-LN

Specify that the intra-experiment and inter-experiment linegraphs defined in project YAML configuration should not be generated. Useful if you are working on something which results in the generation of other types of graphs, and the generation of those linegraphs is not currently needed only slows down your development cycle.

Model linegraphs are still generated, if applicable.

Default: False

--project-no-HM

Specify that the intra-experiment heatmaps defined in project YAML configuration should not be generated. Useful if:

  • You are working on something which results in the generation of other types of graphs, and the generation of heatmaps only slows down your development cycle.

  • You are working on stage5 comparison graphs for bivariate batch criteria, and re-generating many heatmaps during stage4 is taking too long.

Model heatmaps are still generated, if applicable.

New in version 1.2.20.

Default: False

Models

--models-enable

Enable running of all models; otherwise, no models are run, even if they appear in the project config file. The logic behind having models disabled by default is that most users won’t have them.

Default: False

Stage4: Rendering (see also stage1 rendering options)

--render-cmd-opts

Specify the: program: ffmpeg options to appear between the specification of the input image files and the specification of the output file. The default is suitable for use with ARGoS frame grabbing set to a frames size of 1600x1200 to output a reasonable quality video.

Tip

Used by stage {4}; can be omitted otherwise. If omitted: N/A.

Default: “-r 10 -s:v 800x600 -c:v libx264 -crf 25 -filter:v scale=-2:956 -pix_fmt yuv420p”

--project-imagizing

Enable generation of image files from CSV files captured during stage 2 and averaged during stage 3 for each experiment. See Project Rendering for details and restrictions.

Tip

Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.

Default: False

--project-rendering

Enable generation of videos from imagized CSV files created as a result of --project-imagizing. See Project Rendering for details.

Tip

Used by stage {4}; can be omitted otherwise. If omitted: N/A.

Default: False

--bc-rendering

Enable generation of videos from generated graphs, such as heatmaps. Bivariate batch criteria only.

Tip

Used by stage {4}; can be omitted otherwise. If omitted: N/A.

Default: False

Stage5: Comparing Controllers

usage: SIERRA [--controllers-list CONTROLLERS_LIST]
              [--controllers-legend CONTROLLERS_LEGEND]
              [--scenarios-list SCENARIOS_LIST]
              [--scenarios-legend SCENARIOS_LEGEND] [--scenario-comparison]
              [--controller-comparison]
              [--comparison-type {LNraw,HMraw,HMdiff,HMscale,SUraw,SUscale,SUdiff}]
              [--bc-univar] [--bc-bivar]

Stage5: General options for controller comparison

--controllers-list

Comma separated list of controllers to compare within --sierra-root.

The first controller in this list will be used for as the controller of primary interest if --comparison-type is passed.

Tip

Used by stage {5}; can be omitted otherwise. If omitted: N/A.

--controllers-legend

Comma separated list of names to use on the legend for the generated comparison graphs, specified in the same order as the --controllers-list.

Tip

Used by stage {5}; can be omitted otherwise. If omitted: the raw controller names will be used.

--scenarios-list

Comma separated list of scenarios to compare --controller across within --sierra-root.

Tip

Used by stage {5}; can be omitted otherwise. If omitted: N/A.

--scenarios-legend

Comma separated list of names to use on the legend for the generated inter-scenario controller comparison graphs(if applicable), specified in the same order as the --scenarios-list.

Tip

Used by stage {5}; can be omitted otherwise. If omitted: the raw scenario names will be used.

--scenario-comparison

Perform a comparison of --controller across --scenarios-list (univariate batch criteria only).

Tip

Used by stage {5}; can be omitted otherwise. Either --scenario-comparison or --controller-comparison must be passed.

Default: False

--controller-comparison

Perform a comparison of --controllers-list across all scenarios at least one controller has been run on.

Tip

Used by stage {5}; can be omitted otherwise. Either --scenario-comparison or --controller-comparison must be passed.

Default: False

--comparison-type

Possible choices: LNraw, HMraw, HMdiff, HMscale, SUraw, SUscale, SUdiff

Specify how controller comparisons should be performed.

If the batch criteria is univariate, the options are:

  • LNraw - Output raw 1D performance measures using a single SummaryLineGraph for each measure, with all --controllers-list controllers shown on the same graph.

If the batch criteria is bivariate, the options are:

  • LNraw - Output raw performance measures as a set of :class:`~sierra.core.graphs.summary_line_graph.SummaryLineGraph`s, where each line graph is constructed from the i-th row/column for the 2D dataframe for the performance results for all controllers.

    Note

    SIERRA cannot currently plot statistics on the linegraphs built from slices of the 2D CSVs/heatmaps generated during stage4, because statistics generation is limited to stage3. This limitation may be removed in a future release.

  • HMraw - Output raw 2D performance measures as a set of dual heatmaps comparing all controllers against the controller of primary interest(one per pair).

  • HMdiff - Subtract the performance measure of the controller of primary interest against all other controllers, pairwise, outputting one 2D heatmap per comparison.

  • HMscale - Scale controller performance measures against those of the controller of primary interest by dividing, outputing one 2D heatmap per comparison.

  • SUraw - Output raw 3D performance measures as a single, stacked 3D surface plots comparing all controllers(identical plots, but viewed from different angles).

  • SUscale - Scale controller performance measures against those of the controller of primary interest by dividing. This results in a single stacked 3D surface plots comparing all controllers(identical plots, but viewed from different angles).

  • SUdiff - Subtract the performance measure of the controller of primary interest from each controller(including the primary). This results in a set single stacked 3D surface plots comparing all controllers(identical plots, but viewed from different angles), in which the controller of primary interest forms an(X, Y) plane at Z=0.

For all comparison types, --controllers-legend is used if passed for legend.

Tip

Used by stage {5}; can be omitted otherwise. If omitted: N/A.

--bc-univar

Specify that the batch criteria is univariate. This cannot be deduced from the command line --batch-criteria argument in all cases because we are comparing controllers across scenarios, and each scenario (potentially) has a different batch criteria definition, which will result in (potentially) erroneous comparisons if we don’t re-generate the batch criteria for each scenaro we compare controllers within.

Tip

Used by stage {5}; can be omitted otherwise. If omitted: N/A.

Default: False

--bc-bivar

Specify that the batch criteria is bivariate. This cannot be deduced from the command line --batch-criteria argument in all cases because we are comparing controllers across scenarios, and each scenario(potentially) has a different batch criteria definition, which will result in (potentially) erroneous comparisons if we don’t re-generate the batch criteria for each scenaro we compare controllers in .

Tip

Used by stage {5}; can be omitted otherwise. If omitted: N/A.

Default: False