These options are for all Platforms.
Bootstrap+Multi-stage Options
usage: __main__.py [-h] [--project PROJECT] [--version]
[--log-level {ERROR,INFO,WARNING,DEBUG,TRACE}]
[--platform PLATFORM] [--skip-pkg-checks]
[--exec-env EXEC_ENV] [--template-input-file filepath]
[--exp-overwrite] [--sierra-root dirpath]
[--batch-criteria [<category>.<definition>,...]
[[<category>.<definition>,...] ...]]
[--pipeline [PIPELINE ...]] [--exp-range EXP_RANGE]
[--platform-vc] [--processing-serial] [--n-runs N_RUNS]
[--skip-collate] [--plot-log-xscale]
[--plot-enumerated-xscale] [--plot-log-yscale]
[--plot-regression-lines PLOT_REGRESSION_LINES]
[--plot-primary-axis PLOT_PRIMARY_AXIS] [--plot-large-text]
[--plot-transpose-graphs]
Bootstrap options
Bare-bones options for bootstrapping SIERRA
- --project
Specify which Project to load.
Tip
Used by stage {1,2,3,4,5}; can be omitted otherwise. If omitted: N/A.
- --version, -v
Display SIERRA version information on stdout and then exit.
Default: False
- --log-level
Possible choices: ERROR, INFO, WARNING, DEBUG, TRACE
The level of logging to use when running SIERRA.
Tip
Used by stage {1,2,3,4,5}; can be omitted otherwise. If omitted: N/A.
Default: “INFO”
- --platform
This argument defines the Platform you want to run experiments on.
The value of this argument determines the execution environment for experiments; different platforms (e.g., simulator, real robots) will have different configuration options.
Valid values can be any folder name inside a folder on the
SIERRA_PLUGIN_PATH
(with/
replaced with.
as you would expect for using path names to address python packages). The platforms which come with SIERRA are:platform.argos
- This directs SIERRA to run experiments using the ARGoS simulator. Selecting this platform assumes your code has been developed and configured for ARGoS.platform.ros1gazebo
- This directs SIERRA to run experiments using the Gazebo simulator and ROS1. Selecting this platform assumes your code has been developed and configured for Gazebo and ROS1.
Default: “platform.argos”
- --skip-pkg-checks
Skip the usual startup package checks. Only do this if you are SURE you will never use the SIERRA functionality which requires packages you don’t have installed/can’t install.
Default: False
- --exec-env
This argument defines how experiments are going to be run, using the
--platform
you have selected.Valid values can be any folder name inside a folder on the
SIERRA_PLUGIN_PATH
(with/
replaced with.
as you would expect for using path names to address python packages). The execution environments which come with SIERRA are:hpc.local
- This directs SIERRA to run experiments on the local machine. See Local HPC Plugin for a detailed description.hpc.pbs
- The directs SIERRA to run experiments spread across multiple allocated nodes in an HPC computing environment managed by TORQUE-PBS. See PBS HPC Plugin for a detailed description.hpc.slurm
- The directs SIERRA to run experiments spread across multiple allocated nodes in an HPC computing environment managed by SLURM. See SLURM HPC Plugin for a detailed description.hpc.adhoc
- This will direct SIERRA to run experiments on an ad-hoc network of computers. See Adhoc HPC Plugin for a detailed description.robot.turtlebot3
- This will direct SIERRA to run experiments on real Turtlebots.
Not all platforms support all execution environments.
Tip
Used by stage {1,2}; can be omitted otherwise. If omitted: N/A.
Default: “hpc.local”
Multi-stage options
Options which are used in multiple pipeline stages
- --template-input-file
The template
.xml
input file for the batch experiment. Beyond the requirements for the specific--platform
, the content of the file can be any valid XML, with the exception of the SIERRA requirements detailed in Template Input Files.Tip
Used by stage {1,2,3,4}; can be omitted otherwise. If omitted: N/A.
- --exp-overwrite
When SIERRA calculates the batch experiment root (or any child path in the batch experiment root) during stage {1, 2}, if the calculated path already exists it is treated as a fatal error and no modifications to the filesystem are performed. This flag overwrides the default behavior. Provided to avoid accidentally overwrite input/output files for an experiment, forcing the user to be explicit with potentially dangerous actions.
Tip
Used by stage {1,2}; can be omitted otherwise. If omitted: N/A.
Default: False
- --sierra-root
Root directory for all SIERRA generated and created files.
Subdirectories for controllers, scenarios, experiment/experimental run inputs/outputs will be created in this directory as needed. Can persist between invocations of SIERRA.
Tip
Used by stage {1,2,3,4,5}; can be omitted otherwise. If omitted: N/A.
Default: “<home directory>/exp”
- --batch-criteria
Definition of criteria(s) to use to define the experiment.
Specified as a list of 0 or 1 space separated strings, each with the following general structure:
<category>.<definition>
<category>
must be a filename from thecore/variables/
or from a--project
<project>/variables/
directory, and<definition>
must be a parsable namne (according to the requirements of the criteria defined by the parser for<category>
).Tip
Used by stage {1,2,3,4,5}; can be omitted otherwise. If omitted: N/A.
Default: []
- --pipeline
Define which stages of the experimental pipeline to run:
Stage1 - Generate the experiment definition from the template input file, batch criteria, and other command line options. Part of default pipeline.
Stage2 - Run a previously generated experiment. Part of default pipeline.
Stage3 - Post-process experimental results after running the batch experiment; some parts of this can be done in parallel. Part of default pipeline.
Stage4 - Perform deliverable generation after processing results for a batch experiment, which can include shiny graphs and videos. Part of default pipeline.
Stage5 - Perform graph generation for comparing controllers AFTER graph generation for batch experiments has been run. Not part of default pipeline.
Default: [1, 2, 3, 4]
- --exp-range
Set the experiment numbers from the batch to run, average, generate intra-experiment graphs from, or generate inter-experiment graphs from (0 based). Specified in the form
min_exp_num:max_exp_num
(closed interval/inclusive). If omitted, runs, averages, and generates intra-experiment and inter-experiment performance measure graphs for all experiments in the batch (default behavior).This is useful to re-run part of a batch experiment in HPC environments if SIERRA gets killed before it finishes running all experiments in the batch, or to redo a single experiment with real robots which failed for some reason.
Tip
Used by stage {2,3,4}; can be omitted otherwise. If omitted: N/A.
- --platform-vc
For applicable
--platforms
, enable visual capturing of run-time data during stage 2. This data can be frames (i.e., .png files), or rendering videos, depending on the platform. If the captured data was frames, then SIERRA can render the captured frames into videos during stage 4. If the selected--platform
does not support visual capture, then this option has no effect. See Platform Visual Capture for full details.Tip
Used by stage {1,4}; can be omitted otherwise. If omitted: N/A.
Default: False
- --processing-serial
If TRUE, then results processing/graph generation will be performed serially, rather than using parallellism where possible.
Tip
Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.
Default: False
- --n-runs
The # of experimental runs that will be executed and their results processed to form the result of a single experiment within a batch.
If
--platform
is a simulator and--exec-env
is something other thanhpc.local
then this may be be used to determine the concurrency of experimental runs.Tip
Used by stage {1,2}; can be omitted otherwise. If omitted: N/A.
- --skip-collate
Specify that no collation of data across experiments within a batch (stage 4) or across runs within an experiment (stage 3) should be performed. Useful if collation takes a long time and multiple types of stage 4 outputs are desired. Collation is generally idempotent unless you change the stage3 options (YMMV).
Tip
Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.
Default: False
- --plot-log-xscale
Place the set of X values used to generate intra- and inter-experiment graphs into the logarithmic space. Mainly useful when the batch criteria involves large system sizes, so that the plots are more readable.
Tip
Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.
Default: False
- --plot-enumerated-xscale
Instead of using the values generated by a given batch criteria for the X values, use an enumerated list[0, …, len(X value) - 1]. Mainly useful when the batch criteria involves large system sizes, so that the plots are more readable.
Tip
Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.
Default: False
- --plot-log-yscale
Place the set of Y values used to generate intra - and inter-experiment graphs into the logarithmic space. Mainly useful when the batch criteria involves large system sizes, so that the plots are more readable.
Tip
Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.
Default: False
- --plot-regression-lines
For all 2D generated scatterplots, plot a linear regression line and the equation of the line to the legend.
Tip
Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.
- --plot-primary-axis
This option allows you to override the primary axis, which is normally is computed based on the batch criteria.
For example, in a bivariate batch criteria composed of
Population Size on the X axis (rows)
Another batch criteria which does not affect system size (columns)
Metrics will be calculated by computing across .csv rows and projecting down the columns by default, since system size will only vary within a row. Passing a value of 1 to this option will override this calculation, which can be useful in bivariate batch criteria in which you are interested in the effect of the OTHER non-size criteria on various performance measures.
0=criteria of interest varies across rows.
1=criteria of interest varies across columns.
This option only affects generating graphs from bivariate batch criteria.
Tip
Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.
- --plot-large-text
This option specifies that the title, X/Y axis labels/tick labels should be larger than the SIERRA default. This is useful when generating graphs suitable for two column paper format where the default text size for rendered graphs will be too small to see easily. The SIERRA defaults are generally fine for the one column/journal paper format.
Tip
Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.
Default: False
- --plot-transpose-graphs
Transpose the X, Y axes in generated graphs. Useful as a general way to tweak graphs for best use of space within a paper.
Changed in version 1.2.20: Renamed from
--transpose-graphs
to make its relation to other plotting options clearer.Tip
Used by stage {4,5}; can be omitted otherwise. If omitted: N/A.
Default: False
Stage1: Generating Experiments
None for the moment.
Stage2: Running Experiments
None for the moment.
Stage3: Processing Experiment Results
usage: SIERRA [--df-skip-verify] [--storage-medium {storage.csv}]
[--dist-stats {none,all,conf95,bw}]
[--processing-mem-limit PROCESSING_MEM_LIMIT]
[--df-homogenize DF_HOMOGENIZE]
Stage3: General options for eprocessing experiment results
- --df-skip-verify
SIERRA generally assumes/relies on all dataframes with the same name having the same # of columns which are of equivalent length across Experimental Runs (different columns within a dataframe can of course have different lengths). This is strictly verified during stage 3 by default.
If passed, then the verification step will be skipped during experimental results processing, and outputs will be averaged directly.
If not all the corresponding CSV files in all experiments generated the same # rows, then SIERRA will (probably) crash during experiments exist and/or have the stage4. Verification can take a long time with large # of runs and/or dataframes per experiment.
Tip
Used by stage {3}; can be omitted otherwise. If omitted: N/A.
Default: False
- --storage-medium
Possible choices: storage.csv
Specify the storage medium for Experimental Run outputs, so that SIERRA can select an appropriate plugin to read them. Any plugin under
plugins/storage
can be used, but the ones that come with SIERRA are:storage.csv
- Experimental run outputs are stored in a per-run directory as one or more CSV files.
Regardless of the value of this option, SIERRA always generates CSV files as it runs and averages outputs, generates graphs, etc.
Tip
Used by stage {3}; can be omitted otherwise. If omitted: N/A.
Default: “storage.csv”
- --dist-stats
Possible choices: none, all, conf95, bw
Specify what kinds of statistics, if any, should be calculated on the distribution of experimental data during stage 3 for inclusion on graphs during stage 4:
none
- Only calculate and show raw mean on graphs.conf95
- Calculate standard deviation of experimental distribution and show 95% confidence interval on relevant graphs.bw
- Calculate statistics necessary to show box and whisker plots around each point in the graph. :class: ~sierra.core.graphs.summary_line_graph.SummaryLineGraph only).all
- Generate all possible statistics, and plot all possible statistics on graphs.
Tip
Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.
Default: “none”
- --processing-mem-limit
Specify, as a percent in [0, 100], how much memory SIERRA should try to limit itself to using. This is useful on systems with limited memory, or on systems which are shared with other users without per-user memory restrictions.
Tip
Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.
Default: 90
- --df-homogenize
SIERRA generally assumes/relies on all dataframes with the same name having the same # of columns which are of equivalent length across: term: Experimental Runs < Experimental Run > (different columns within a dataframe can of course have different lengths). This is checked during stage 3 unless
--df-skip-verify
is passed. If strict verification is skipped, then SIERRA provides the following options when processing dataframes during stage {3, 4} to to homogenize them:none
- Don’t do anything. This may or may not produce crashes during stage 4, depending on what you are doing.pad
- Project last valid value in columns which are too short down the column to make it match those which are longer.Note that this may result in invalid data/graphs if the filled columns are intervallic, interval average, or cumulative average data. If the data is a cumulative count of something, then this policy will have no ill effects.
zero
- Same aspad
, but always fill with zeroes.
Homogenization is performed just before writing dataframes to the specified storage medium. Useful with real robot experiments if the number of datapoints captured per-robot is slightly different, depending on when they started executing relative to when the experiment started.
Default: “none”
Stage4: Deliverable Generation
usage: SIERRA [--exp-graphs {intra,inter,all,none}] [--project-no-LN]
[--project-no-HM] [--models-enable]
[--render-cmd-opts RENDER_CMD_OPTS] [--project-imagizing]
[--project-rendering] [--bc-rendering]
Stage4: General options for generating graphs
- --exp-graphs
Possible choices: intra, inter, all, none
Specify which types of graphs should be generated from experimental results:
intra
- Generate intra-experiment graphs from the results of a single experiment within a batch, for each experiment in the batch(this can take a long time with large batch experiments). If any intra-experiment models are defined and enabled, those are run and the results placed on appropriate graphs.inter
- Generate inter-experiment graphs _across_ the results of all experiments in a batch. These are very fast to generate, regardless of batch experiment size. If any inter-experiment models are defined and enabled, those are run and the results placed on appropriate graphs.all
- Generate all types of graphs.none
- Skip graph generation; provided to skip graph generation if only video outputs are desired.
Tip
Used by stage {4}; can be omitted otherwise. If omitted: N/A.
Default: “all”
- --project-no-LN
Specify that the intra-experiment and inter-experiment linegraphs defined in project YAML configuration should not be generated. Useful if you are working on something which results in the generation of other types of graphs, and the generation of those linegraphs is not currently needed only slows down your development cycle.
Model linegraphs are still generated, if applicable.
Default: False
- --project-no-HM
Specify that the intra-experiment heatmaps defined in project YAML configuration should not be generated. Useful if:
You are working on something which results in the generation of other types of graphs, and the generation of heatmaps only slows down your development cycle.
You are working on stage5 comparison graphs for bivariate batch criteria, and re-generating many heatmaps during stage4 is taking too long.
Model heatmaps are still generated, if applicable.
New in version 1.2.20.
Default: False
Models
- --models-enable
Enable running of all models; otherwise, no models are run, even if they appear in the project config file. The logic behind having models disabled by default is that most users won’t have them.
Default: False
Stage4: Rendering (see also stage1 rendering options)
- --render-cmd-opts
Specify the: program: ffmpeg options to appear between the specification of the input image files and the specification of the output file. The default is suitable for use with ARGoS frame grabbing set to a frames size of 1600x1200 to output a reasonable quality video.
Tip
Used by stage {4}; can be omitted otherwise. If omitted: N/A.
Default: “-r 10 -s:v 800x600 -c:v libx264 -crf 25 -filter:v scale=-2:956 -pix_fmt yuv420p”
- --project-imagizing
Enable generation of image files from CSV files captured during stage 2 and averaged during stage 3 for each experiment. See Project Rendering for details and restrictions.
Tip
Used by stage {3,4}; can be omitted otherwise. If omitted: N/A.
Default: False
- --project-rendering
Enable generation of videos from imagized CSV files created as a result of
--project-imagizing
. See Project Rendering for details.Tip
Used by stage {4}; can be omitted otherwise. If omitted: N/A.
Default: False
- --bc-rendering
Enable generation of videos from generated graphs, such as heatmaps. Bivariate batch criteria only.
Tip
Used by stage {4}; can be omitted otherwise. If omitted: N/A.
Default: False
Stage5: Comparing Controllers
usage: SIERRA [--controllers-list CONTROLLERS_LIST]
[--controllers-legend CONTROLLERS_LEGEND]
[--scenarios-list SCENARIOS_LIST]
[--scenarios-legend SCENARIOS_LEGEND] [--scenario-comparison]
[--controller-comparison]
[--comparison-type {LNraw,HMraw,HMdiff,HMscale,SUraw,SUscale,SUdiff}]
[--bc-univar] [--bc-bivar]
Stage5: General options for controller comparison
- --controllers-list
Comma separated list of controllers to compare within
--sierra-root
.The first controller in this list will be used for as the controller of primary interest if
--comparison-type
is passed.Tip
Used by stage {5}; can be omitted otherwise. If omitted: N/A.
- --controllers-legend
Comma separated list of names to use on the legend for the generated comparison graphs, specified in the same order as the
--controllers-list
.Tip
Used by stage {5}; can be omitted otherwise. If omitted: the raw controller names will be used.
- --scenarios-list
Comma separated list of scenarios to compare
--controller
across within--sierra-root
.Tip
Used by stage {5}; can be omitted otherwise. If omitted: N/A.
- --scenarios-legend
Comma separated list of names to use on the legend for the generated inter-scenario controller comparison graphs(if applicable), specified in the same order as the
--scenarios-list
.Tip
Used by stage {5}; can be omitted otherwise. If omitted: the raw scenario names will be used.
- --scenario-comparison
Perform a comparison of
--controller
across--scenarios-list
(univariate batch criteria only).Tip
Used by stage {5}; can be omitted otherwise. Either
--scenario-comparison
or--controller-comparison
must be passed.Default: False
- --controller-comparison
Perform a comparison of
--controllers-list
across all scenarios at least one controller has been run on.Tip
Used by stage {5}; can be omitted otherwise. Either
--scenario-comparison
or--controller-comparison
must be passed.Default: False
- --comparison-type
Possible choices: LNraw, HMraw, HMdiff, HMscale, SUraw, SUscale, SUdiff
Specify how controller comparisons should be performed.
If the batch criteria is univariate, the options are:
LNraw
- Output raw 1D performance measures using a singleSummaryLineGraph
for each measure, with all--controllers-list
controllers shown on the same graph.
If the batch criteria is bivariate, the options are:
LNraw
- Output raw performance measures as a set of :class:`~sierra.core.graphs.summary_line_graph.SummaryLineGraph`s, where each line graph is constructed from the i-th row/column for the 2D dataframe for the performance results for all controllers.Note
SIERRA cannot currently plot statistics on the linegraphs built from slices of the 2D CSVs/heatmaps generated during stage4, because statistics generation is limited to stage3. This limitation may be removed in a future release.
HMraw
- Output raw 2D performance measures as a set of dual heatmaps comparing all controllers against the controller of primary interest(one per pair).HMdiff
- Subtract the performance measure of the controller of primary interest against all other controllers, pairwise, outputting one 2D heatmap per comparison.HMscale
- Scale controller performance measures against those of the controller of primary interest by dividing, outputing one 2D heatmap per comparison.SUraw
- Output raw 3D performance measures as a single, stacked 3D surface plots comparing all controllers(identical plots, but viewed from different angles).SUscale
- Scale controller performance measures against those of the controller of primary interest by dividing. This results in a single stacked 3D surface plots comparing all controllers(identical plots, but viewed from different angles).SUdiff
- Subtract the performance measure of the controller of primary interest from each controller(including the primary). This results in a set single stacked 3D surface plots comparing all controllers(identical plots, but viewed from different angles), in which the controller of primary interest forms an(X, Y) plane at Z=0.
For all comparison types,
--controllers-legend
is used if passed for legend.Tip
Used by stage {5}; can be omitted otherwise. If omitted: N/A.
- --bc-univar
Specify that the batch criteria is univariate. This cannot be deduced from the command line
--batch-criteria
argument in all cases because we are comparing controllers across scenarios, and each scenario (potentially) has a different batch criteria definition, which will result in (potentially) erroneous comparisons if we don’t re-generate the batch criteria for each scenaro we compare controllers within.Tip
Used by stage {5}; can be omitted otherwise. If omitted: N/A.
Default: False
- --bc-bivar
Specify that the batch criteria is bivariate. This cannot be deduced from the command line
--batch-criteria
argument in all cases because we are comparing controllers across scenarios, and each scenario(potentially) has a different batch criteria definition, which will result in (potentially) erroneous comparisons if we don’t re-generate the batch criteria for each scenaro we compare controllers in .Tip
Used by stage {5}; can be omitted otherwise. If omitted: N/A.
Default: False