Pipeline Stage 5
The main idea of this pipeline stage is to “collate” the results of one or more Summary .csv files present in different Batch Experiments into a Inter-Batch .csv file, and then use that file to generate graphs. Any Summary .csv that is present in multiple Batch Experiments can be used during stage 5! This gives this pipeline stage tremendous flexibility as a camera-ready graph generation tool.
In general, stage 5 is always run separate from stages 1-4 (i.e., a separate SIERRA invocation), because the options are quite different, but you don’t have to do this.
Important
You cannot use this stage before successfully running stage 4 for each of the Batch Experiments you want to include on the final graph.
Warning
Because SIERRA never deletes stuff for you, running stage 5 is NOT idempotent. Running the same stage 5 invocation comparing 3 controllers in a single scenario (for example) could result in linegraphs containing 3,6,9,…, lines with duplicated data. In general, you want to delete the directories generated by stage 5 between successive runs. See SIERRA Runtime Directory Tree for details on what directories are generated.
Intra-Scenario Comparison
Intra-scenario comparison compares the of experiments using one or more
controllers on the same --scenario
. To use it, you need to pass the
following options to SIERRA (see Command Line Interface for documentation):
--scenario-comparison
--bc-univar
or--bc-bivar
--dist-stats
(to get statistics generated during stage 3 to show up on the final graph).
Other --plot-*
options providing for fine-grained control of the generated
graphs may also be of interest.
For YAML configuration, see Intra-Scenario Comparison.
Inter-Scenario Comparison
Inter-scenario comparison compares the same --controller
across multiple
--scenarios
. To use it, you need to pass the following options to SIERRA
when running stage 5 (see Command Line Interface for documentation):
--controller-comparison
--bc-univar
or--bc-bivar
--dist-stats
(to get statistics generated during stage 3 to show up on the final graph).
Other --plot-*
options providing for fine-grained control of the generated
graphs may also be of interest.
For YAML configuration, see Inter-Scenario Comparison.