Pipeline Stage 5

The main idea of this pipeline stage is to “collate” the results of one or more Summary .csv files present in different Batch Experiments into a Inter-Batch .csv file, and then use that file to generate graphs. Any Summary .csv that is present in multiple Batch Experiments can be used during stage 5! This gives this pipeline stage tremendous flexibility as a camera-ready graph generation tool.

In general, stage 5 is always run separate from stages 1-4 (i.e., a separate SIERRA invocation), because the options are quite different, but you don’t have to do this.

Important

You cannot use this stage before successfully running stage 4 for each of the Batch Experiments you want to include on the final graph.

Warning

Because SIERRA never deletes stuff for you, running stage 5 is NOT idempotent. Running the same stage 5 invocation comparing 3 controllers in a single scenario (for example) could result in linegraphs containing 3,6,9,…, lines with duplicated data. In general, you want to delete the directories generated by stage 5 between successive runs. See SIERRA Runtime Directory Tree for details on what directories are generated.

Intra-Scenario Comparison

Intra-scenario comparison compares the of experiments using one or more controllers on the same --scenario. To use it, you need to pass the following options to SIERRA (see Command Line Interface for documentation):

  • --scenario-comparison

  • --bc-univar or --bc-bivar

  • --dist-stats (to get statistics generated during stage 3 to show up on the final graph).

Other --plot-* options providing for fine-grained control of the generated graphs may also be of interest.

For YAML configuration, see Intra-Scenario Comparison.

Inter-Scenario Comparison

Inter-scenario comparison compares the same --controller across multiple --scenarios. To use it, you need to pass the following options to SIERRA when running stage 5 (see Command Line Interface for documentation):

  • --controller-comparison

  • --bc-univar or --bc-bivar

  • --dist-stats (to get statistics generated during stage 3 to show up on the final graph).

Other --plot-* options providing for fine-grained control of the generated graphs may also be of interest.

For YAML configuration, see Inter-Scenario Comparison.