Graph Comparison#

Important

Only Batch Summary Data files can be used as inputs to comparison with this plugin.

This page has the following sections:

All configuration for this plugin is in <project>/config/graphs.yaml file.

OS Packages#

Same as for the prod.graphs plugin.

Usage#

This plugin can be selected by adding compare.graphs to --compare during stage 5.

Cmdline Interface#

sierra - CLI interface#

sierra [--things THINGS] [--things-legend THINGS_LEGEND]
       [--across {controllers,scenarios,criterias}] [--comparison-type {LNraw}]
       [--bc-cardinality BC_CARDINALITY]
sierra Stage 5 options#

Options for product comparison

  • --things THINGS -

    Comma separated list of things to compare within --sierra-root.

    The first things in this list will be used as the thing of primary interest if --comparison-type is passed.

    (default: None)

  • --things-legend THINGS_LEGEND -

    Comma separated list of names to use on the legend for the generated comparison graphs, specified in the same order as the --things.

    (default: None)

  • --across ACROSS -

    Perform a comparison across --things (controllers, scenarios, batch criteria), as configured.

    (default: intercc)

  • --comparison-type COMPARISON_TYPE -

    Specify how controller comparisons should be performed.

    If the batch criteria is univariate, the options are:

    • LNraw - Output raw 1D performance measures using a single Summary Line for each measure, with all --things shown on the same graph.

    If the batch criteria is bivariate, the options are:

    • LNraw - Output raw performance measures as a set of Summary Line, where each line graph is constructed from the i-th row/column for the 2D dataframe for the performance results for all --things.

    Note

    SIERRA cannot currently plot statistics on the linegraphs built from slices of the 2D CSVs/heatmaps generated during stage4, because statistics generation is limited to stage3. This limitation may be removed in a future release.

    For all comparison types, --legend is used if passed for legend.

    (default: None)

  • --bc-cardinality BC_CARDINALITY -

    Specify the cardinality of the batch criteria used. It is not trivial to deduce this correctly for all --across / --comparison-type combinations, so for now the solution is to require that this be passed.

    (default: None)

Inter-Controller Comparison#

Inter-controller comparison compares the results of multiple controllers on the same --scenario. Any collated CSV/graph can be used as a comparison graph! This includes any additional CSVs that a project creates on its own/by extending SIERRA via hooks.

When active, this comparison type will create the following directory tree. For the purposes of explanation, I will use the following partial SIERRA option sets to explain the additions to the experiment tree for stage 5:

--pipeline 5 \
--controller-comparison \
--batch-criteria population_size.Log8 \
--controllers-list d0.CRW,d0.DPO \
--sierra-root=$HOME/exp"

This invocation will cause SIERRA to create the following directory structure as it runs:

$HOME/exp
   |-- d0.CRW+d0.DPO-cc-csvs/
   |-- d0.CRW+d0.DPO-cc-graphs/

d0.CRW+d0.DPO-cc-graphs/ is the directory holding the comparison graphs for each scenario for which d0.CRW and d0.DPO were run (scenarios are computed by examining the directory tree for stages 1-4). Controller names are arbitrary for the purposes of stage 5 and entirely depend on the project). d0.CRW+d0.DPO-cc-csvs/ are the files used to create the graphs.

Graph YAML Config#

Unless stated otherwise, all keys are required.

# Inter-controller comparison: For a set of controllers which have all been
# run in the same scenario.
inter-controller:
  # The filename (no path, extension) of the .csv within the collated .csv
  # output directory for each batch experiment which contains the information
  # to collate across controllers/scenarios.
  #
  # The src_stem must match the dest_stem from an inter-experiment line graph
  # in order to generate the comparison graph!
  #
  # Note that if you are using bivariate batch criteria + intra-scenario
  # comparison, you *may* have to append the interval # to the end of the
  # stem, because collated 2D CSV files form a temporal sequence over the
  # duration of the experiment. If you forget to do this, you will get a
  # warning and no graph will be generated, because SIERRA won't know which
  # 2D csv you want to us as source. Frequently you are interested in steady
  # state behavior, so putting 'n_intervals - 1' is desired.
  #
  # If the src_stem is from a CSV you generated outside of the SIERRA
  # core/via a hook, then this restriction does not apply.
  - src_stem: PM-ss-raw

    # The filename (no path, extent) of the .csv file within the
    # controller/scenario comparison directory in ``--sierra-root`` (outside
    # of the directories for each controller/scenario!) which should contain
    # the data collated from each batch experiment. I usually put a prefix
    # such as ``cc`` (controller comparison) to help distinguish these graphs
    # from the collated graphs in stage 4.
    dest_stem: cc-PM-ss-raw

    # The title the graph should have. This cannot be computed from the batch
    # criteria in general in stage 5, because you can comparison results
    # across scenarios which use different batch criteria. Defaults to '' if
    # omitted.
    title: ''

    # The Y or Z label of the graph (depending on the type of comparison
    # graph selected on the cmdline). This key is optional.
    label: 'Avg. Object Collection Rate'

    # For bivariate batch criteria, select which criteria should be on the X
    # axis if the comparison graphs are linegraphs. 0=criteria1,
    # 1=criteria2. Ignored for univariate batch criteria or if heatmaps are
    # selected for as the comparison type. This key is optional, and
    # defaults to 0 if omitted.
    primary_axis: 0

   # The experiments from each batch experiment which should be included on
   # the comparison graph. This is useful to exclude exp0 (for example), if
   # you are interested in behavior under non-ideal conditions and exp0
   # contains behavior under ideal conditions as the "base case" in the
   # batch. Syntax is parsed as a python slice, so as ``X:Y`` as you would
   # expect. This key is optional, and defaults to ``:`` if omitted.
    include_exp: '2:'

    # Specify the backend to use to generate the graph. Defaults to
    # --graphs-backend if omitted.
    backend: "matplotlib"

Inter-Scenario Comparison#

Inter-scenario comparison compares the same --controller across multiple scenarios. Only supports univariate batch criteria. Any collated CSV/graph can be used as a comparison graph! This includes any additional CSVs that a project creates on its own/by extending SIERRA via hooks.

When active, this comparison type will create the following directory tree. For the purposes of explanation, I will use the following partial SIERRA option sets to explain the additions to the experiment tree for stage 5:

--pipeline 5 \
--across=scenarios \
--batch-criteria population_size.Log8 \
--scenarios-list=RN.16x16x2,PL.16x16x2 \
--sierra-root=$HOME/exp"

This invocation will cause SIERRA to create the following directory structure as it runs:

$HOME/exp/
   |-- RN.16x16x2+PL.16x16x2-sc-graphs/
   |-- RN.16x16x2+PL.16x16x2-sc-csvs/

RN.16x16x2+PL.16x16x2-sc-graphs/ is the directory holding the comparison graphs for all controllers which were previously run on the scenarios RN.16x16x2 and PL.16x16x2 (scenario names are arbitrary for the purposes of stage 5 and entirely depend on the project). RN.16x16x2+PL.16x16x2-sc-csvs/ are the Inter-Batch Data files used to create the graphs.

Graph YAML Config#

Same syntax and meaning as the configuration for inter-controller comparison graphs.

# Inter-controller comparison: For a set of controllers which have all been
# run in the same scenario.
inter-controller:
  # The filename (no path, extension) of the .csv within the collated .csv
  # output directory for each batch experiment which contains the information
  # to collate across controllers/scenarios.
  #
  # The src_stem must match the dest_stem from an inter-experiment line graph
  # in order to generate the comparison graph!
  #
  # Note that if you are using bivariate batch criteria + intra-scenario
  # comparison, you *may* have to append the interval # to the end of the
  # stem, because collated 2D CSV files form a temporal sequence over the
  # duration of the experiment. If you forget to do this, you will get a
  # warning and no graph will be generated, because SIERRA won't know which
  # 2D csv you want to us as source. Frequently you are interested in steady
  # state behavior, so putting 'n_intervals - 1' is desired.
  #
  # If the src_stem is from a CSV you generated outside of the SIERRA
  # core/via a hook, then this restriction does not apply.
  - src_stem: PM-ss-raw

    # The filename (no path, extent) of the .csv file within the
    # controller/scenario comparison directory in ``--sierra-root`` (outside
    # of the directories for each controller/scenario!) which should contain
    # the data collated from each batch experiment. I usually put a prefix
    # such as ``cc`` (controller comparison) to help distinguish these graphs
    # from the collated graphs in stage 4.
    dest_stem: cc-PM-ss-raw

    # The title the graph should have. This cannot be computed from the batch
    # criteria in general in stage 5, because you can comparison results
    # across scenarios which use different batch criteria. Defaults to '' if
    # omitted.
    title: ''

    # The Y or Z label of the graph (depending on the type of comparison
    # graph selected on the cmdline). This key is optional.
    label: 'Avg. Object Collection Rate'

    # For bivariate batch criteria, select which criteria should be on the X
    # axis if the comparison graphs are linegraphs. 0=criteria1,
    # 1=criteria2. Ignored for univariate batch criteria or if heatmaps are
    # selected for as the comparison type. This key is optional, and
    # defaults to 0 if omitted.
    primary_axis: 0

   # The experiments from each batch experiment which should be included on
   # the comparison graph. This is useful to exclude exp0 (for example), if
   # you are interested in behavior under non-ideal conditions and exp0
   # contains behavior under ideal conditions as the "base case" in the
   # batch. Syntax is parsed as a python slice, so as ``X:Y`` as you would
   # expect. This key is optional, and defaults to ``:`` if omitted.
    include_exp: '2:'

    # Specify the backend to use to generate the graph. Defaults to
    # --graphs-backend if omitted.
    backend: "matplotlib"