Graph Comparison#

Important

Only Batch Summary Data files can be used as inputs to comparison with this plugin.

This page has the following sections:

Inter-Controller Comparison: How to generate comparison graphs for a set of controllers which have all been run on a single scenario and Batch Criteria.
Inter-Scenario Comparison: How to generate comparison graphs for a set of scenarios which have all been run using a single controller and Batch Criteria.

All configuration for this plugin is in <project>/config/graphs.yaml file.

This plugin works with inter-experiment Batch Summary Data files from stage4. Currently, it only works with files targeting inter-experiment stacked_line graphs. That means, the src field of configuration items for this plugin must correspond to the dest field of a stacked_line inter-experiment linegraph which has been previously generated.

This plugin performs stage-5 Data Collation: it takes the per-batch Collated Output Data for each compared thing (controller or scenario) and places them side by side -- one column per compared thing, indexed by experiment -- in a single Inter-Batch Data file per measure. For a visualization of this reshape, see Stage 5 Inter-Batch Dataflow.

OS Packages#

Same as for the prod.graphs plugin.

Usage#

This plugin can be selected by adding compare.graphs to --compare during stage 5.

Cmdline Interface#

sierra - CLI interface#

sierra [--things THINGS] [--across {controllers,scenarios,criterias}]
       [--bc-cardinality BC_CARDINALITY]

sierra Stage 5 options#

Options for product comparison

--things THINGS -
Comma separated list of things to compare within --sierra-root.

The first things in this list will be used as the thing of primary interest if --comparison-type is passed.
(default: None)
--across ACROSS -
Perform a comparison across --things (controllers, scenarios, batch criteria), as configured.
(default: controllers)
--bc-cardinality BC_CARDINALITY -
Specify the cardinality of the batch criteria used. It is much easier to specify this here rather than try to deduce this before creating the batch criteria for each scenario/controller to compare for all --across combinations.
(default: None)

Model Overlay#

If models were run during stage 4 (i.e., the proc.modelrunner processor was included in --proc), and an inter-experiment model targets the same src as a comparison graph, then the model's predictions are collated alongside the empirical data and overlaid on the comparison graph as an additional line per compared thing. This works for both inter-controller and inter-scenario comparison, and for univariate batch criteria.

The collated predictions and their legends are written to the -cc-models / -sc-models directory (see the trees below). No extra configuration is required beyond having run the models in stage 4: the overlay is keyed off the comparison graph's src, so a model whose target matches that path is picked up automatically. If no matching model output exists, the comparison graph is generated normally without an overlay.

Note

Model overlay is currently supported for univariate batch criteria only. For bivariate batch criteria the comparison graphs are generated without a model overlay even if models were run.

Experiment Selection (`include_exp`)#

The include_exp key (see the YAML config below) selects which experiments from each batch are included on a comparison graph, as a python-style slice. The selection is applied consistently to everything on the graph: the collated data, any overlaid model predictions, and the graph's X-axis ticks are all filtered identically. As a result the collated .csv, the collated .model (if any), and the plotted axes always have the same set of experiments, for both inter-controller and inter-scenario comparison.

For example, include_exp: '1:' on a batch of three experiments drops the first experiment, leaving two data rows, two model rows (if a model was overlaid), and two X-axis ticks.

Inter-Controller Comparison#

Inter-controller comparison compares the results of multiple controllers on the same --scenario. Any collated CSV/graph can be used as a comparison graph! This includes any additional CSVs that a project creates on its own/by extending SIERRA via hooks.

When active, this comparison type will create the following directory tree. For the purposes of explanation, I will use the following partial SIERRA option sets to explain the additions to the experiment tree for stage 5:

--pipeline 5 \
--across=controllers \
--batch-criteria population_size.Log8 \
--things=d0.CRW,d0.DPO \
--sierra-root=$HOME/exp

This invocation will cause SIERRA to create the following directory structure as it runs:

$HOME/exp
   |-- d0.CRW+d0.DPO-cc-csvs/
   |-- d0.CRW+d0.DPO-cc-graphs/
   |-- d0.CRW+d0.DPO-cc-models/

d0.CRW+d0.DPO-cc-graphs/ is the directory holding the comparison graphs for each scenario for which d0.CRW and d0.DPO were run (scenarios are computed by examining the directory tree for stages 1-4). Controller names are arbitrary for the purposes of stage 5 and entirely depend on the project). d0.CRW+d0.DPO-cc-csvs/ are the files used to create the graphs. d0.CRW+d0.DPO-cc-models/ holds collated model predictions, and is populated only when stage-4 inter-experiment models were run for the compared measures. When present, these predictions are also overlaid on the corresponding comparison graphs (see Model Overlay).

Graph YAML Config#

Comparison graphs live under the inter-controller key in <project>/config/graphs.yaml. Unlike the intra-exp/inter-exp sections used by the prod.graphs plugin, this section is a flat list of graphs with no category level, because the things being compared are named directly on the cmdline rather than being enabled/disabled via controller YAML.

src, dest and type are required; everything else is optional and defaults as noted below.

Important

Comparison graphs use type: comparison_line, which is not the same as the summary_line type used by prod.graphs, despite both rendering as lines. A summary_line plots one point per Experiment within a single batch; a comparison_line plots one line per controller (or scenario) across batches. The two accept nearly disjoint key sets, and are validated by different schemas.

Changed in version 1.5.12: Renamed from summary_line, which collided with the prod.graphs type of the same name. Existing configs must be updated.

# Inter-controller comparison: For a set of controllers which have all been
# run in the same scenario.
#
# Inter-scenario comparison uses an identical key set; only the section name
# differs (use 'inter-scenario' instead of 'inter-controller').
#
# Note that unlike the intra-exp/inter-exp sections used by the prod.graphs
# plugin, comparison graphs are a FLAT LIST: there is no category level, since
# the things being compared are named directly on the cmdline rather than
# enabled/disabled via controller YAML.
inter-controller:
  # The filename (no path, extension) of the .csv within the collated .csv
  # output directory for each batch experiment which contains the information
  # to collate across controllers/scenarios.
  #
  # The src_stem must match the dest_stem from an inter-experiment line graph
  # in order to generate the comparison graph!
  #
  # Note that if you are using bivariate batch criteria + intra-scenario
  # comparison, you *may* have to append the interval # to the end of the
  # stem, because collated 2D CSV files form a temporal sequence over the
  # duration of the experiment. If you forget to do this, you will get a
  # warning and no graph will be generated, because SIERRA won't know which
  # 2D csv you want to us as source. Frequently you are interested in steady
  # state behavior, so putting 'n_intervals - 1' is desired.
  #
  # If the src_stem is from a CSV you generated outside of the SIERRA
  # core/via a hook, then this restriction does not apply.
  - src_stem: PM-ss-raw

    # The filename (no path, extent) of the .csv file within the
    # controller/scenario comparison directory in ``--sierra-root`` (outside
    # of the directories for each controller/scenario!) which should contain
    # the data collated from each batch experiment. I usually put a prefix
    # such as ``cc`` (controller comparison) to help distinguish these graphs
    # from the collated graphs in stage 4.
    dest_stem: cc-PM-ss-raw

    # The type of graph. Must be 'comparison_line'.
    #
    # NOTE: this is NOT the same as the 'summary_line' type used by the
    # prod.graphs plugin, despite both rendering as lines. A summary_line
    # plots one point per EXPERIMENT within a single batch; a comparison_line
    # plots one line per CONTROLLER (or scenario) across batches. The two
    # accept nearly disjoint key sets.
    type: 'comparison_line'

    # The title the graph should have. This cannot be computed from the batch
    # criteria in general in stage 5, because you can comparison results
    # across scenarios which use different batch criteria. Defaults to '' if
    # omitted.
    title: ''

    # The Y label of the graph. Optional; defaults to '' if omitted.
    label: 'Avg. Object Collection Rate'

    # For bivariate batch criteria, select which criteria should be on the X
    # axis of the comparison linegraphs. 0=criteria1, 1=criteria2. Ignored for
    # univariate batch criteria. This key is optional, and defaults to 0 if
    # omitted.
    primary_axis: 0

    # Which point in the time series from each batch experiment to use as the
    # representative value for that experiment. Negative values index from the
    # end, as in python. Optional; defaults to -1 (the final point, i.e.
    # steady-state behavior) if omitted.
    index: -1

    # The experiments from each batch experiment which should be included on
    # the comparison graph. This is useful to exclude exp0 (for example), if
    # you are interested in behavior under non-ideal conditions and exp0
    # contains behavior under ideal conditions as the "base case" in the
    # batch. Syntax is parsed as a python slice, so ``X:Y``, ``X:`` and ``:Y``
    # all work as you would expect. This key is optional, and defaults to ``:``
    # (all experiments) if omitted.
    #
    # The selection is applied consistently to the collated data, any overlaid
    # model predictions, and the graph's X-axis ticks, so all three always
    # cover the same set of experiments. This holds for both inter-controller
    # and inter-scenario comparison.
    include_exp: '2:'

    # Specify the backend to use to generate the graph. Defaults to
    # --graphs-backend if omitted.
    backend: "matplotlib"

Inter-Scenario Comparison#

Inter-scenario comparison compares the same --controller across multiple scenarios. Only supports univariate batch criteria. Any collated CSV/graph can be used as a comparison graph! This includes any additional CSVs that a project creates on its own/by extending SIERRA via hooks.

When active, this comparison type will create the following directory tree. For the purposes of explanation, I will use the following partial SIERRA option sets to explain the additions to the experiment tree for stage 5:

--pipeline 5 \
--across=scenarios \
--batch-criteria population_size.Log8 \
--controller=d0.DPO \
--things=RN.16x16x2,PL.16x16x2 \
--sierra-root=$HOME/exp

This invocation will cause SIERRA to create the following directory structure as it runs:

$HOME/exp/
   |-- RN.16x16x2+PL.16x16x2-sc-graphs/
   |-- RN.16x16x2+PL.16x16x2-sc-csvs/
   |-- RN.16x16x2+PL.16x16x2-sc-models/

RN.16x16x2+PL.16x16x2-sc-graphs/ is the directory holding the comparison graphs for the single --controller (here d0.DPO) which was previously run on the scenarios RN.16x16x2 and PL.16x16x2 (scenario names are arbitrary for the purposes of stage 5 and entirely depend on the project). RN.16x16x2+PL.16x16x2-sc-csvs/ are the Inter-Batch Data files used to create the graphs. RN.16x16x2+PL.16x16x2-sc-models/ holds collated model predictions, and is populated only when stage-4 inter-experiment models were run for the compared measures. When present, these predictions are also overlaid on the corresponding comparison graphs (see Model Overlay).

Graph YAML Config#

Same syntax and meaning as the configuration for inter-controller comparison graphs, but under the inter-scenario key rather than inter-controller. The example below shows the inter-controller form; substitute the section name for inter-scenario comparison.

# Inter-controller comparison: For a set of controllers which have all been
# run in the same scenario.
#
# Inter-scenario comparison uses an identical key set; only the section name
# differs (use 'inter-scenario' instead of 'inter-controller').
#
# Note that unlike the intra-exp/inter-exp sections used by the prod.graphs
# plugin, comparison graphs are a FLAT LIST: there is no category level, since
# the things being compared are named directly on the cmdline rather than
# enabled/disabled via controller YAML.
inter-controller:
  # The filename (no path, extension) of the .csv within the collated .csv
  # output directory for each batch experiment which contains the information
  # to collate across controllers/scenarios.
  #
  # The src_stem must match the dest_stem from an inter-experiment line graph
  # in order to generate the comparison graph!
  #
  # Note that if you are using bivariate batch criteria + intra-scenario
  # comparison, you *may* have to append the interval # to the end of the
  # stem, because collated 2D CSV files form a temporal sequence over the
  # duration of the experiment. If you forget to do this, you will get a
  # warning and no graph will be generated, because SIERRA won't know which
  # 2D csv you want to us as source. Frequently you are interested in steady
  # state behavior, so putting 'n_intervals - 1' is desired.
  #
  # If the src_stem is from a CSV you generated outside of the SIERRA
  # core/via a hook, then this restriction does not apply.
  - src_stem: PM-ss-raw

    # The filename (no path, extent) of the .csv file within the
    # controller/scenario comparison directory in ``--sierra-root`` (outside
    # of the directories for each controller/scenario!) which should contain
    # the data collated from each batch experiment. I usually put a prefix
    # such as ``cc`` (controller comparison) to help distinguish these graphs
    # from the collated graphs in stage 4.
    dest_stem: cc-PM-ss-raw

    # The type of graph. Must be 'comparison_line'.
    #
    # NOTE: this is NOT the same as the 'summary_line' type used by the
    # prod.graphs plugin, despite both rendering as lines. A summary_line
    # plots one point per EXPERIMENT within a single batch; a comparison_line
    # plots one line per CONTROLLER (or scenario) across batches. The two
    # accept nearly disjoint key sets.
    type: 'comparison_line'

    # The title the graph should have. This cannot be computed from the batch
    # criteria in general in stage 5, because you can comparison results
    # across scenarios which use different batch criteria. Defaults to '' if
    # omitted.
    title: ''

    # The Y label of the graph. Optional; defaults to '' if omitted.
    label: 'Avg. Object Collection Rate'

    # For bivariate batch criteria, select which criteria should be on the X
    # axis of the comparison linegraphs. 0=criteria1, 1=criteria2. Ignored for
    # univariate batch criteria. This key is optional, and defaults to 0 if
    # omitted.
    primary_axis: 0

    # Which point in the time series from each batch experiment to use as the
    # representative value for that experiment. Negative values index from the
    # end, as in python. Optional; defaults to -1 (the final point, i.e.
    # steady-state behavior) if omitted.
    index: -1

    # The experiments from each batch experiment which should be included on
    # the comparison graph. This is useful to exclude exp0 (for example), if
    # you are interested in behavior under non-ideal conditions and exp0
    # contains behavior under ideal conditions as the "base case" in the
    # batch. Syntax is parsed as a python slice, so ``X:Y``, ``X:`` and ``:Y``
    # all work as you would expect. This key is optional, and defaults to ``:``
    # (all experiments) if omitted.
    #
    # The selection is applied consistently to the collated data, any overlaid
    # model predictions, and the graph's X-axis ticks, so all three always
    # cover the same set of experiments. This holds for both inter-controller
    # and inter-scenario comparison.
    include_exp: '2:'

    # Specify the backend to use to generate the graph. Defaults to
    # --graphs-backend if omitted.
    backend: "matplotlib"

Graph Comparison#

OS Packages#

Usage#

Cmdline Interface#

sierra - CLI interface#

sierra Stage 5 options#

Model Overlay#

Experiment Selection (include_exp)#

Inter-Controller Comparison#

Graph YAML Config#

Inter-Scenario Comparison#

Graph YAML Config#

Experiment Selection (`include_exp`)#