Statistics Generation

When doing Monte Carlo simulations, or dealing with any sort of Engine or Project which contains randomness, data analysis an the ensemble level is required. This plugin supports such analysis by automatically computing statistics to e.g., enable plotting 95% confidence intervals on graph products in stage 4. It supports the following measures of central tendency and (optionally) associated measures of spread:

• mean - 95% confidence intervals, box-and-whisker calculations (IQR, CI high/low, median, etc).

• median - IQR.

This plugin processes at the file level for each Experimental Run. All Raw Output Data files produced by each run are gathered and statistics calculated, and the results written out as described in the Runtime Directory Tree.

This plugin requires that the selected storage plugin supports pl.DataFrame objects.

When run:

• Floating point numeric data is rounded to 8 decimals.

• Integer data is not rounded.

• Categorical data is "averaged" via mode(). Thus, only --spread=mean is supported for categorical data.

Note

This plugin is not intended for use with projects whose output is deterministic. That is, if you always use --n-runs=1 because your code doesn't have any randomness/produces deterministic output, then you should consider using Pseudo-Statistics instead of this plugin.

Ordering Considerations

Data Decompression should proceed this plugin in the --proc chain if you previously compressed the data.

Usage

This plugin can be selected by adding proc.statistics to the list passed to --proc. When active it will create <batchroot>/statistics, and all statistics generated during stage 3 will accrue under this root directory. Each experiment will get their own directory in this root for their statistics. E.g.:

|-- <batchroot>
    |-- statistics
        |-- c1-exp0
        |-- c1-exp1
        |-- c1-exp2
        |-- c1-exp3
        |-- exec

exec/ contains automatically statistics about SIERRA runtime w.r.t. each experiment. Useful for capturing runtime of specific experiments to better plan/schedule time on HPC clusters. Not currently in dataframe/.csv format, though that might change in the future.

Cmdline Interface

sierra - CLI interface

sierra [--center {mean,median}] [--spread {none,conf95,bw,iqr}]

sierra Multi-stage options

Options which are used in multiple pipeline stages

• --center CENTER -

  Specify the measure of central tendency: mean or median.

  (default: mean)

• --spread SPREAD -

  Specify what kinds of measures of statistical spread, if any,

  should be calculated on the distribution of experimental data:

  • none - Do not generate any additional distribution stats.

  • For --center=mean:

    • conf95 - Calculate standard deviation of experimental distribution and show 95%% confidence interval on relevant graphs w.r.t. the calculated mean.

    • bw - Calculate statistics necessary to show box and whisker plots around each mean point in supported graphs.

  • For --center=median:

    • iqr - Calculate interquartile range (IQR) of experimental

    distribution and show distribution on relevant graphs w.r.t. the calculated median.

  (default: none)

Configuration

This plugin reads graphs.yaml for intra- and inter-experiment graphs. If either are present, then it only gathers and processing data for the selected graphs. If graphs.yaml is missing or doesn't contain specs for those graph types, then all output data files are gathered and processed. This can take a looooonnngggg time, depending on the amount of data produced and the filesystem speed.

Note

A graph's src is matched against output files exactly: the value is compared against a file's path relative to the run output root (not a substring of it), so a bare name resolves at the output root and a file in a subdirectory must be named by its path (e.g. subdir/output1D). This is the same matching rule used by Intra-Experiment Data Collation, so a src like output1D gathers exactly output1D and not, say, output1D_extended or a nested subdir/output1D.