Pseudo-Statistics#

When doing deterministic Experimental Runs, there is no benefit to running Statistics Generation over all experimental run data for an Experiment, because --n-runs=1. Thus, the statistical analysis via e.g., confidence intervals doesn't make sense. This plugin processes at the file level for each Experimental Run. All Raw Output Data files produced by each run are copied (or moved)from their original locations and written to the Runtime Directory Tree under statistics/. This is so it can be used with Graph Generation, which expects its outputs to be under that prefix. The contents of the graphs.yaml for the project is ignored, if the file exists; this may be changed in the future.

Note

This plugin is not intended for use with projects whose output is non-deterministic (i.e., contain randomness). That is, if you always use --n-runs=>1 , then you should use Statistics Generation instead of this plugin.

Ordering Considerations#

Same as proc.statistics; this plugin is a no-op drop-in for that plugin.

Usage#

This plugin can be selected by adding proc.pseudostats to the list passed to --proc. When active it will create <batchroot>/statistics, and each experiment will get their own directory in this root for their statistics. E.g.:

|-- <batchroot>
    |-- statistics
        |-- c1-exp0
        |-- c1-exp1
        |-- c1-exp2
        |-- c1-exp3

All experimental outputs are copied which are supported by the currently active storage plugin.

Cmdline Interface#

sierra - CLI interface#

sierra [--center {mean,median}] [--dataop {copy,move}]
sierra Multi-stage options#

Options which are used in multiple pipeline stages

  • --center CENTER -

    Specify the measure of central tendency: mean or median, so files can be moved/copied appropriately for downstream consumption.

    (default: mean)

  • --dataop DATAOP -

    Specify what kinds of data operation should be performed.

    • copy - Data files are copied from each run output directory to <batchroot>/statistics.

    • move - Data files are moved from each run output directory to <batchroot>/statistics.

    Unless your code generates TONS of outputs, prefer copy to move to preserve stage 3 idempotency.

    (default: copy)

Warning

If you use --dataop=move, all of your experimental data will be moved from its original output directories to <batchroot>/statistics! This will break stage 3 idempotency.

Configuration#

None for the moment.