Experimental Design

This is the landing page for designing experiments and working with data as it flows through SIERRA.

In SIERRA, there are 3 main faculties that are used to design experiments:

• The controller. In SIERRA, a --controller is simply a designation of some aspect of an algorithm which you are running. Even if what you designate as the controller is not actually controlling a significant part of the behaviors, it IS the controlling knob from the perspective of the experiment you want to run. Put another way, controllers designate the algorithm/thing under test.

• The scenario. In SIERRA, the --scenario is simply a designation of some aspect(s) of the experimental context in which the controller will run. This may be a more literal "scenario" in the sense of arena size, what it contains, etc., or it might just be a collection of features which are the inputs into your simulator, or anything in between.

  Tip

  The line between what a controller vs. scenario is intentionally blurry; SIERRA makes every effort in the source to treat them as equitably as possible, to allow you to define things with semantic labels which make sense to YOU.

• The Batch Criteria. In SIERRA, the --batch-criteria is the main workhorse of designing experiments. It defines the independent variables (you can have any number, defining an N-D experimental space). Each variable can make pretty much any changes it wants to the experimental inputs, giving you massive flexibility on how to set up your experiments. Variables are combined combinatorically by the sets of changes they make to experimental inputs to define the N-D experimental space.

Beyond this, the dependent variables are those which you select as important when you declare how you want your data to be processed, and/or what products/deliverables you want to generate. Selection is declarative; you just say something like "I'm interested in THIS column of THIS .csv file under the different conditions present throughout the different experiment, and I want the data to appear like THIS on THIS type of graph". SIERRA handles the rest!

Variables vs. Batch Criteria

Batch criteria define the axes of variation across a batch experiment — each value of a criterion generates a separate experiment. Variables, by contrast, apply uniformly to every experiment in the batch. All batch criteria are variables in the general sense, but not all variables are batch criteria.

The most commonly used non-criteria variable is experiment setup (Experiment Setup), which configures experiment duration and controller cadence uniformly across all experiments in a batch. For the full list of configurable variables and their cmdline syntax, see Configurable SIERRA Variables.

Now that you understand the basics of how to create experiments, proceed to Dataflow Across Pipeline Stages to understand how data in your experiment flows through the SIERRA pipeline.