Debugging and Logging#

This page covers SIERRA's logging system, how to diagnose common failure patterns, and techniques for isolating problems in your project or plugin code.

Log Levels#

SIERRA's verbosity is controlled by --log-level. Levels from least to most verbose:

ERROR

Only fatal errors. Use when you want to suppress all informational output and check only the exit status.

WARNING

Errors and warnings. SIERRA emits warnings for recoverable conditions such as non-existent paths on SIERRA_PLUGIN_PATH or missing optional configuration.

INFO

The default. Shows progress through pipeline stages, what plugins are loaded, and what experiments are being generated or run.

DEBUG

Shows internal decision-making: which plugins were found and why, how batch criteria are parsed, what paths are being constructed. Use this first when something is not working as expected.

TRACE

Maximum verbosity. Shows every graph being considered for generation and why it is being skipped or produced, every CSV file being processed, and every sys.path modification made by the plugin loader. Use this to diagnose missing graphs (stage 4) or CSV processing failures (stage 3).

sierra ... --log-level DEBUG
sierra ... --log-level TRACE

Tip

For stage 4 graph problems, start with TRACE — it logs exactly which output CSV stem is being matched against which graph definition stem. A mismatch between the two is the most common cause of missing graphs.

Stage Failure Patterns#

"Stage X crashed immediately"

Stage N requires stage N-1 to have completed successfully. If you skipped a stage or it failed silently, the next stage will crash on missing inputs. Check that all prior stages ran to completion before investigating further.

"SIERRA hangs during stage 3 or 4"

The most common cause is inconsistent CSV shapes: not all experimental runs produced output files with the same number of rows and columns. SIERRA relies on uniform CSV shapes for statistics generation and does not validate them by default. Pass --df-verify to catch this explicitly:

sierra ... --pipeline 3 --df-verify

If verification fails, examine which runs produced shorter outputs and why. For real robot experiments where minor timing differences are expected, --df-homogenize can pad or zero-fill short columns rather than failing:

sierra ... --pipeline 3 --df-homogenize pad

Use pad only if the filled columns represent cumulative counts or similarly stable data. For intervallic or averaged data, padding produces incorrect statistics.

"No graphs generated in stage 4"

SIERRA matches the stem of each output CSV against the src_stem field in graphs.yaml. If they don't match exactly, no graph is produced and SIERRA does not warn about it by default. Run with --log-level TRACE during stage 4 to see every match attempt:

sierra ... --pipeline 4 --log-level TRACE

Look for lines showing which src_stem values are being searched and which CSV files are present. A common cause is a trailing directory component in the output path that the graph definition doesn't account for.

"Missing graphs for some controllers but not others"

Check controllers.yaml — only the graph categories listed for a given controller are generated. A controller that is missing a category will silently produce no graphs for it.

"Stage 2 fails on HPC but works locally" Almost always a missing environment

variable on the compute nodes. Check that all required variables are forwarded via PARALLEL — see Environment Variables. Run with --log-level DEBUG to see what SIERRA is passing to GNU parallel.

Diagnosing Plugin Import Failures#

When SIERRA cannot find a plugin or project module, it reports only "cannot locate plugin X" without showing the underlying cause. This is because Python suppresses import errors when loading modules dynamically. To surface the real error, import the module directly:

python3 -m full.dotted.path.to.module

For example, if SIERRA cannot find myproject.generators.scenario:

python3 -m myproject.generators.scenario

This runs the module as a script and prints any ImportError, ModuleNotFoundError, or syntax error that would otherwise be swallowed. Common causes:

  • A transitive dependency is missing (pip install it)

  • A relative import in the module doesn't resolve (check __init__.py exists at each package level)

  • A C extension library is not on LD_LIBRARY_PATH

  • PYTHONPATH or SIERRA_PLUGIN_PATH is not set correctly for the current shell

To verify the plugin search path SIERRA is actually using, run with --log-level DEBUG and look for lines beginning with Searching for plugins in:

sierra ... --log-level DEBUG --pipeline 1 2>&1 | grep "plugins in\|sys.path\|Updated"

Isolating Problems with a Minimal Batch#

Most problems are faster to diagnose with the smallest possible batch. Use builtin.MonteCarlo.C1 to produce a single experiment and --pipeline 1 to exercise only stage 1:

sierra                                   \
  --project        myproject                 \
  --engine         engine.argos              \
  --expdef-template exp/template.argos       \
  --batch-criteria builtin.MonteCarlo.C1     \
  --scenario       myScenario.10x10x1        \
  --controller     myCategory.myController   \
  --pipeline 1                               \
  --log-level DEBUG

Stage 1 exercises plugin loading, project discovery, batch criteria parsing, and experiment generation without running any simulations. If stage 1 passes, the problem is in stage 2 or later. Add stages incrementally until the failure appears.

Once stage 1 passes with a single experiment, add --pipeline 2 with a short experiment duration (via --exp-setup if your engine supports it) to validate stage 2 before committing to a full batch run.

This same invocation doubles as an environment setup check. If stage 1 passes, SIERRA_PLUGIN_PATH, PYTHONPATH, and plugin discovery are all configured correctly for stages 2–5.