Output Marker Protocol

Steps pass data to downstream steps by printing structured markers to stdout. daggle parses these markers during execution, strips them from terminal output, and injects the values as environment variables for dependent steps.

Marker format

::daggle-output name=<key>::<value>
  • Key must match [a-zA-Z_][a-zA-Z0-9_]* (start with letter or underscore, then alphanumeric or underscore).
  • Value is everything after the second :: until the end of the line, with leading and trailing whitespace trimmed.
  • Markers are single-line only. For complex or multi-line data, write to a file and pass the file path as the output value.

Environment variable naming

Downstream steps receive outputs as environment variables following this convention:

DAGGLE_OUTPUT_<STEP_ID>_<KEY>

Both the step ID and key are uppercased. Hyphens in the step ID are replaced with underscores.

For example, step fit-model emitting accuracy becomes:

DAGGLE_OUTPUT_FIT_MODEL_ACCURACY

Parsing behavior

  • Markers are stripped from terminal output so they do not clutter the console.
  • Markers are preserved in log files (<step>.stdout.log) for debugging and auditing.
  • Only the last value for a given key is kept if a step emits the same key multiple times.

Examples

R

# Emit a single output
cat("::daggle-output name=row_count::", nrow(data), "\n")

# Emit multiple outputs
cat("::daggle-output name=mean_score::", mean(scores), "\n")
cat("::daggle-output name=data_path::", output_file, "\n")

Shell

# From a command step
echo "::daggle-output name=commit_sha::$(git rev-parse HEAD)"

# Conditional output
if [ -f "results.csv" ]; then
  echo "::daggle-output name=results_path::results.csv"
fi

Reading outputs downstream

In a YAML step definition:

- id: fit-model
  r_expr: |
    cat("::daggle-output name=accuracy::", round(acc, 4), "\n")

- id: report
  depends: [fit-model]
  script: report.R
  args: ["--accuracy", "$DAGGLE_OUTPUT_FIT_MODEL_ACCURACY"]

Or read directly in R:

accuracy <- Sys.getenv("DAGGLE_OUTPUT_FIT_MODEL_ACCURACY")

Summary markers

Steps can emit rich summaries (typically markdown) for display in dashboards and the API.

Format

::daggle-summary format=<format>::<content>
  • format: Content format identifier (e.g. markdown).
  • content: The summary content.

Examples

cat("::daggle-summary format=markdown::## Results\n\nProcessed **42** rows successfully.\n")
cat("::daggle-summary format=markdown::Model accuracy: 0.95\n")
echo "::daggle-summary format=markdown::## Deploy complete"

Behavior

  • Multiple summaries per step are concatenated with newlines.
  • Written to {step_id}.summary.md in the run directory.
  • Available via GET /api/v1/dags/{name}/runs/{run_id}/summaries.

Metadata markers

Steps can emit typed metadata entries for structured reporting.

Format

::daggle-meta type=<type> name=<name>::<value>
  • type: One of numeric, text, table, image.
  • name: Identifier matching [a-zA-Z_][a-zA-Z0-9_]*.
  • value: Interpretation depends on type:
    • numeric – A number as a string (e.g. 1542, 0.95).
    • text – Free-form text.
    • table – A JSON array of objects (e.g. [{"x":1},{"x":2}]).
    • image – A file path relative to the working directory.

Examples

cat("::daggle-meta type=numeric name=row_count::1542\n")
cat("::daggle-meta type=text name=model_desc::Linear regression with 3 predictors\n")
cat('::daggle-meta type=table name=top5::[{"name":"Alice","score":98},{"name":"Bob","score":95}]\n')
cat("::daggle-meta type=image name=residuals::output/residuals.png\n")
echo "::daggle-meta type=numeric name=elapsed_sec::34"

Behavior

  • Multiple metadata entries per step are supported.
  • Written to {step_id}.meta.json in the run directory.
  • Available via GET /api/v1/dags/{name}/runs/{run_id}/metadata.

Validation markers

Steps can emit structured validation results for data quality checks.

Format

::daggle-validation status=<status> name=<name>::<message>
  • status: One of pass, warn, fail.
  • name: Identifier matching [a-zA-Z_][a-zA-Z0-9_]*.
  • message: Human-readable description of the validation result.

Examples

cat("::daggle-validation status=pass name=row_count::Expected > 0, got 1542\n")
cat("::daggle-validation status=warn name=missing_pct::12% missing (threshold: 20%)\n")
cat("::daggle-validation status=fail name=schema::Column 'date' expected date, got character\n")
echo "::daggle-validation status=pass name=file_exists::data.csv found"

Behavior

  • Multiple validations per step are supported.
  • Written to {step_id}.validations.json in the run directory.
  • Available via GET /api/v1/dags/{name}/runs/{run_id}/validations.
  • A validation with status=fail causes the step to be treated as failed when the step’s error_on setting is "error" (the default). This applies even when the step process exits with code 0.
  • Validations with status=warn are recorded but do not affect step success.
  • Validations with status=pass are informational.

General notes

  • All markers (output, summary, metadata, validation) are stripped from terminal output but preserved in log files.
  • Marker names must start with a letter or underscore, followed by letters, digits, or underscores.
  • Markers are parsed line-by-line from stdout only (not stderr).
  • Invalid marker lines (wrong format, unknown status, etc.) are treated as regular output.