Skip to content

Reproducibility and validation

A reproducible HERMES product identifies six layers.

Layer Record
Code HERMES release and full Git commit
Data hermes-data version and manifest/archive SHA-256
Workflow hermes-examples commit or complete analysis script
Models every ingredient, constructor parameter, and interaction table
Numerics NSIDE, energy grid, mask, observer, cache grid, thread count, build options
Product FITS file checksum and validated header metadata

Capture an environment record

git -C /path/to/hermes rev-parse HEAD
git -C /path/to/hermes status --short
git -C /path/to/hermes-examples rev-parse HEAD
python --version
cmake --version
echo "HERMES_NUM_THREADS=${HERMES_NUM_THREADS:-auto}"

From Python:

from pathlib import Path
import hashlib
import pyhermes

version_file = Path(pyhermes.getDataPath("VERSION"))
manifest = Path(pyhermes.getDataPath("MANIFEST.json"))

print("HERMES", pyhermes.__version__)
print("module", Path(pyhermes.__file__).resolve())
print("data", version_file.read_text().strip())
print("manifest sha256", hashlib.sha256(manifest.read_bytes()).hexdigest())

Record a dirty Git status. Uncommitted changes can affect the result even when the nominal commit is known.

Validation ladder

  1. Data validation: archive checksum, manifest, expected paths, and scientific data invariants.
  2. Unit tests: local functions, boundaries, exceptions, threading, and I/O.
  3. Scientific regressions: pinned representative values from model and integration layers.
  4. Workflow smoke test: a very small end-to-end product.
  5. Convergence: resolution, cache, energy grid, and integration settings.
  6. Domain comparison: compare against an analytic limit, independent code, published figure, or observational benchmark appropriate to the analysis.

Passing a regression test does not prove a new physical configuration is correct. It shows that a carefully selected existing behavior did not change.

Changing reference values

When an intentional correction changes a pinned value:

  1. demonstrate the old behavior and the physical or numerical defect;
  2. add a focused test that fails for the defect;
  3. implement the correction;
  4. quantify changes across representative positions and energies;
  5. update reference values with an explanation and tolerance rationale;
  6. state the change in the release notes.

Never widen a tolerance or regenerate reference output solely because CI failed.

FITS product checklist

Before publication, verify that:

  • the file opens without CFITSIO/Astropy warnings;
  • PIXTYPE, ORDERING, COORDSYS, NSIDE, and pixel count agree;
  • masked pixels equal BAD_DATA and are excluded from statistics;
  • ENERGY or FREQ has the documented unit;
  • TUNIT1 matches all plotting conversions;
  • VERSION and PROCESS identify the producing code;
  • the final file SHA-256 is recorded.