Testing#
This page describes how PolyPhys is tested: which data a test may use, the kinds of test the project writes, the shared fixtures, and the pytest settings.
Running the tests#
From the repository root, in an environment with the dev and viz extras
installed; --doctest-modules imports polyphys/viz/, which needs
matplotlib, the viz extra:
python -m pytest polyphys README.md --cov=polyphys --cov-report=term-missing --doctest-modules --doctest-glob='README.md'
The README’s examples run as doctests, and so do the Examples sections of the
docstrings in polyphys/.
CI’s Python-matrix jobs, which have no MDAnalysis, run the same command
with --cov-config=.coveragerc-no-mda, which leaves the probe layer out of
the measured set, and --cov-fail-under=95, so a pull request fails when
total coverage drops below 95 percent; the MDAnalysis job measures the probe
layer and has no floor. coverage.py rounds the total to a whole percent
before it compares, so 94.6 percent passes. Add
--cov-fail-under=95 to the command above to run the same check locally.
A CI job on Python 3.11 tests the lowest versions that pyproject.toml
allows of NumPy, pandas, SciPy, and the viz extra’s matplotlib, with
uv’s --resolution lowest-direct, and runs the
suite without MDAnalysis or coverage. To run it locally, with uv installed:
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python --resolution lowest-direct -e ".[viz]"
uv pip install --python .venv/bin/python pytest hypothesis
.venv/bin/python -m pytest polyphys README.md --doctest-modules --doctest-glob='README.md'
Test data: three tiers#
Every test uses data from one of three tiers.
Tier |
Data |
Where it lives |
Who runs it |
|---|---|---|---|
A |
Synthetic inputs a test generates itself: lineage trees of tiny files, short LAMMPS files, random or analytic arrays |
the test module or |
everyone, including CI |
B |
Small trimmed real files, such as a few dump frames or a filename listing, used where a file format is under test and, for the probes, to check results on real frames (see “Real frames of the probes”); the maintainer approves each one |
|
everyone, including CI |
C |
Full-size processed artifacts, such as a whole space |
outside the repository, under |
the maintainer, locally; never CI |
Tier A is the default. The algorithms are what is under test, and properties such as conservation, symmetry, invariance, or agreement with an analytic result do not need real data. Tier C tests compare a result against the maintainer’s processed data, for example when a module replaces code that produced published numbers.
Kinds of test#
Kind |
Catches |
Notes |
|---|---|---|
Unit test |
a function breaking its own contract |
tier A data; one behavior per test |
Known-answer test |
code that runs but computes the wrong physics |
the expected value comes from an analytic result or a cited reference |
Property test |
edge cases nobody wrote down |
checks an invariant, such as conservation of histogram counts, on hand-picked or Hypothesis-generated inputs |
Integration test |
lineage levels that pass alone but disagree at a seam |
runs a lineage tree from |
Regression test |
a fixed bug returning |
written before the fix and seen to fail against the unfixed code |
Comparison against old code |
a port that changed a number |
tier C; the old and new implementations run on the same data, and any number that changes is explained in the pull request |
Doctest |
examples drifting from behavior |
every new or materially changed public function, method, and class has an |
Static check |
type and style errors |
|
Expected values never come from the code under test. Floating-point results are
compared with a stated tolerance, using numpy.testing.assert_allclose for
arrays and pytest.approx for scalars. Uncertainties of quantities measured over
correlated molecular-dynamics frames use block averaging or another
autocorrelation-aware estimate, not a per-frame standard error.
Property tests with Hypothesis#
A property test can let Hypothesis
generate its inputs: the @given decorator draws them from strategies, and
when an input breaks the property, Hypothesis shrinks it to a minimal failing
example and reports it. For example,
test_axial_histogram_counts_every_position_in_range_once in
polyphys/tests/analyze/test_measurer.py draws a bin size, a range, and
positions within that range, and checks that each position is counted exactly
once.
Five more properties are checked on generated inputs, none against a reference value:
a parser name built from the parser’s own pattern parses to its values and rebuilds to itself, for every synthetic parser in
polyphys/tests/synthetic.py;polyphys.tests.strategies.lineage_namesdraws the names;block_averageofa x + bkeeps every blocking level, with the meana mean + b, the variancea² var, the standard error|a| sem, and the inefficiency unchanged;on a generated grid, each row of
share_matrixwhose bead stays in the domain sums to the bead volume;species profiles that add up to one constant give a normalized
sum_rule_profilesof one in every bin;a generated linear or ring chain, with or without angles, survives
write_dataandread_dataunchanged.
Hypothesis is in the dev extra. Its built-in ci settings profile is active
whenever the CI environment variable is set, as it is on GitHub Actions: CI
runs then draw the same inputs every time (derandomize=True) and keep no
example database. Local runs draw new inputs each time and store failing
examples under .hypothesis/, which git ignores, so a failure found once is
tried first on the next run. To run a test as CI does, set CI=true for that
run. See
Built-in profiles
in the Hypothesis documentation.
Real frames of the probes#
The probe tests in polyphys/tests/probe/ also run on real frames of the
four live studies, 22 wholes in all: per study, two replicates without
crowders and two at a bulk crowder volume fraction of about 0.36 to 0.38;
for the Hns studies, also the wholes without H-NS; and for HnsCyl, crowders
of size 1 as well as 2. The maintainer trimmed them from the studies’ runs.
Tier B, in
polyphys/tests/data/probe/<whole>/: every whole’s last 50 frames of the polymer group dump,<whole>.ring.<group>.last50.lammpstrj.gz; and, for the wholes without crowders and the HnsCyl wholes atac2 nc386, theallframes of the same time window,<whole>.j<NN>.ring.all.window.lammpstrj.gz, and the whole’s data file,<whole>.ring.all.data.gz.Tier C, in
<POLYPHYS_DATA_DIR>/<project>/trjs/<whole>/, with<project>the study’s project name, such asHnsCub: theallwindows and data files of the other crowded wholes, under the runs’ own names.
The HnsCyl runs name their wholes without epshc, which the HnsCyl parser
reads; the tier B copies insert epshc1, the value of their H-NS
core-crowder pair coefficients.
A polymer dump holds only the polymer group, so it is opened with the data
file of the whole without crowders of its family and replicate, whose atoms
are that group; the polymer atoms carry the IDs 1 to n in every system. The
RealWhole cases and the fixtures that open the files are in
polyphys/tests/probe/conftest.py.
The four TransFociCub wholes also have a tier C reference: the research
code’s probe of the same 50 frames of their polymer dumps, under
<POLYPHYS_DATA_DIR>/TransFociCub/probes/<space>-bug-probe/<whole>.ring/,
and its reduction of that probe, under
<POLYPHYS_DATA_DIR>/TransFociCub/analysis/<space>-bug-{ens,ensAvg,space}/.
Tier C tests compare each stage of the bug pipeline with it: the probe
tests (test_chain_size.py, test_pairs.py, test_core.py) compare the
chain-size observables, the principal axes up to sign, the foci bonds and
clusters histograms, and the stamps with the research probe;
test_pipeline.py reduces the research probe, and the package’s own probe
of the same dumps, to the research ensAvg, and builds the research space
tables from the research ensAvg; and viz/test_helpers.py draws a space
table with the helpers. The foci pair distances and the autocorrelation
functions are left out on purpose: PolyPhys takes the minimum image, which
the research code did not, and acf rejects lags beyond a 50-frame
series, which the research code’s statsmodels call clamped. The
research_probe_file fixture finds the probe files.
Tests that need LAMMPS#
A test that runs LAMMPS is marked lammps and takes the lammps_executable
fixture:
import pytest
@pytest.mark.lammps
def test_lammps_runs_from_a_built_start(lammps_executable, tmp_path):
subprocess.run([lammps_executable, "-in", "in.kg", ...], cwd=tmp_path)
The executable is the one
LMPnames, elselmp_serialonPATH.When neither exists, the test is skipped. CI has no LAMMPS.
When
LMPnames something that is not an executable, the test errors.
The input scripts live in polyphys/tests/data/lammps_build/; the tests
build their start files in tmp_path and read the log LAMMPS writes there.
Each such test takes LAMMPS itself as its oracle: the run finishes with no
error and no unexpected warning.
Tests that need local data#
A tier C test is marked private_data and takes the private_data_dir
fixture:
import pytest
@pytest.mark.private_data
def test_profiles_match_the_research_code(private_data_dir):
folder = private_data_dir / "TransFociCyl" / "golden" / "density" / "r"
...
POLYPHYS_DATA_DIR names the maintainer’s tier C folder, which holds one
folder per study, named by its project (HnsCub, HnsCyl, TransFociCub,
TransFociCyl). Each study folder holds the phase folders of
make_database, such as trjs/, probes/, and analysis/, and golden/
for test inputs derived from them by the maintainer.
When
POLYPHYS_DATA_DIRis unset, the test is skipped. CI never sets it.When it names a directory, the test runs and
private_data_diris that directory.When it names something that is not a directory, the test errors, so a misconfigured machine is not mistaken for one without data.
Keep the path out of tracked files; set the variable in your shell.
pytest settings#
pyproject.toml sets these options under [tool.pytest.ini_options]:
Setting |
Effect |
|---|---|
|
An unregistered marker, such as a misspelled |
|
A warning while reading the pytest configuration is an error |
|
A test marked |
|
The summary lists every test that did not pass, with the reason for each skip |
|
Test modules are imported without changing |
|
Loads the |
|
Runs the |
|
Runs the examples of |
|
The directory pytest collects when no path is given |
|
Every warning raised during a test is an error; a test that expects one wraps the call in |
The documented command passes polyphys README.md, so the modules’
docstrings and the README are collected as well as polyphys/tests. The examples of five pages, quickstart.md,
lineage.md, studies.md, usage.md, and viz.md, run through
polyphys/tests/test_docs.py.
See Working with custom markers and the configuration reference in the pytest documentation.
filterwarnings reaches only warnings that pytest’s own process raises
while a test runs. The resource tracker of multiprocessing reports
leaked semaphores from a process of its own after pytest has exited, so
test_multiprocessing_run_leaves_no_leaked_semaphore in
polyphys/tests/probe/test_core.py runs a parallel probe in a subprocess
and checks its standard error instead.
The documentation build replaces MDAnalysis and matplotlib with mocks, and
Python 3.13, which Read the Docs runs, evaluates annotations when a function
is defined, where Python 3.14 defers them. So
test_mocked_modules_import_with_their_annotations_evaluated in
polyphys/tests/test_docs.py imports the probe and viz modules under the
mocks of Sphinx and evaluates every annotation. Sphinx is in the dev
extra so that the test runs in every CI job that installs it.