Lineage, phases, and stages#
This page explains how PolyPhys names and places every artifact of a parameter sweep: the five levels of the lineage, the phase directories the data passes through, and the stage directories inside a phase. The README’s “The core idea: artifact lineage” introduces the lineage; this page is the reference.
Notation#
- artifact
One file of the sweep: a trajectory, a log, a probe’s time series, an ensemble table, or a
spacetable.- lineage
The level of an artifact in the hierarchy
segment → whole → ensemble_long → ensemble → space, the values ofLineageT.- state point
One set of input parameters of the sweep, such as one chain length, one cylinder diameter, and one number of crowders.
- replicate
One of the independent runs of a state point, started from its own random seed; the keyword
ensnumbers them in a filename.- phase
A top-level directory of a sweep that holds one kind of data, such as trajectories or analysis tables, the values of
PhaseT.- stage
A directory inside a phase that holds the artifacts of one lineage level or one step of the reduction, the values of
StageT.- group
The particles a trajectory holds, such as
bug(the chain) orall(every particle); a parser class lists its groups.
The lineage#
Each level aggregates the one before it:
Level |
One artifact is |
Its name ends at |
|---|---|---|
|
a chunk of one run, as LAMMPS wrote it |
the segment, |
|
one run, its segments joined |
the replicate, |
|
one state point, every input parameter in the name |
the last input keyword |
|
one state point, its replicates collapsed |
the short state name |
|
the parameter sweep: the state points of one study |
the space name |
A segment is a chunk of a simulation run, not a segment of the polymer
chain, and an ensemble is a set of replicates of one state point, not a
statistical-mechanics ensemble such as NVT.
A parser, a subclass of ParserBase that a
study declares, reads an artifact’s name and
gives the name of the artifact at its own level and at every level above it,
so a file knows its replicate, its state point, and its sweep without
touching the file system:
>>> from polyphys.manage.parser import KeywordSpec, ParserBase, Species
>>> long_keys = ('N', 'r', 'lz', 'ac', 'nc', 'dt', 'bdump', 'adump')
>>> bulk = ['dmon', 'phi_bulk_m', 'rho_bulk_m', 'phi_bulk_c', 'rho_bulk_c']
>>> class ChainCyl(ParserBase):
... """A linear chain among crowders in a cylinder, periodic along z."""
... _geometry = 'cylindrical'
... _topology = 'linear'
... _groups = ['bug', 'all']
... _dump_keywords = {'bug': 'bdump', 'all': 'adump'}
... _keywords = {
... 'N': KeywordSpec('nmon', int),
... 'r': KeywordSpec('dcyl', float, scale=2.0, offset=-1.0),
... 'D': KeywordSpec('dcyl', float),
... 'lz': KeywordSpec('lcyl', float, scale=2.0),
... 'ac': KeywordSpec('dcrowd', float),
... 'nc': KeywordSpec('ncrowd', int),
... 'dt': KeywordSpec('dt', float),
... 'bdump': KeywordSpec('bdump', int),
... 'adump': KeywordSpec('adump', int),
... 'ens': KeywordSpec('ensemble_id', int),
... 'j': KeywordSpec('segment_id', int),
... }
... _patterns = {
... 'segment': (*long_keys, 'ens', 'j'),
... 'whole': (*long_keys, 'ens'),
... 'ensemble_long': long_keys,
... 'ensemble': ('N', 'D', 'ac', 'nc'),
... 'space': ('N', 'D', 'ac'),
... }
... _defaults = {'dmon': 1}
... _species = (Species('m', 'nmon', 'dmon'),
... Species('c', 'ncrowd', 'dcrowd'))
... _project_attributes = {
... 'segment': bulk, 'whole': bulk, 'ensemble_long': bulk,
... 'ensemble': ['dmon'], 'space': ['dmon']}
>>> artifact = ChainCyl(
... 'N200r10.5lz25.0ac2.0nc1000dt0.005bdump2000adump5000ens1.j03',
... lineage='segment', group='all')
>>> for level in artifact.lineage_genealogy:
... print(f'{level:13} {getattr(artifact, level)}')
segment N200r10.5lz25.0ac2.0nc1000dt0.005bdump2000adump5000ens1.j03
whole N200r10.5lz25.0ac2.0nc1000dt0.005bdump2000adump5000ens1
ensemble_long N200r10.5lz25.0ac2.0nc1000dt0.005bdump2000adump5000
ensemble N200D20ac2nc1000
space N200D20ac2
The long names, segment, whole, and ensemble_long, are cut from the
parsed name; the short names, ensemble and space, are built from the
parsed values, so r10.5 becomes the cylinder diameter D20 and ac2.0
the crowder size ac2. “Lineage names” below gives the rules that fix their
text, and the Studies page shows how a study declares which
keywords each level’s name holds.
The organizer and the pipeline use these names as join keys: the segments of
one whole share its whole name, the replicates of one state point share
their ensemble_long name, and the state points of one sweep share their
space name.
Lineage names#
A parser sets the artifact’s name at its own lineage and at each parent
lineage, as the attributes segment, whole, ensemble_long, ensemble,
and space; the organizer uses them as join keys. Two rules fix their text.
The long names,
segment,whole, andensemble_long, are cut from the parsed name, so they keep its text, and they carry no topology suffix. A segment…ens1.j03.linearand a whole…ens1.linearof one simulation both give thewholename…ens1.The short names,
ensembleandspace, are built from the parsed values. A float that is a whole number prints as an integer, and any other float prints as Python’srepr, the shortest text that reads back as the same float. Thespaceabove isN200D20ac2although the name wroteac2.0andr10.5, while a crowder size of1.5staysac1.5.
>>> artifact = ChainCyl(
... 'N200r10.5lz25ac1.5nc0dt0.005bdump2000adump5000ens1.linear', 'whole',
... 'bug')
>>> artifact.whole, artifact.space
('N200r10.5lz25ac1.5nc0dt0.005bdump2000adump5000ens1', 'N200D20ac1.5')
Names written by the research code that preceded PolyPhys can differ from
these in two ways: a whole it parsed from a whole file keeps .ring or
.linear, and a short name can print ac2.0. Parsing such a short name at
its lineage and reading the rebuilt name gives the current form.
Values a parser derives#
Geometry is derived, not just read. The cylindrical wall is a layer of beads
of size a_w = 1 whose centers lie on the cylinder of radius r set in
LAMMPS, so the cylinder size is D = 2r - a_w, stored as dcyl; lz is the
half-length, so the cylinder length is Z = 2 lz, stored as lcyl. For a
species α of size a_α, particle centers are confined to
D_eff,α = D - a_α, and its bulk number density and volume fraction both use
the effective volume V_eff,α = π Z D_eff,α² / 4:
>>> artifact = ChainCyl(
... 'N200r10.5lz25.0ac2.0nc1000dt0.005bdump2000adump5000ens1.j03',
... lineage='segment', group='all')
>>> float(artifact.dcyl) # from 'r10.5': 2 * 10.5 - 1.0
20.0
>>> round(float(artifact.phi_bulk_c), 4)
0.3292
Phases#
A sweep’s data lives under one root directory, in one subdirectory per
phase. PhaseT lists the phases in the order
the data flows through them:
Phase |
Holds |
|---|---|
|
simulation inputs |
|
LAMMPS log files |
|
LAMMPS trajectories |
|
the per-frame observables and stamps that the probe writes |
|
the reduction of the probes down the lineage, and the |
|
data prepared for figures |
|
tables that combine spaces |
|
a study’s tables of one row per whole, such as the equilibrium values that |
PolyPhys reads the files of the logs and trjs phases
(polyphys.simulate, polyphys.probe) and writes the analysis
phase (polyphys.analyze.pipeline, polyphys.manage.organizer,
polyphys.analyze.sumrule). A study’s driver decides where the probe
writes, and no function of PolyPhys writes the viz, galaxy, or
allInOne phase yet. The Workflow page follows the data from
trjs to space.
Stages#
A phase holds one directory per particle group and stage, named
<prefix>-<group>-<stage>; in the examples on these pages, <prefix> is
the space name. The probe’s output directories are the exception: a study’s
driver names them, and the examples on these pages end them in -probe,
which is not a stage. make_database()
reads only their prefix.
StageT lists the stages:
Stage |
Holds |
Written by |
|---|---|---|
|
segment artifacts |
— |
|
whole artifacts joined from their segments |
|
|
the replicates of one state point side by side, one file per property |
|
|
the mean, variance, and standard error across the replicates |
|
|
one table per property for the whole sweep |
|
|
tables that combine spaces |
— |
Three words belong to two vocabularies: segment and space are lineage
levels and stages, and galaxy is a phase and a stage, not a lineage level.
Where a word appears says which it is: as a lineage argument or the
lineage of a name it is a level; as a directory directly under the sweep’s
root, a phase; as the last field of a directory inside a phase, a stage.
make_database() builds a stage directory
from another directory of the same sweep: it keeps the parent of the old
phase directory, puts the new directory in the phase it is given, and takes
the prefix from the old directory’s name, up to its first hyphen:
>>> import tempfile
>>> from pathlib import Path
>>> from polyphys.manage.organizer import make_database
>>> tmp = tempfile.TemporaryDirectory()
>>> probe = Path(tmp.name) / 'probes' / 'N200D20ac2-bug-probe'
>>> ens_avg = make_database(probe, 'analysis', 'ensAvg', 'bug')
>>> Path(ens_avg).relative_to(tmp.name).as_posix()
'analysis/N200D20ac2-bug-ensAvg'
>>> tmp.cleanup()
The old directory must sit directly in a phase directory, so that the new
one lands at the same depth. A phase or stage outside the two vocabularies,
a group that contains -, or an old directory whose parent is not a phase
raises ValueError, so a misspelled or misplaced name fails before
anything is written.
File names inside a stage#
A file in a stage directory is named for its lineage name, its group, and
its property, joined by hyphens, as
create_fullname() builds them:
Stage |
File |
|---|---|
|
|
|
|
|
|
|
|
A property is one observable of one species, such as gyrTMon, the radius
of gyration (gyrT, a time series) of the monomers (Mon). The
Quickstart writes one file of each kind.
A file name carries the lineage name of its own level. It ends in a stage
word only where two stages hold files of one level, as ens and ensAvg
both hold ensemble_long files, or where the level is space.
reduce_stamps() writes one stamps table
into each of the three reduction stages, named
<space>-<group>-stamps-<level>.csv for the lineage level of its rows:
Stage |
Stamps table |
One row per |
|---|---|---|
|
|
segment |
|
|
whole |
|
|
state point |
The wholeSim table exists only when the probe ran on segments.
The grammar of a name#
Fields are joined by
-:<lineage name>-<group>-<property>, then the stage word, if any, then the extension.A lineage name never contains
-: a parser cuts a name at its first hyphen. A group cannot contain one either, whichmake_database()checks.A property may contain
-, which joins an observable to a measure of it, as ingyrTMon-acf. A property is therefore everything between the group and the stage word or the extension, not one hyphen-separated field.Inside a lineage name, a segment joins its
j<NN>keyword with a dot, as in...ens1.j03, as the simulation writes it, and so do a topology suffix such as.ringand the file extension. A file is grouped on its full name with the matched extension removed, never on the part before its last dot, so...epshc1.0nc0and...epshc1.0nc500stay apart.