Pipeline: reducing a space down the lineage#
This page explains what polyphys.analyze.pipeline does with the
probe artifacts of one space, how a study describes its analysis as data,
what the post-hooks derive, where the outputs go, and what changed from the
research code it replaces.
Notation#
- segment, whole, ensemble, ensemble average
The levels of the lineage, from one chunk of one run up to the replicates of one state point collapsed into statistics. The README’s “The core idea: artifact lineage” defines them; the pipeline walks them in that order.
- property
One probe artifact without its lineage name and group, such as
gyrTMon: a name (gyrT) and a species (Mon). Its files end in-gyrTMon.npy(a whole) or.gyrTMon.npy(a segment).- relation
How the segments of a whole combine:
'tseries'concatenates along the frames,'histogram'sums,'bin_edge'keeps one copy of identical edges.- whole type
The shape of one whole:
'vector','matrix','bin_edge', or'dataframe'. It decides how the wholes stack into an ensemble and what the ensemble average looks like.- post-hook
A callable
(wholes, context) -> tuple[Derived, ...]applied to a property’s wholes before the ensembles are built. It returns every artifact to reduce.- \(r_k\)
The autocorrelation at lag \(k\) frames, from
acf().- \(\bar P(R)\)
The probability density of the separation vector of a pair of foci, in one over the length unit cubed, from
pair_separation_density().
A study’s analysis as data#
A study declares one AnalysisTable per
particle group: the group, the lineage of its probe artifacts ('segment'
when the probe phase wrote them in chunks, 'whole' when it wrote one file
per run), whether its stamp files are combined, and a tuple of
PropertySpec records. Five constructors
build the kinds the studies use:
Constructor |
Property |
Relation |
Whole type |
Ensemble average |
|---|---|---|---|---|
|
|
|
|
|
|
|
|
|
the same, with |
|
|
|
|
one array per statistic |
|
|
|
|
the edges |
|
|
|
|
one array per statistic |
The density profiles that LocalDensities
adds to a histogram, <direction>Rho<species> and
<direction>Phi<species>, also carry <property>-norm-sem in their
ensemble average: the SEM across profiles normalized separately for each
whole, not generally an uncertainty estimate for the normalized ensemble
mean, which space_profiles()
carries into the space table.
A table lists each artifact once; listing one twice is an error, and so is
naming one output twice, among the outputs the hooks declare with their
outputs method: a hook output that another entry also yields would
overwrite it. A property whose files are not found, such as a typo, raises
FileNotFoundError before any directory is made. The tables of the maintainer’s
studies live beside each study’s spec in the companion repository
polyphys-lab, so a data correction or a special case is code in
the study’s folder, never an option of the module.
from polyphys.analyze.pipeline import Autocorrelation, LocalDensities
from polyphys.spec import (
AnalysisTable, edge, histogram, histogram_2d, tseries)
ANALYSIS = (
AnalysisTable(group='bug', lineage='whole', stamps=True, properties=(
tseries('gyrT', 'Mon', post_hook=Autocorrelation(nlags=7000)),
)),
AnalysisTable(group='all', lineage='segment', stamps=False, properties=(
histogram('r', 'Crd', post_hook=LocalDensities()),
histogram_2d('xy', 'Crd'),
edge('x'),
)),
)
The post-hooks#
A hook sees a property’s wholes and a
HookContext: the study’s spec, the
group, the parser instance of each whole (which holds its sizes and
counts), the bin edges, and the injected density kernel. It returns
Derived records, each an output
property with its wholes and whole type. Without a hook, the wholes pass
through under the property’s own name.
TimeAverageAverages a per-frame array over the frames: a
(n_frames, n_sizes)cluster-size histogram time series becomes onen_sizesvector per whole. The output name is given to the hook;drop_first_columnremoves the count of empty clusters thatcount_clusters()keeps in column 0.AutocorrelationPasses a time series through and adds \(r_k\) for \(k = 0, \dots,\)
nlagsunder<property>-acf. No confidence band is stored: Bartlett’s band is a function of the stored \(r_k\), the number of frames, and the level, sobartlett_half_width()rebuilds it at any level when it is drawn.LocalDensitiesPasses a histogram through and adds
<direction>Rho<species>and<direction>Phi<species>, the local number density and volume fraction, through theDensityKernelgiven toreduce_space(). The species’ diameter is read from the attribute the spec’sspecies_sizesnames. The module imports no density model; the study’s driver supplies one, as “Building the density kernel” below shows. The two profiles’ ensemble averages carry<property>-norm-sembeside-mean,-var, and-sem.FociPairDistancesTurns the
(n_frames, n_foci, n_foci)distance-matrix time series of the foci into three DataFrame wholes with one column per pair, taggedloc<i>loc<j>genDist<d>with \(d\) the genomic distance in bonds: the histogram of each pair’s separation, \(\bar P(R)\), and the separation time series. Every pair has one separation per frame, so each histogram column sums ton_framesand \(\sum_k \bar P_k \Delta V_k = 1\) exactly, provided the grid covers every separation. The grid’s upper end comes from the system, not the data:half_contour()gives the largest separation a ring allows, half its bonds times the largest bond extension, \(R_0 = 1.5\,\sigma_{ij}\) for the FENE bond (Kremer and Grest, 1990). A separation above the grid raises and names the whole, since the chain cannot reach it. Abin_sizeorr_maxmust be a finite positive number, asresolve()checks, andn_monomersa positive integer, a Python or NumPy integer but never a float, even10.0, so it is never truncated. A constant is checked when the hook is built, and the hook raisesValueErrornaming the quantity; a function of the whole’s parser is checked on what it gives for each whole, and the error names the quantity and the whole.
Running a space#
from polyphys.analyze.pipeline import reduce_space
from polyphys_lab.trans_foci_cub.spec import ANALYSIS, SPEC
for table in ANALYSIS:
reduce_space(f'probes/ns400nl5al5ac1-{table.group}-probe', SPEC, table,
density=kernel)
kernel is the study’s density kernel, which the all table’s
LocalDensities hooks need; “Building the density kernel” below shows how to build it.
Each table reads the probe directory of its own group, here
ns400nl5al5ac1-bug-probe and ns400nl5al5ac1-all-probe, whose prefix is
the space name the parser gives. The study’s specs are the
polyphys_lab.trans_foci_cub package of
polyphys-lab, installed beside PolyPhys.
reduce_space lists the files under <input_database>/<hierarchy> (the
default hierarchy is the spec’s dir/file layout), combines the stamps if
the table asks for it, and then, one property at a time: selects the
property’s files by their last name token, builds the wholes (segments are
combined and written to wholeSim), loads and checks the edges, applies
the hook, and builds the ensembles and ensemble averages. It returns the
ensemble averages by output property and ensemble name; save=False skips
the writing.
Stage |
Directory |
File |
|---|---|---|
|
|
|
|
|
|
|
|
|
The ensemble of DataFrame wholes is their long form, every whole’s rows
with a whole and a row column, so no whole is lost before the ensemble
average; the average is taken row by row across the wholes and carries
<column>-mean, <column>-var, and <column>-sem per column.
The reduction stops with ValueError rather than reshape incomplete data:
when the segments of a whole skip a segment_id, when two files give one
whole name, when the wholes of an ensemble differ in length (a shorter
replicate is an incomplete run, not one to average over), and when a stamps
row has no parent name.
Building the density kernel#
A table with a LocalDensities hook,
such as the all table of every study in polyphys-lab, needs the density
argument of reduce_space; without it the hook raises. The argument is any
callable of the DensityKernel protocol:
it takes one whole’s histogram, summed over its frames, with its edges, the
whole’s parser instance, the direction, and the species’ diameter, and
returns the number density and the volume fraction. The study’s driver maps
the parser’s attributes onto
density_profiles():
from polyphys.analyze import density
class StudyKernel:
"""The DensityKernel of one space."""
def __init__(self, spec, n_frames):
self.geometry = spec.geometry
self.n_frames = n_frames # whole name -> frames summed into counts
def __call__(self, counts, edges, info, direction, diameter):
if self.geometry == 'cylindrical':
box = {'lcyl': info.lcyl, 'dcyl': info.dcyl}
else:
box = {'lcube': info.lcube}
out = density.density_profiles(
counts, edges, diameter, self.geometry, direction,
n_frames=self.n_frames[info.name], **box)
return out.rho, out.phi
Frames. The counts are summed over the whole’s frames, and no parser attribute holds the number of frames, so the driver supplies it per whole: the
wholeandn_framescolumns of the whole stamps thatreduce_stamps()returns for the group’s probe stamps, wheren_framesis the sum over a whole’s segments.The slab diameter. The kernel passes the parser’s full cylinder diameter
dcyl, \(D\);density_profilesitself takes \(D - a\) for the slabs along \(z\), so a uniform fluid returns the bulk volume fraction as the parsers define it. The Density page, “Conventions the kernels leave to the caller”, gives the reason.Cost.
density_profilesbuilds the share matrix on every call, about a second for 300 cylindrical shells. A driver with many wholes on the same edges can build it once per edges and diameter withshare_matrix()and apply it withnumber_density()andvolume_fraction().
Then density=StudyKernel(SPEC, n_frames) is the density argument of
“Running a space”.
The equilibrium value of each whole#
reduce_space keeps every frame of a time series. The allInOne tables
of a study need one number per whole with its error:
equilibrium_wholes() reads each ensemble
file of the ens stage once and reduces each of its columns, one whole’s
time series, to <output>-mean, the time average, <output>-var, the
sample variance of the frames, and <output>-sem, the blocked standard
error of the average from sem_blocked(),
then merges them with the whole stamps on whole, one row per whole. The
output name of each property is declared, so gyrTMon can become
gyrMon, and a whole that is in the stamps but not in a property’s files,
or the other way round, raises and is named.
import pandas as pd
from polyphys.analyze.pipeline import equilibrium_wholes
whole_stamps = pd.read_csv(
'analysis/ns400nl5al5ac1-bug-ens/ns400nl5al5ac1-bug-stamps-whole.csv')
table = equilibrium_wholes(
'analysis/ns400nl5al5ac1-bug-ens',
{'gyrTMon': 'gyrMon', 'shapeTMon': 'shapeMon'},
whole_stamps, topology='ring')
The frames of one run are correlated, so the error of a time average is
larger than the per-frame standard error by about \(\sqrt{s}\), with \(s\)
the statistical inefficiency, the number of frames that make one
independent sample. Block averaging (Flyvbjerg and Petersen, 1989) finds
that error without a model of the correlation; the 'lee' plateau rule
(Lee et al., 2011) is the default and 'sem_err' the alternative, both
as select_plateau() defines them. Blocking
applies within a whole only: the wholes of an ensemble are independent
runs, so the ensemble average keeps the plain standard error across them.
A series that is too short for its correlation time gets a RuntimeWarning
and a NaN or unreliable error; a constant series gets an error of 0.
topology is for stamps the probe phase keyed by the whole’s file name,
such as ...ens1.ring, while the ensemble files name a whole without the
suffix: it is appended to the ensemble’s names before the merge, and the
table keeps the stamps’ names.
What changed from the research code#
The module replaces the research code’s analyzer, whose nine
near-identical blocks differed only in the six facts a
PropertySpec holds. The outputs match
it except where the change was deliberate:
A property with no file raises; the research code wrote nothing and said nothing.
The wholes of one ensemble must hold identical bin edges; the research code took the first whole’s.
The autocorrelation is stored alone; the
acfLowerCiandacfUpperCifiles are gone, since the band is a function of the stored values.The time average of a per-frame histogram is named by the hook, not by dropping every
Tfrom the property’s name.The foci pair histograms share one grid up to a bound from the system; the research code set each whole’s grid from its own largest separation, so the ensemble mean mixed different sets of wholes in different bins.
pairDistRdfis now \(\bar P(R)\) rather than a unit-sum curve, and thegenDisttags count bonds, one more than the research code’s monomers between.The ensemble of DataFrame wholes keeps every whole; the research code averaged them at the ensemble step, so the ensemble average had no variance or standard error.
The error of a whole’s time average is the blocked standard error; the research code’s
measurer.semwas the per-frame one, smaller by about \(\sqrt{s}\). Its variance hasddof = 1; the research code’snp.varon a pandas column gave the population variance. The output name is declared, not made by dropping the lastT; and a whole on one side of the merge only raises, where the research code dropped or emptied the row.
References#
K. Kremer and G. S. Grest, “Dynamics of entangled linear polymer melts: A molecular-dynamics simulation”, J. Chem. Phys. 92, 5057 (1990). https://doi.org/10.1063/1.458541
M. S. Bartlett, “On the theoretical specification and sampling properties of autocorrelated time-series”, Supplement to the Journal of the Royal Statistical Society 8, 27-41 (1946). https://doi.org/10.2307/2983611
H. Flyvbjerg and H. G. Petersen, “Error estimates on averages of correlated data”, J. Chem. Phys. 91, 461-466 (1989). https://doi.org/10.1063/1.457480
R. M. Lee, G. J. Conduit, N. Nemec, P. López Ríos, and N. D. Drummond, “Strategies for improving the efficiency of quantum Monte Carlo calculations”, Phys. Rev. E 83, 066706 (2011); erratum 85, 029908 (2012). https://doi.org/10.1103/PhysRevE.83.066706