Pipeline: reducing a space down the lineage#

This page explains what polyphys.analyze.pipeline does with the probe artifacts of one space, how a study describes its analysis as data, what the post-hooks derive, where the outputs go, and what changed from the research code it replaces.

Notation#

segment, whole, ensemble, ensemble average

The levels of the lineage, from one chunk of one run up to the replicates of one state point collapsed into statistics. The README’s “The core idea: artifact lineage” defines them; the pipeline walks them in that order.

property

One probe artifact without its lineage name and group, such as gyrTMon: a name (gyrT) and a species (Mon). Its files end in -gyrTMon.npy (a whole) or .gyrTMon.npy (a segment).

relation

How the segments of a whole combine: 'tseries' concatenates along the frames, 'histogram' sums, 'bin_edge' keeps one copy of identical edges.

whole type

The shape of one whole: 'vector', 'matrix', 'bin_edge', or 'dataframe'. It decides how the wholes stack into an ensemble and what the ensemble average looks like.

post-hook

A callable (wholes, context) -> tuple[Derived, ...] applied to a property’s wholes before the ensembles are built. It returns every artifact to reduce.

\(r_k\)

The autocorrelation at lag \(k\) frames, from acf().

\(\bar P(R)\)

The probability density of the separation vector of a pair of foci, in one over the length unit cubed, from pair_separation_density().

A study’s analysis as data#

A study declares one AnalysisTable per particle group: the group, the lineage of its probe artifacts ('segment' when the probe phase wrote them in chunks, 'whole' when it wrote one file per run), whether its stamp files are combined, and a tuple of PropertySpec records. Five constructors build the kinds the studies use:

Constructor

Property

Relation

Whole type

Ensemble average

tseries('gyrT', 'Mon')

gyrTMon

'tseries'

'vector'

-mean, -var, -sem per row

histogram('r', 'Crd')

rHistCrd + edges

'histogram'

'vector'

the same, with bin_center

histogram_2d('xy', 'Crd')

xyHistCrd

'histogram'

'matrix'

one array per statistic

edge('x')

xEdge

'bin_edge'

'bin_edge'

the edges

matrix_tseries('principalT', 'Mon')

principalTMon

'tseries'

'matrix'

one array per statistic

The density profiles that LocalDensities adds to a histogram, <direction>Rho<species> and <direction>Phi<species>, also carry <property>-norm-sem in their ensemble average: the SEM across profiles normalized separately for each whole, not generally an uncertainty estimate for the normalized ensemble mean, which space_profiles() carries into the space table.

A table lists each artifact once; listing one twice is an error, and so is naming one output twice, among the outputs the hooks declare with their outputs method: a hook output that another entry also yields would overwrite it. A property whose files are not found, such as a typo, raises FileNotFoundError before any directory is made. The tables of the maintainer’s studies live beside each study’s spec in the companion repository polyphys-lab, so a data correction or a special case is code in the study’s folder, never an option of the module.

from polyphys.analyze.pipeline import Autocorrelation, LocalDensities
from polyphys.spec import (
    AnalysisTable, edge, histogram, histogram_2d, tseries)

ANALYSIS = (
    AnalysisTable(group='bug', lineage='whole', stamps=True, properties=(
        tseries('gyrT', 'Mon', post_hook=Autocorrelation(nlags=7000)),
    )),
    AnalysisTable(group='all', lineage='segment', stamps=False, properties=(
        histogram('r', 'Crd', post_hook=LocalDensities()),
        histogram_2d('xy', 'Crd'),
        edge('x'),
    )),
)

The post-hooks#

A hook sees a property’s wholes and a HookContext: the study’s spec, the group, the parser instance of each whole (which holds its sizes and counts), the bin edges, and the injected density kernel. It returns Derived records, each an output property with its wholes and whole type. Without a hook, the wholes pass through under the property’s own name.

TimeAverage

Averages a per-frame array over the frames: a (n_frames, n_sizes) cluster-size histogram time series becomes one n_sizes vector per whole. The output name is given to the hook; drop_first_column removes the count of empty clusters that count_clusters() keeps in column 0.

Autocorrelation

Passes a time series through and adds \(r_k\) for \(k = 0, \dots,\) nlags under <property>-acf. No confidence band is stored: Bartlett’s band is a function of the stored \(r_k\), the number of frames, and the level, so bartlett_half_width() rebuilds it at any level when it is drawn.

LocalDensities

Passes a histogram through and adds <direction>Rho<species> and <direction>Phi<species>, the local number density and volume fraction, through the DensityKernel given to reduce_space(). The species’ diameter is read from the attribute the spec’s species_sizes names. The module imports no density model; the study’s driver supplies one, as “Building the density kernel” below shows. The two profiles’ ensemble averages carry <property>-norm-sem beside -mean, -var, and -sem.

FociPairDistances

Turns the (n_frames, n_foci, n_foci) distance-matrix time series of the foci into three DataFrame wholes with one column per pair, tagged loc<i>loc<j>genDist<d> with \(d\) the genomic distance in bonds: the histogram of each pair’s separation, \(\bar P(R)\), and the separation time series. Every pair has one separation per frame, so each histogram column sums to n_frames and \(\sum_k \bar P_k \Delta V_k = 1\) exactly, provided the grid covers every separation. The grid’s upper end comes from the system, not the data: half_contour() gives the largest separation a ring allows, half its bonds times the largest bond extension, \(R_0 = 1.5\,\sigma_{ij}\) for the FENE bond (Kremer and Grest, 1990). A separation above the grid raises and names the whole, since the chain cannot reach it. A bin_size or r_max must be a finite positive number, as resolve() checks, and n_monomers a positive integer, a Python or NumPy integer but never a float, even 10.0, so it is never truncated. A constant is checked when the hook is built, and the hook raises ValueError naming the quantity; a function of the whole’s parser is checked on what it gives for each whole, and the error names the quantity and the whole.

Running a space#

from polyphys.analyze.pipeline import reduce_space
from polyphys_lab.trans_foci_cub.spec import ANALYSIS, SPEC

for table in ANALYSIS:
    reduce_space(f'probes/ns400nl5al5ac1-{table.group}-probe', SPEC, table,
                 density=kernel)

kernel is the study’s density kernel, which the all table’s LocalDensities hooks need; “Building the density kernel” below shows how to build it.

Each table reads the probe directory of its own group, here ns400nl5al5ac1-bug-probe and ns400nl5al5ac1-all-probe, whose prefix is the space name the parser gives. The study’s specs are the polyphys_lab.trans_foci_cub package of polyphys-lab, installed beside PolyPhys.

reduce_space lists the files under <input_database>/<hierarchy> (the default hierarchy is the spec’s dir/file layout), combines the stamps if the table asks for it, and then, one property at a time: selects the property’s files by their last name token, builds the wholes (segments are combined and written to wholeSim), loads and checks the edges, applies the hook, and builds the ensembles and ensemble averages. It returns the ensemble averages by output property and ensemble name; save=False skips the writing.

Stage

Directory

File

wholeSim

<root>/analysis/<prefix>-<group>-wholeSim

<whole>-<group>-<property>.npy (segment lineage only)

ens

<root>/analysis/<prefix>-<group>-ens

<ensemble_long>-<group>-<property>.csv or .npy

ensAvg

<root>/analysis/<prefix>-<group>-ensAvg

<ensemble_long>-<group>-<property>-ensAvg.csv, or -ensAvg-<statistic>.npy per statistic

The ensemble of DataFrame wholes is their long form, every whole’s rows with a whole and a row column, so no whole is lost before the ensemble average; the average is taken row by row across the wholes and carries <column>-mean, <column>-var, and <column>-sem per column.

The reduction stops with ValueError rather than reshape incomplete data: when the segments of a whole skip a segment_id, when two files give one whole name, when the wholes of an ensemble differ in length (a shorter replicate is an incomplete run, not one to average over), and when a stamps row has no parent name.

Building the density kernel#

A table with a LocalDensities hook, such as the all table of every study in polyphys-lab, needs the density argument of reduce_space; without it the hook raises. The argument is any callable of the DensityKernel protocol: it takes one whole’s histogram, summed over its frames, with its edges, the whole’s parser instance, the direction, and the species’ diameter, and returns the number density and the volume fraction. The study’s driver maps the parser’s attributes onto density_profiles():

from polyphys.analyze import density


class StudyKernel:
    """The DensityKernel of one space."""

    def __init__(self, spec, n_frames):
        self.geometry = spec.geometry
        self.n_frames = n_frames  # whole name -> frames summed into counts

    def __call__(self, counts, edges, info, direction, diameter):
        if self.geometry == 'cylindrical':
            box = {'lcyl': info.lcyl, 'dcyl': info.dcyl}
        else:
            box = {'lcube': info.lcube}
        out = density.density_profiles(
            counts, edges, diameter, self.geometry, direction,
            n_frames=self.n_frames[info.name], **box)
        return out.rho, out.phi
  • Frames. The counts are summed over the whole’s frames, and no parser attribute holds the number of frames, so the driver supplies it per whole: the whole and n_frames columns of the whole stamps that reduce_stamps() returns for the group’s probe stamps, where n_frames is the sum over a whole’s segments.

  • The slab diameter. The kernel passes the parser’s full cylinder diameter dcyl, \(D\); density_profiles itself takes \(D - a\) for the slabs along \(z\), so a uniform fluid returns the bulk volume fraction as the parsers define it. The Density page, “Conventions the kernels leave to the caller”, gives the reason.

  • Cost. density_profiles builds the share matrix on every call, about a second for 300 cylindrical shells. A driver with many wholes on the same edges can build it once per edges and diameter with share_matrix() and apply it with number_density() and volume_fraction().

Then density=StudyKernel(SPEC, n_frames) is the density argument of “Running a space”.

The equilibrium value of each whole#

reduce_space keeps every frame of a time series. The allInOne tables of a study need one number per whole with its error: equilibrium_wholes() reads each ensemble file of the ens stage once and reduces each of its columns, one whole’s time series, to <output>-mean, the time average, <output>-var, the sample variance of the frames, and <output>-sem, the blocked standard error of the average from sem_blocked(), then merges them with the whole stamps on whole, one row per whole. The output name of each property is declared, so gyrTMon can become gyrMon, and a whole that is in the stamps but not in a property’s files, or the other way round, raises and is named.

import pandas as pd
from polyphys.analyze.pipeline import equilibrium_wholes

whole_stamps = pd.read_csv(
    'analysis/ns400nl5al5ac1-bug-ens/ns400nl5al5ac1-bug-stamps-whole.csv')
table = equilibrium_wholes(
    'analysis/ns400nl5al5ac1-bug-ens',
    {'gyrTMon': 'gyrMon', 'shapeTMon': 'shapeMon'},
    whole_stamps, topology='ring')

The frames of one run are correlated, so the error of a time average is larger than the per-frame standard error by about \(\sqrt{s}\), with \(s\) the statistical inefficiency, the number of frames that make one independent sample. Block averaging (Flyvbjerg and Petersen, 1989) finds that error without a model of the correlation; the 'lee' plateau rule (Lee et al., 2011) is the default and 'sem_err' the alternative, both as select_plateau() defines them. Blocking applies within a whole only: the wholes of an ensemble are independent runs, so the ensemble average keeps the plain standard error across them. A series that is too short for its correlation time gets a RuntimeWarning and a NaN or unreliable error; a constant series gets an error of 0.

topology is for stamps the probe phase keyed by the whole’s file name, such as ...ens1.ring, while the ensemble files name a whole without the suffix: it is appended to the ensemble’s names before the merge, and the table keeps the stamps’ names.

What changed from the research code#

The module replaces the research code’s analyzer, whose nine near-identical blocks differed only in the six facts a PropertySpec holds. The outputs match it except where the change was deliberate:

  • A property with no file raises; the research code wrote nothing and said nothing.

  • The wholes of one ensemble must hold identical bin edges; the research code took the first whole’s.

  • The autocorrelation is stored alone; the acfLowerCi and acfUpperCi files are gone, since the band is a function of the stored values.

  • The time average of a per-frame histogram is named by the hook, not by dropping every T from the property’s name.

  • The foci pair histograms share one grid up to a bound from the system; the research code set each whole’s grid from its own largest separation, so the ensemble mean mixed different sets of wholes in different bins. pairDistRdf is now \(\bar P(R)\) rather than a unit-sum curve, and the genDist tags count bonds, one more than the research code’s monomers between.

  • The ensemble of DataFrame wholes keeps every whole; the research code averaged them at the ensemble step, so the ensemble average had no variance or standard error.

  • The error of a whole’s time average is the blocked standard error; the research code’s measurer.sem was the per-frame one, smaller by about \(\sqrt{s}\). Its variance has ddof = 1; the research code’s np.var on a pandas column gave the population variance. The output name is declared, not made by dropping the last T; and a whole on one side of the merge only raises, where the research code dropped or emptied the row.

References#

  • K. Kremer and G. S. Grest, “Dynamics of entangled linear polymer melts: A molecular-dynamics simulation”, J. Chem. Phys. 92, 5057 (1990). https://doi.org/10.1063/1.458541

  • M. S. Bartlett, “On the theoretical specification and sampling properties of autocorrelated time-series”, Supplement to the Journal of the Royal Statistical Society 8, 27-41 (1946). https://doi.org/10.2307/2983611

  • H. Flyvbjerg and H. G. Petersen, “Error estimates on averages of correlated data”, J. Chem. Phys. 91, 461-466 (1989). https://doi.org/10.1063/1.457480

  • R. M. Lee, G. J. Conduit, N. Nemec, P. López Ríos, and N. D. Drummond, “Strategies for improving the efficiency of quantum Monte Carlo calculations”, Phys. Rev. E 83, 066706 (2011); erratum 85, 029908 (2012). https://doi.org/10.1103/PhysRevE.83.066706