polyphys.analyze.pipeline#

polyphys.analyze.pipeline#

The polyphys.analyze.pipeline module reduces the probe artifacts of one space down the lineage, segment to whole to ensemble to ensemble average, for every property that a study’s analysis table declares. It holds no physics of its own: it selects the files of each property, calls polyphys.manage.organizer to build the wholes, the ensembles, and the ensemble averages, and lets a post-hook derive further artifacts from the wholes, such as a time average, an autocorrelation function, a density profile, or the pair-separation histograms of the foci.

A study describes its analysis as data: one AnalysisTable per particle group, listing a PropertySpec per probe artifact; both records are in polyphys.spec. The tables of the maintainer’s studies live beside each study’s ProjectSpec in the companion repository polyphys-lab, so a data correction or a special case is code in the study’s folder, never an option of this module.

Notation#

property

The name of one probe artifact without its lineage name and group, such as gyrTMon or rHistCrd: a name (gyrT, rHist) followed by a species (Mon, Crd). Its files end in -<property>.npy (a whole) or .<property>.npy (a segment).

relation

How the segments of a whole combine, as whole_from_segments() defines it: 'tseries' concatenates along the frames, 'histogram' sums, and 'bin_edge' keeps one copy of identical edges.

whole type

The shape of one whole, as ensemble() defines it: 'vector' (a 1-D array), 'matrix' (an N-D array, stacked along a new first axis), 'bin_edge' (edges shared by the wholes), or 'dataframe' (a table whose columns are elements, such as the pairs of foci). The ensemble average of a vector is a DataFrame of -mean, -var, and -sem columns; of a matrix, one array per statistic; of edges, the edges; of a DataFrame, a DataFrame with those three statistics per column.

post-hook

A callable (wholes, context) -> tuple[Derived, ...] applied to a property’s wholes before the ensembles are built. It returns every artifact to reduce, so a hook can pass the wholes through under the property’s own name, add derived artifacts, or replace the wholes with a reduction of them.

lineage of a table

'segment' when the probe phase wrote the group’s artifacts in segments, 'whole' when it wrote one file per whole. A property of the study and the group, not of this module.

\(r_k\)

The autocorrelation at lag \(k\) frames, from acf().

\(\bar P(R)\)

The probability density of the separation vector of a pair of foci, in one over the length unit cubed, from pair_separation_density().

Contents#

Records: Derived is one artifact a hook returns, and HookContext what a hook may read. The table records, PropertySpec and AnalysisTable, and their builders are in polyphys.spec.

Hooks: TimeAverage averages a time series over its frames; Autocorrelation adds the autocorrelation function of a series; LocalDensities adds the number density and volume fraction profiles of a histogram through an injected DensityKernel; FociPairDistances turns the distance-matrix time series of the foci into their pair histograms, separation densities, and separation time series. half_contour() is the separation bound of a ring at full bond extension.

Drivers: reduce_space() runs every property of a table over one space; space_tseries_from_spec() and space_hists_from_spec() stack the ensemble averages of one property into the space table with the spec’s columns and rounding step; reduce_stamps() combines the stamps of a space; equilibrium_wholes() reduces each whole’s time series to its equilibrium value with a blocked error, beside the whole’s stamps.

Conventions#

  • A property whose files are not found raises FileNotFoundError before any directory is made or any property reduced; nothing is skipped in silence.

  • No output overwrites another: AnalysisTable rejects a repeated name among the outputs its hooks declare, and reduce_space() raises ValueError when a hook that declares none returns a name already produced.

  • The wholes of one ensemble must hold identical bin edges, or ValueError names the ensemble and the two wholes; the organizer takes the first whole’s edges without comparing them.

  • Outputs go to <root>/analysis/<prefix>-<group>-<stage> for the stages wholeSim (the segment lineage only), ens, and ensAvg, as make_database() builds them, with the names save_artifact() gives.

  • The ensemble of 'dataframe' wholes is their long form, every whole’s rows with a whole and a row column, so no whole is lost before the ensemble average; the average is taken row by row across the wholes.

  • The error of a whole’s time average is the blocked standard error of sem_blocked(), since the frames of one run are correlated; across the wholes of an ensemble, independent runs, the organizer’s plain standard error stays.

  • A density profile’s ensemble average also carries <property>-norm-sem, the standard error over the wholes of each whole’s profile divided by its own value in the bin that the sum rule’s default normalizer names (PROFILE_NORMALIZERS): the SEM across profiles normalized separately for each whole. Normalization is nonlinear, so it is not generally an uncertainty estimate for the normalized ensemble mean, and neither is the ensemble average’s -sem. See Derived.

  • Progress is logged at INFO level; no warning is raised for a renamed artifact, because every output name is declared.

  • Complexities are stated in n_files (the files of one property), n_frames, n_bins, n_wholes, and n_pairs.

Dependencies#

References

  • K. Kremer and G. S. Grest, “Dynamics of entangled linear polymer melts: A molecular-dynamics simulation”, J. Chem. Phys. 92, 5057 (1990), for the FENE bond and its maximum extension \(R_0 = 1.5\,\sigma\). https://doi.org/10.1063/1.458541

Examples

A table that reads as data: the gyration radius time series with its autocorrelation, the radial histogram of the crowders with their density profiles, and the planar histograms with their edges:

>>> from polyphys.analyze.pipeline import Autocorrelation, LocalDensities
>>> from polyphys.spec import (
...     AnalysisTable, edge, histogram, histogram_2d, tseries)
>>> table = AnalysisTable(
...     group='all', lineage='segment', stamps=False, properties=(
...         tseries('gyrT', 'Mon', post_hook=Autocorrelation(nlags=100)),
...         histogram('r', 'Crd', post_hook=LocalDensities()),
...         histogram_2d('xy', 'Crd'),
...         edge('x'),
...     ))
>>> [prop.prop for prop in table.properties]
['gyrTMon', 'rHistCrd', 'xyHistCrd', 'xEdge']

Added in version 0.6.0.

Changed in version 0.9.0: AnalysisTable, PropertySpec, PostHook, and the builders tseries(), histogram(), histogram_2d(), edge(), and matrix_tseries() moved to polyphys.spec; importing them from here warned with DeprecationWarning.

Changed in version 1.0.0: Those names are no longer served from here; import them from polyphys.spec.

Functions

equilibrium_wholes(ens_database, properties, ...)

The equilibrium value of each whole's time series, with its error, beside the whole's stamps.

half_contour(n_monomers, bond_max)

The largest separation two monomers of a ring can have.

reduce_space(input_database, spec, table, *)

Reduce every property of a table over one space of probe artifacts.

reduce_stamps(stamp_files, lineage[, save_to])

Combine the stamps of one space into whole and ensemble stamps.

space_hists_from_spec(input_database, spec, ...)

Stack the ensemble-averaged histograms of one property over a space, with the columns and the rounding step the study's spec declares.

space_tseries_from_spec(input_database, ...)

Stack the ensemble-averaged time series of one property over a space, with the columns and the rounding step the study's spec declares.

Classes

Autocorrelation(nlags)

Pass a time series through and add its autocorrelation; a post-hook.

DensityKernel(*args, **kwargs)

The volume-fraction kernel that LocalDensities calls.

Derived(name, wholes, whole_type[, edges, ...])

One artifact a post-hook returns, ready for the ensemble stages.

FociPairDistances(bin_size, r_max, n_monomers)

Reduce the distance-matrix time series of the foci; a post-hook.

HookContext(prop[, spec, group, infos, ...])

What a post-hook may read besides the wholes.

LocalDensities()

Pass a histogram through and add its density profiles; a post-hook.

TimeAverage(output[, drop_first_column])

Average a time series over its frames; a post-hook.

class polyphys.analyze.pipeline.Autocorrelation(nlags)[source]#

Bases: object

Pass a time series through and add its autocorrelation; a post-hook.

Variables:

nlags (int) – The largest lag, in frames; see acf().

Parameters:

nlags (int)

Notes

Returns the series under its own name and \(r_k\) for \(k = 0, \dots,\) nlags under <property>-acf, both 'vector' wholes. No confidence band is stored: Bartlett’s band is a function of the stored \(r_k\), the number of frames, and the level, so bartlett_half_width() rebuilds it at any level when it is drawn. Time is \(O(n_\mathrm{frames} \log n_\mathrm{frames})\) per whole.

Examples

The alternating series has \(r_1 = -7/8\) and \(r_2 = 6/8\):

>>> import numpy as np
>>> from polyphys.spec import tseries
>>> hook = Autocorrelation(nlags=2)
>>> context = HookContext(tseries('gyrT', 'Mon'))
>>> series, acf_out = hook({'w1': np.tile([1.0, -1.0], 4)}, context)
>>> series.name, acf_out.name
('gyrTMon', 'gyrTMon-acf')
>>> np.round(acf_out.wholes['w1'], 6)
array([ 1.   , -0.875,  0.75 ])

Added in version 0.6.0.

nlags: int#
outputs(prop)[source]#

The series, passed through, and <property>-acf.

Examples

>>> from polyphys.spec import tseries
>>> Autocorrelation(nlags=2).outputs(tseries('gyrT', 'Mon'))
('gyrTMon', 'gyrTMon-acf')

Added in version 0.9.0.

Return type:

tuple[str, ...]

Parameters:

prop (PropertySpec)

class polyphys.analyze.pipeline.DensityKernel(*args, **kwargs)[source]#

Bases: typing.Protocol

The volume-fraction kernel that LocalDensities calls.

A kernel turns the histogram of one species in one whole into its local number density and local volume fraction, bin by bin, from the geometry the whole’s parser instance describes. The study’s driver supplies one, so this module imports no density model.

Added in version 0.6.0.

__call__(counts, edges, info, direction, diameter)[source]#
Parameters:
  • counts (numpy.ndarray) – The histogram of one whole, shape (n_bins,): the number of particle centers in each bin, summed over the frames.

  • edges (numpy.ndarray) – Its bin edges, shape (n_bins + 1,), in the length unit.

  • info (polyphys.manage.parser.ParserBase) – The whole’s parser instance, which holds the box sizes. It holds no number of frames, so a kernel that divides counts by the frames gets them from the study’s driver.

  • direction (str) – The direction of the histogram, such as 'r' or 'z'.

  • diameter (float) – The diameter of the species, in the length unit.

Returns:

The number density and the volume fraction, each of shape (n_bins,).

Return type:

tuple[numpy.typing.ArrayLike, numpy.typing.ArrayLike]

class polyphys.analyze.pipeline.Derived(name, wholes, whole_type, edges=None, normalizer=None)[source]#

Bases: object

One artifact a post-hook returns, ready for the ensemble stages.

Variables:
  • name (str) – The output property, which names the files; for example 'rRhoCrd' or 'gyrTMon-acf'.

  • wholes (dict of str to numpy.ndarray or pandas.DataFrame) – The artifact of each whole, by whole name.

  • whole_type ({'vector', 'matrix', 'bin_edge', 'dataframe'}) – The shape of each whole; see the module notation.

  • edges (Mapping of str to numpy.ndarray, optional) – The bin edges of each whole, for a 'vector' histogram, so its ensembles carry a bin_center column.

  • normalizer ({'first', 'last', 'max'}, optional) – For a 'vector' profile, the reference bin of a per-whole normalization: the first bin, the last bin, or the bin where the ensemble average is largest. Each whole is divided by its own value in that bin, and the standard error over the wholes of these separately normalized profiles is written beside the ensemble average as <name>-norm-sem; it is not generally an uncertainty estimate for the normalized ensemble mean. The ensemble average itself is not normalized; polyphys.analyze.sumrule.space_profiles() does that and carries the column through.

Raises:

ValueError – If whole_type is unknown, if edges or normalizer is given for a whole type other than 'vector', or if normalizer is not one of its three values.

Parameters:

Notes

The wholes of an ensemble are independent runs, so their spread is a valid error of any function of the profile, the normalized one included, with no covariance between a bin and its normalizer to track. The ensemble picks the bin so that every whole is normalized at the same place: a per-whole 'max' would bias each ratio towards one before the spread is taken. The error is zero in the reference bin by construction. The scale of a profile, such as the 1 / a or a^2 of the sum rule, cancels in the ratio, so the column holds for the scaled profile too.

Examples

>>> import numpy as np
>>> out = Derived('gyrMon', {'w1': np.array([1.0])}, 'vector')
>>> out.name, out.whole_type
('gyrMon', 'vector')

Added in version 0.6.0.

Changed in version 0.7.0: Added normalizer.

edges: collections.abc.Mapping[str, numpy.ndarray] | None = None#
name: str#
normalizer: Optional[Literal['first', 'last', 'max']] = None#
whole_type: Literal['vector', 'matrix', 'bin_edge', 'dataframe']#
wholes: dict[str, numpy.ndarray] | dict[str, pandas.DataFrame]#
class polyphys.analyze.pipeline.FociPairDistances(bin_size, r_max, n_monomers)[source]#

Bases: object

Reduce the distance-matrix time series of the foci; a post-hook.

On distMatT<species>, whose whole is the (n_frames, n_foci, n_foci) matrix of center-to-center distances between the foci with the foci’s backbone indices on the diagonal of frame 0, it writes three 'dataframe' wholes with one column per pair, tagged loc<i>loc<j>genDist<d> with i < j the backbone indices and d their genomic distance in bonds:

  • pairDistHist<species>: the histogram of each pair’s separation on one grid shared by every pair and whole, with a bin_center column;

  • pairDistRdf<species>: \(\bar P(R)\), the probability density of each pair’s separation vector on that grid, with bin_center;

  • pairDistT<species>: each pair’s separation over the frames.

Variables:
  • bin_size (float or callable()) – The bin width \(\Delta r\), in the length unit, or a function of the whole’s parser instance that gives it.

  • r_max (float or callable()) – The upper end of the grid, or a function of the parser instance that gives it, such as half_contour() of the ring; the grid runs to the first multiple of bin_size at or above it.

  • n_monomers (int or callable()) – The number of monomers on the backbone, or a function of the parser instance that gives it.

Raises:

ValueError – When built: if a constant bin_size or r_max is not a finite positive number, or a constant n_monomers is not a positive integer; a Python or NumPy number, never a bool, and for n_monomers never a float, even 10.0. When called: if the context has no spec (the topology comes from it); if a whole is not a (n_frames, n_foci, n_foci) array; if a callable quantity gives a value that breaks the same rule, which names the quantity and the whole; or if a separation lies above the grid, which names the whole and the separation, since a dropped frame would break the normalization.

Parameters:

Notes

Every pair has one separation per frame, so each histogram column sums to n_frames and \(\sum_k \bar P_k \Delta V_k = 1\) exactly. The research code set the grid from each whole’s own largest separation, so wholes of one ensemble had different edges; a bound from the system gives one grid. Time is \(O(n_\mathrm{pairs} n_\mathrm{frames} \log n_\mathrm{bins})\) and space \(O(n_\mathrm{pairs} (n_\mathrm{frames} + n_\mathrm{bins}))\) per whole.

Examples

Three foci at backbone indices 0, 4, and 8 of a ring of 10 monomers, over two frames, on a grid of unit bins to 3:

>>> import numpy as np
>>> from polyphys.tests.synthetic import RingCub
>>> from polyphys.spec import ProjectSpec, matrix_tseries
>>> spec = ProjectSpec(
...     parser=RingCub, group='bug', divisor=0.025,
...     crowder_attr='phi_bulk_c',
...     physical_attrs=('space',), species_sizes={},
...     tseries_properties=())
>>> frame0 = np.array([[0.0, 1.5, 2.5], [1.5, 4.0, 0.5], [2.5, 0.5, 8.0]])
>>> frame1 = np.array([[0.0, 1.5, 2.9], [1.5, 0.0, 2.5], [2.9, 2.5, 0.0]])
>>> hook = FociPairDistances(bin_size=1.0, r_max=3.0, n_monomers=10)
>>> context = HookContext(matrix_tseries('distMatT', 'Foci'), spec=spec)
>>> hist, density, series = hook({'w1': np.stack([frame0, frame1])},
...                              context)
>>> list(hist.wholes['w1'].columns)
['loc0loc4genDist4', 'loc0loc8genDist2', 'loc4loc8genDist4', 'bin_center']
>>> hist.wholes['w1']['loc0loc8genDist2'].to_list()
[0, 0, 2]
>>> series.wholes['w1']['loc4loc8genDist4'].to_list()
[0.5, 2.5]

A constant that breaks the rule raises when the hook is built:

>>> FociPairDistances(bin_size=1.0, r_max=3.0, n_monomers=10.5)
Traceback (most recent call last):
...
ValueError: FociPairDistances: 'n_monomers' must be a positive integer ...

Added in version 0.6.0.

Changed in version 1.0.0: The constants are checked when the hook is built, and n_monomers, a constant or what a callable gives, must be a positive integer, where a float such as 10.5 was truncated to 10.

bin_size: Union[float, Callable[[polyphys.manage.parser.ParserBase], float]]#
n_monomers: int | collections.abc.Callable[[polyphys.manage.parser.ParserBase], int]#
outputs(prop)[source]#

The pair histograms, separation densities, and separation time series of the species of prop.

Examples

>>> from polyphys.spec import matrix_tseries
>>> hook = FociPairDistances(bin_size=1.0, r_max=3.0, n_monomers=10)
>>> hook.outputs(matrix_tseries('distMatT', 'Foci'))
('pairDistHistFoci', 'pairDistRdfFoci', 'pairDistTFoci')

Added in version 0.9.0.

Return type:

tuple[str, ...]

Parameters:

prop (PropertySpec)

r_max: Union[float, Callable[[polyphys.manage.parser.ParserBase], float]]#
class polyphys.analyze.pipeline.HookContext(prop, spec=None, group='bug', infos=dict(), edges=None, density=None)[source]#

Bases: object

What a post-hook may read besides the wholes.

Added in version 0.6.0.

Variables:
  • prop (PropertySpec) – The property the wholes belong to.

  • spec (ProjectSpec, optional) – The study’s spec: its parser, topology, and species sizes.

  • group (str) – The particle group.

  • infos (Mapping of str to ParserBase) – The parser instance of each whole, by whole name, which holds the whole’s attributes such as sizes and counts.

  • edges (Mapping of str to numpy.ndarray, optional) – The bin edges of each whole, for a property with an edge file.

  • density (DensityKernel, optional) – The injected volume-fraction kernel.

Parameters:
density: polyphys.analyze.pipeline.DensityKernel | None = None#
edges: collections.abc.Mapping[str, numpy.ndarray] | None = None#
group: str = 'bug'#
infos: collections.abc.Mapping[str, polyphys.manage.parser.ParserBase]#
prop: polyphys.spec.PropertySpec#
spec: polyphys.spec.ProjectSpec | None = None#
class polyphys.analyze.pipeline.LocalDensities[source]#

Bases: object

Pass a histogram through and add its density profiles; a post-hook.

On <direction>Hist<species> it adds <direction>Rho<species>, the local number density, and <direction>Phi<species>, the local volume fraction, from the DensityKernel given to reduce_space(), with the species’ diameter read from the attribute the study’s species_sizes names.

Raises:

ValueError – When called: if the property is not a histogram(), if the context has no kernel, no spec, or no edges, or if the species is not in the spec’s species_sizes.

Notes

The three outputs share the histogram’s edges, so their ensembles carry the same bin_center column. The two profiles get the normalizer of PROFILE_NORMALIZERS for the parser’s geometry and the histogram’s direction, when the pair has one, so their ensemble averages carry <property>-norm-sem, the SEM across profiles normalized separately for each whole (see Derived). Time is one kernel call per whole.

Examples

With a stub kernel that returns the counts and a constant, on one whole of the ChainCyl study:

>>> import numpy as np
>>> from polyphys.tests.synthetic import ChainCyl
>>> from polyphys.spec import ProjectSpec, histogram
>>> spec = ProjectSpec(
...     parser=ChainCyl, group='bug', divisor=0.025,
...     crowder_attr='phi_bulk_c',
...     physical_attrs=('space',), species_sizes={'Crd': 'dcrowd'},
...     tseries_properties=())
>>> whole = ('N20r5.5lz25.0ac1.0nc100'
...          'dt0.005bdump1000adump5000ens1')
>>> def kernel(counts, edges, info, direction, size):
...     return counts, np.full(counts.shape, size)
>>> context = HookContext(
...     histogram('r', 'Crd'), spec=spec, group='all',
...     infos={whole: ChainCyl(whole, 'whole', 'all')},
...     edges={whole: np.array([0.0, 1.0, 2.0])}, density=kernel)
>>> outputs = LocalDensities()({whole: np.array([3.0, 5.0])}, context)
>>> [out.name for out in outputs]
['rHistCrd', 'rRhoCrd', 'rPhiCrd']
>>> outputs[2].wholes[whole]
array([1., 1.])
>>> outputs[2].normalizer
'first'

Added in version 0.6.0.

Changed in version 0.7.0: The profiles carry the sum rule’s default normalizer.

outputs(prop)[source]#

The histogram, passed through, and its <direction>Rho and <direction>Phi profiles.

Raises:

ValueError – If prop is not a histogram() property.

Return type:

tuple[str, ...]

Parameters:

prop (PropertySpec)

Examples

>>> from polyphys.spec import histogram
>>> LocalDensities().outputs(histogram('z', 'Crd'))
('zHistCrd', 'zRhoCrd', 'zPhiCrd')

Added in version 0.9.0.

class polyphys.analyze.pipeline.TimeAverage(output, drop_first_column=False)[source]#

Bases: object

Average a time series over its frames; a post-hook.

Variables:
  • output (str) – The name of the average, without the species, which is appended: TimeAverage('clustersHist') on 'clustersHistTFoci' writes 'clustersHistFoci'.

  • drop_first_column (bool, default False) – Drop column 0 of each frame before averaging. The size histograms of count_clusters() carry the count of empty clusters, always 0, in column 0.

Parameters:
  • output (str)

  • drop_first_column (bool)

Notes

The frames are axis 0; the average has the shape of one frame, so a (n_frames, n_sizes) histogram time series becomes an n_sizes vector, reduced as a 'vector' whole. Time and space are \(O(n_\mathrm{frames} n_\mathrm{sizes})\) per whole.

Examples

>>> import numpy as np
>>> from polyphys.spec import matrix_tseries
>>> hook = TimeAverage('clustersHist', drop_first_column=True)
>>> context = HookContext(matrix_tseries('clustersHistT', 'Foci'))
>>> wholes = {'w1': np.array([[0, 1, 2], [0, 3, 4], [0, 5, 6]])}
>>> (out,) = hook(wholes, context)
>>> out.name, out.wholes['w1']
('clustersHistFoci', array([3., 4.]))

Added in version 0.6.0.

drop_first_column: bool = False#
output: str#
outputs(prop)[source]#

The name of the average: output then the species of prop.

Examples

>>> from polyphys.spec import matrix_tseries
>>> TimeAverage('clustersHist').outputs(
...     matrix_tseries('clustersHistT', 'Foci'))
('clustersHistFoci',)

Added in version 0.9.0.

Return type:

tuple[str, ...]

Parameters:

prop (PropertySpec)

polyphys.analyze.pipeline.equilibrium_wholes(ens_database, properties, whole_stamps, *, method='lee', topology=None)[source]#

The equilibrium value of each whole’s time series, with its error, beside the whole’s stamps.

For each property, every ensemble file of the ens stage is read once, and each of its columns, one whole’s time series, is reduced to its time average, its variance, and the blocked standard error of the average. The results of all the properties are merged with whole_stamps on whole, one row per whole: the table the research code’s equilibrium_wholes built for the allInOne phase.

Parameters:
  • ens_database (str | os.PathLike[str]) – The ens directory of one space and group, <root>/analysis/<prefix>-<group>-ens, whose <ensemble_long>-<group>-<property>.csv files reduce_space() wrote.

  • properties (collections.abc.Mapping[str, str]) – The time-series property of each ensemble file, mapped to its output name: {'gyrTMon': 'gyrMon'} reads the -gyrTMon.csv files and writes gyrMon-mean, gyrMon-var, and gyrMon-sem.

  • whole_stamps (pandas.DataFrame) – The stamps of the space’s wholes, one row per whole, with a whole column: the first result of reduce_stamps().

  • method (typing.Literal['lee', 'sem_err']) – The plateau rule of sem_blocked(); 'lee' is the published rule.

  • topology (str | None) – Appended to each whole’s name as .<topology> before the merge, for stamps that the probe phase keyed by the whole’s file name, such as ...ens1.ring; the ensemble files name a whole without it.

Returns:

whole_stamps followed by, for each property in the order of properties, the columns <output>-mean, <output>-var, and <output>-sem, with the rows in the order of whole_stamps.

Return type:

pandas.DataFrame

Raises:
  • FileNotFoundError – If ens_database holds no CSV file, or a property matches none.

  • ValueError – If whole_stamps has no whole column or lists a whole twice; if two properties share an output name; if the wholes of a property’s files differ from those of the stamps, which names the wholes on each side; or as block_average() raises for a series that is too short or holds NaN.

Warns:

RuntimeWarning – As select_plateau(): when a whole’s series is too short for its correlation time, its -sem is NaN or unreliable.

Notes

-mean is the time average over the frames, -var the sample variance of the frames (ddof = 1), and -sem the standard error of the time average by block averaging, which accounts for the correlation between consecutive frames of one run (Flyvbjerg and Petersen, 1989; the plateau rule of Lee et al., 2011; both in polyphys.analyze.stats). A constant series has a -sem of 0. Blocking applies within a whole only: the wholes of an ensemble are independent runs, so ensemble_avg() keeps the plain standard error across them. The research code took -sem as the per-frame standard error, which the blocked one exceeds by about \(\sqrt{s}\) for a statistical inefficiency \(s\), and its -var as the population variance.

With \(n_\mathrm{frames}\) the frames of the longest series and \(n_\mathrm{wholes}\) the wholes of the space, time is \(O(n_\mathrm{wholes}\, n_\mathrm{frames})\) per property, and space is one ensemble file at a time plus a float64 copy of it.

Examples

Two wholes of one ensemble whose series alternate between two values, so their time averages are exact and the blocked error vanishes at a block size of two frames:

>>> import tempfile
>>> import numpy as np
>>> import pandas as pd
>>> stem = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> wholes = [f'{stem}ens1', f'{stem}ens2']
>>> root = tempfile.mkdtemp()
>>> pd.DataFrame({wholes[0]: np.tile([0.0, 2.0], 2048),
...               wholes[1]: np.tile([1.0, 3.0], 2048)}
...              ).to_csv(f'{root}/{stem}-bug-gyrTMon.csv', index=False)
>>> stamps = pd.DataFrame({'whole': wholes, 'n_frames': [4096, 4096]})
>>> table = equilibrium_wholes(root, {'gyrTMon': 'gyrMon'}, stamps)
>>> table.drop(columns='whole').round(3)
   n_frames  gyrMon-mean  gyrMon-var  gyrMon-sem
0      4096          1.0         1.0         0.0
1      4096          2.0         1.0         0.0

Added in version 0.6.0.

polyphys.analyze.pipeline.half_contour(n_monomers, bond_max)[source]#

The largest separation two monomers of a ring can have.

Parameters:
  • n_monomers (int) – The number of monomers on the ring, at least 2.

  • bond_max (float) – The largest length a bond can have, in the length unit; for the FENE bond, \(R_0 = 1.5\,\sigma_{ij}\) with \(\sigma_{ij}\) the collision diameter of the bonded pair (Kremer and Grest, 1990).

Returns:

(n_monomers // 2) * bond_max: the two monomers are at most n_monomers // 2 bonds apart along the ring, and a path of bonds is longest when straight and fully extended.

Return type:

float

Raises:

ValueError – If n_monomers is below 2 or bond_max is not positive.

Notes

A bound, not a typical value: it lets a pair-distance histogram cover every separation the chain allows, so that its counts conserve the number of frames and a separation above it marks a broken bond or an unwrapping error. Pass the largest bond’s \(R_0\) when the bonds differ; the bound is then loose by the ratio of the largest to the typical bond, which costs bins, not correctness.

Examples

A ring of 10 beads with bonds that stretch to 1.5:

>>> half_contour(10, 1.5)
7.5

Added in version 0.6.0.

polyphys.analyze.pipeline.reduce_space(input_database, spec, table, *, hierarchy=None, density=None, save=True)[source]#

Reduce every property of a table over one space of probe artifacts.

Parameters:
  • input_database (str | os.PathLike[str]) – The probe directory of the space, <root>/probes/<prefix>-<group>-probe; it may hold a glob, as the research code’s drivers pass it.

  • spec (polyphys.spec.ProjectSpec) – The study’s spec: its parser, topology, and species sizes.

  • table (polyphys.spec.AnalysisTable) – The group’s analysis table.

  • hierarchy (str | None) – The glob, relative to input_database, that lists the artifact files; the default is <spec.hierarchy>/<spec.hierarchy>, the research code’s dir/file layout.

  • density (polyphys.analyze.pipeline.DensityKernel | None) – The volume-fraction kernel for LocalDensities.

  • save (bool) – Write the stages’ files under <root>/analysis/; False returns the ensemble averages without writing anything.

Returns:

The ensemble averages by output property, then by ensemble name: a DataFrame for a 'vector' or 'dataframe' output, a dict of arrays for a 'matrix', and the edges for a 'bin_edge'.

Return type:

dict[str, dict[str, numpy.ndarray | pandas.DataFrame]]

Raises:
  • FileNotFoundError – If the glob matches no file, or a property or its edge property matches no file; checked for every property before any directory is made.

  • ValueError – If the wholes of one ensemble hold different bin edges; if a histogram’s edge files do not cover its wholes; if a hook returns an output name the table already produced; as the organizer and the hooks raise.

Notes

For each property in turn: its files are selected by their last name token, wholes are built (for the 'segment' lineage with whole_from_segments(), written to wholeSim; else with whole_from_file()), the edges are loaded and checked, the post-hook is applied, and every returned artifact goes through ensemble() and ensemble_avg(), or their long-form counterparts for 'dataframe' wholes. With table.stamps, the stamps are combined first by reduce_stamps(). One property’s wholes are in memory at a time, so peak memory is the largest property’s n_wholes wholes plus its ensembles; time is \(O(n_\mathrm{files}\, n_\mathrm{frames})\) per property plus the hook’s.

Examples

Two replicates of one state point of the ChainCyl study, written as wholes and reduced without saving:

>>> import tempfile
>>> import numpy as np
>>> from polyphys.tests.synthetic import ChainCyl
>>> from polyphys.spec import AnalysisTable, ProjectSpec, tseries
>>> spec = ProjectSpec(
...     parser=ChainCyl, group='bug', divisor=0.025,
...     crowder_attr='phi_bulk_c',
...     physical_attrs=('space',), species_sizes={'Mon': 'dmon'},
...     tseries_properties=('gyrTMon',))
>>> stem = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> root = tempfile.mkdtemp()
>>> np.save(f'{root}/{stem}ens1-bug-gyrTMon.npy', np.array([1.0, 2.0]))
>>> np.save(f'{root}/{stem}ens2-bug-gyrTMon.npy', np.array([3.0, 6.0]))
>>> table = AnalysisTable('bug', 'whole', False, (tseries('gyrT', 'Mon'),))
>>> averages = reduce_space(root, spec, table, hierarchy='*', save=False)
>>> averages['gyrTMon'][stem]
   gyrTMon-mean  gyrTMon-var  gyrTMon-sem
0           2.0          2.0          1.0
1           4.0          8.0          2.0

Added in version 0.6.0.

polyphys.analyze.pipeline.reduce_stamps(stamp_files, lineage, save_to=(None, None, None))[source]#

Combine the stamps of one space into whole and ensemble stamps.

Parameters:
Return type:

tuple[pandas.DataFrame, pandas.DataFrame]

Returns:

Raises:

ValueError – As children_stamps() and parents_stamps().

Notes

Segment stamps combine with children_stamps(), then roll up once with parents_stamps() into whole stamps; whole stamps combine directly. Either way the whole stamps roll up once more into ensemble stamps. Time and space are \(O(n_\mathrm{files}\, n_\mathrm{columns})\).

Examples

whole_stamps, ensemble_stamps = reduce_stamps(
    [(path,) for path in sorted(stamp_paths)], 'whole')

Added in version 0.6.0.

polyphys.analyze.pipeline.space_hists_from_spec(input_database, spec, prop, *, group=None, bin_center=None, normalize=False, round_to=3, save=False)[source]#

Stack the ensemble-averaged histograms of one property over a space, with the columns and the rounding step the study’s spec declares.

The spec supplies every study choice of space_hists(), as space_tseries_from_spec() describes; no study default is taken from the library.

Parameters:
  • input_database (str | os.PathLike[str]) – The ensAvg directory of the space, which holds the files <ensemble_long>-<group>-<prop>-ensAvg.csv.

  • spec (polyphys.spec.ProjectSpec) – The study’s spec.

  • prop (str) – The property, such as 'rHistCrd'.

  • group (str | None) – The particle group of the files; the default is the polymer group, spec.group. A histogram of the crowders is in the 'all' group.

  • bin_center (numpy.ndarray | None) – The bin centers, shape (n_bins,); otherwise each file must hold a bin_center column.

  • normalize (bool) – Add <prop>-norm, each file’s <prop>-mean divided by its sum.

  • round_to (int) – The number of decimal places of the rounded \(\phi_c\).

  • save (bool) – Also write the table to <space>-<group>-<prop>-space.csv in the analysis phase’s space directory next to input_database.

Returns:

The table of space_hists(): each file’s columns, one column per physical attribute, a spec.crowder_attr column, and phi_bulk_c_round.

Return type:

pandas.DataFrame

Warns:

UserWarning – If normalize is true and a histogram sums to zero, as space_hists() warns.

Notes

The cost is that of space_hists(): \(O(n_\mathrm{rows}\, n_\mathrm{cols})\) time and memory.

Examples

>>> import tempfile
>>> import numpy as np
>>> import pandas as pd
>>> from pathlib import Path
>>> from polyphys.tests.synthetic import ChainCyl
>>> from polyphys.spec import ProjectSpec
>>> spec = ProjectSpec(
...     parser=ChainCyl, group='bug', divisor=0.1,
...     crowder_attr='phi_bulk_c', physical_attrs=('nmon',),
...     species_sizes={'Mon': 'dmon'}, tseries_properties=())
>>> database = Path(tempfile.mkdtemp()) / 'analysis' / 'N20-all-ensAvg'
>>> database.mkdir(parents=True)
>>> state = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> pd.DataFrame({'rHistMon-mean': [1.0, 3.0]}).to_csv(
...     database / f'{state}-all-rHistMon-ensAvg.csv', index=False)
>>> space = space_hists_from_spec(
...     database, spec, 'rHistMon', group='all',
...     bin_center=np.array([0.5, 1.5]), normalize=True)
>>> space[['bin_center', 'rHistMon-norm', 'phi_bulk_c_round']]
   bin_center  rHistMon-norm  phi_bulk_c_round
0         0.5           0.25               0.0
1         1.5           0.75               0.0

Added in version 0.9.0.

polyphys.analyze.pipeline.space_tseries_from_spec(input_database, spec, prop, *, group=None, round_to=3, save=False)[source]#

Stack the ensemble-averaged time series of one property over a space, with the columns and the rounding step the study’s spec declares.

The spec supplies every study choice of space_tseries(): the parser, the ensemble_long glob prefix (spec.hierarchy), the attributes that become columns (spec.physical_attrs), the rounding step \(\Delta\) of the bulk volume fraction of crowders \(\phi_c\) (spec.divisor), and the parser attribute that holds \(\phi_c\) (spec.crowder_attr). No study default is taken from the library.

Parameters:
  • input_database (str | os.PathLike[str]) – The ensAvg directory of the space, which holds the files <ensemble_long>-<group>-<prop>-ensAvg.csv.

  • spec (polyphys.spec.ProjectSpec) – The study’s spec.

  • prop (str) – The property, such as 'gyrTMon'.

  • group (str | None) – The particle group of the files; the default is the polymer group, spec.group.

  • round_to (int) – The number of decimal places of the rounded \(\phi_c\).

  • save (bool) – Also write the table to <space>-<group>-<prop>-space.csv in the analysis phase’s space directory next to input_database.

Returns:

The table of space_tseries(): each file’s columns, t_index, time, one column per physical attribute, a spec.crowder_attr column, and phi_bulk_c_round.

Return type:

pandas.DataFrame

Notes

The cost is that of space_tseries(): \(O(n_\mathrm{rows}\, n_\mathrm{cols})\) time and memory for the rows and columns of the result.

Examples

The spec’s step of 0.1 rounds \(\phi_c \approx 0.0123\) of this state point to 0.0:

>>> import tempfile
>>> import pandas as pd
>>> from pathlib import Path
>>> from polyphys.tests.synthetic import ChainCyl
>>> from polyphys.spec import ProjectSpec
>>> spec = ProjectSpec(
...     parser=ChainCyl, group='bug', divisor=0.1,
...     crowder_attr='phi_bulk_c', physical_attrs=('nmon',),
...     species_sizes={'Mon': 'dmon'}, tseries_properties=('gyrTMon',))
>>> database = Path(tempfile.mkdtemp()) / 'analysis' / 'N20-bug-ensAvg'
>>> database.mkdir(parents=True)
>>> state = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> pd.DataFrame({'gyrTMon-mean': [1.0, 2.0]}).to_csv(
...     database / f'{state}-bug-gyrTMon-ensAvg.csv', index=False)
>>> space = space_tseries_from_spec(database, spec, 'gyrTMon')
>>> space[['time', 'gyrTMon-mean', 'nmon', 'phi_bulk_c_round']]
   time  gyrTMon-mean  nmon  phi_bulk_c_round
0   0.0           1.0    20               0.0
1   5.0           2.0    20               0.0

Added in version 0.9.0.