polyphys.analyze.correlations#

polyphys.analyze.correlations#

The polyphys.analyze.correlations module measures how a signal is correlated with itself: the autocorrelation function of a time series with its confidence band, the decay model fitted to such a curve, and the cosine correlation between the bond vectors of a polymer chain, from which a persistence length follows. Every routine is a pure NumPy or SciPy kernel on arrays; reading time series or frames, choosing the lags to fit, and averaging over frames are the caller’s work.

Notation#

x

A time series, shape (n_frames,) or (n_frames, n_series), in the caller’s unit; the frames are equally spaced in time and run along axis 0, and each column is an independent series.

nlags

The largest lag \(k\) at which the autocorrelation is reported, in frames; a lag is the number of frames between the two values compared.

acf

The sample autocorrelation function \(r_k\), the covariance of the series with itself \(k\) frames later divided by its variance: \(r_0 = 1\) and \(|r_k| \le 1\).

half_width

The half-width of the confidence interval around \(r_k\) at level \(1 - \alpha\), from Bartlett’s large-sample variance of \(r_k\); the interval is acf - half_width to acf + half_width.

decay model

One of the forms of the stretched exponential (Kohlrausch-Williams-Watts) function \(A \exp[-(x / \tau)^\beta] + B\) listed in DECAY_MODELS, with the decay time \(\tau\) in the unit of x, the stretching exponent \(\beta\) (1 for a pure exponential), the amplitude \(A\), and the offset \(B\).

positions

The coordinates of the monomers of one chain in chain order, shape (..., n_atoms, n_dims), unwrapped (no periodic image breaks a bond); any leading axes, such as frames, are carried through.

topology

'linear' or 'ring': whether the chain’s last monomer is bonded back to its first.

bond vector

\(\mathbf{b}_i = \mathbf{r}_{i+1} - \mathbf{r}_i\), and on a ring also \(\mathbf{b}_{n-1} = \mathbf{r}_0 - \mathbf{r}_{n-1}\); n_bonds is n_atoms - 1 on a linear chain and n_atoms on a ring. \(\hat{\mathbf{u}}_i\) is the unit bond vector.

bond cosine correlation

\(C(k) = \langle \hat{\mathbf{u}}_i \cdot \hat{\mathbf{u}}_{i+k} \rangle\), the mean cosine of the angle between two bonds \(k\) bonds apart along the chain, averaged over the pairs in one frame; \(C(0) = 1\).

Contents#

Autocorrelation: acf() estimates \(r_k\) for \(k = 0, \dots,\) nlags by the fast Fourier transform, with Bartlett’s confidence band, as an AcfResult; bartlett_half_width() is that band on its own, for a stored \(r_k\) at any level.

Decay fits: DECAY_MODELS names the decay models; initial_guess() reads a starting point for each parameter off the data; fit_decay() fits a model with scipy.optimize.curve_fit() and reports the parameters, their standard errors, and whether the fit converged as a DecayFit, whose DecayFit.evaluate() draws the fitted curve.

Bonds: bond_vectors() differences the positions along the chain; bond_cosine_correlation() averages the cosines between bond pairs at every lag by the fast Fourier transform.

Conventions#

  • The autocorrelation uses the 1/n_frames estimator of the autocovariance at every lag, which keeps the sequence positive semi-definite and is the default of statsmodels.tsa.stattools.acf; dividing by n_frames - k instead is unbiased per lag but can give \(|r_k| > 1\).

  • Bartlett’s variance for lag \(k\) sums the squared autocorrelations at the lags below it, \(\mathrm{var}(r_k) = (1 + 2 \sum_{j=1}^{k-1} r_j^2) / n_\mathrm{frames}\) for \(k \ge 2\), with \(\mathrm{var}(r_1) = 1 / n_\mathrm{frames}\) and \(\mathrm{var}(r_0) = 0\); this is the convention of statsmodels, so a band computed here matches one computed there.

  • Decay models are written in the decay time \(\tau\), not the rate \(\omega = \tau^{-\beta}\) of some research code; convert with \(\tau = \omega^{-1/\beta}\).

  • A failed fit never raises: DecayFit.converged is false and the parameters are NaN.

  • On a ring every lag \(k\) has n_bonds pairs, going round the ring, so \(C(k) = C(n_\mathrm{bonds} - k)\) and the lags that carry information are \(0 \le k \le n_\mathrm{bonds} / 2\). On a linear chain lag \(k\) has n_bonds - k pairs.

  • Complexities are stated per call in n_frames (\(n\)), n_series (\(m\)), n_bonds (\(N\)), the number of leading frames of a positions array (\(F\)), and n_points of a fit.

References

Added in version 0.6.0.

Module Attributes

Topology

The polymer topologies the bond vectors are defined for.

DECAY_MODELS

The decay models fit_decay() accepts, by name.

Functions

acf(x, nlags[, alpha])

Estimate the autocorrelation function of a time series by the FFT.

bartlett_half_width(values, n_frames[, alpha])

The half-width of Bartlett's confidence band around an autocorrelation.

bond_cosine_correlation(positions, topology)

Average the cosine between bond vectors at every lag along a chain.

bond_vectors(positions, topology)

Difference the positions of a chain into its bond vectors.

fit_decay(x, y, model[, p0])

Fit a decay model to a curve by nonlinear least squares.

initial_guess(x, y, model)

Read a starting point for each parameter of a decay model off the data.

Classes

AcfResult(values, half_width, alpha)

The sample autocorrelation of one or more time series.

DecayFit(model, names, params, stderr, pcov, ...)

The result of fitting a decay model to a curve.

DecayModel(function, names, formula)

A decay model: its function, parameter names, and formula.

class polyphys.analyze.correlations.AcfResult(values, half_width, alpha)[source]#

Bases: object

The sample autocorrelation of one or more time series.

Added in version 0.6.0.

Variables:
  • values (numpy.ndarray) – \(r_k\) for lags 0 to nlags, shape (nlags + 1,) for a one-dimensional series or (nlags + 1, n_series) for columns. values[0] is 1.

  • half_width (numpy.ndarray or None) – The half-width of the confidence interval around each value at level 1 - alpha, the same shape as values; half_width[0] is 0. None when no band was asked for.

  • alpha (float or None) – The significance level of the band, or None.

Parameters:
  • values (NDArray[float64])

  • half_width (NDArray[float64] | None)

  • alpha (float | None)

alpha: float | None#
half_width: numpy.typing.NDArray[numpy.float64] | None#
values: numpy.typing.NDArray[numpy.float64]#
polyphys.analyze.correlations.DECAY_MODELS: collections.abc.Mapping[str, polyphys.analyze.correlations.DecayModel] = {'exp': DecayModel(function=<function _exp>, names=('tau',), formula='exp(-x/tau)'), 'stretched': DecayModel(function=<function _stretched>, names=('tau', 'beta'), formula='exp(-(x/tau)^beta)'), 'stretched_amp': DecayModel(function=<function _stretched_amp>, names=('tau', 'beta', 'amp'), formula='amp exp(-(x/tau)^beta)'), 'stretched_amp_offset': DecayModel(function=<function _stretched_amp_offset>, names=('tau', 'beta', 'amp', 'offset'), formula='amp exp(-(x/tau)^beta) + offset')}#

The decay models fit_decay() accepts, by name.

'exp' is the pure exponential \(\exp(-x/\tau)\); 'stretched' the Kohlrausch-Williams-Watts form \(\exp[-(x/\tau)^\beta]\); 'stretched_amp' scales it by an amplitude \(A\); and 'stretched_amp_offset' adds an offset \(B\). The parameters are ordered tau, beta, amp, offset. \(\tau\) is in the unit of x and positive; \(\beta\) is positive, 1 for a pure exponential.

Added in version 0.6.0.

class polyphys.analyze.correlations.DecayFit(model, names, params, stderr, pcov, p0, converged)[source]#

Bases: object

The result of fitting a decay model to a curve.

Added in version 0.6.0.

Variables:
  • model (str) – The key of the model in DECAY_MODELS.

  • names (tuple of str) – The parameter names, in the order of params.

  • params (numpy.ndarray) – The fitted parameters, shape (n_params,); all NaN when the fit did not converge.

  • stderr (numpy.ndarray) – The standard error of each parameter, the square root of the diagonal of pcov; NaN where the covariance could not be estimated or the fit did not converge.

  • pcov (numpy.ndarray) – The estimated covariance of the parameters, shape (n_params, n_params), as scipy.optimize.curve_fit() returns it; all NaN when the fit did not converge.

  • p0 (numpy.ndarray) – The starting point the fit used, shape (n_params,).

  • converged (bool) – Whether scipy.optimize.curve_fit() returned a solution.

Parameters:
  • model (str)

  • names (tuple[str, ...])

  • params (NDArray[float64])

  • stderr (NDArray[float64])

  • pcov (NDArray[float64])

  • p0 (NDArray[float64])

  • converged (bool)

converged: bool#
evaluate(x)[source]#

Evaluate the fitted model at x.

Parameters:

x (numpy.typing.ArrayLike) – Points in the unit of the fitted x.

Returns:

The model at x, the same shape; all NaN when the fit did not converge.

Return type:

numpy.typing.NDArray[numpy.double]

Examples

>>> import numpy as np
>>> from polyphys.analyze.correlations import fit_decay
>>> x = np.arange(0.0, 20.0)
>>> fit = fit_decay(x, np.exp(-x / 4.0), 'exp')
>>> np.round(fit.evaluate([0.0, 4.0]), 6)
array([1.      , 0.367879])
model: str#
names: tuple[str, ...]#
p0: numpy.typing.NDArray[numpy.float64]#
params: numpy.typing.NDArray[numpy.float64]#
pcov: numpy.typing.NDArray[numpy.float64]#
stderr: numpy.typing.NDArray[numpy.float64]#
class polyphys.analyze.correlations.DecayModel(function, names, formula)[source]#

Bases: object

A decay model: its function, parameter names, and formula.

Added in version 0.6.0.

Variables:
  • function (callable()) – function(x, *params), vectorized over x.

  • names (tuple of str) – The parameter names in the order function takes them, from 'tau', 'beta', 'amp', and 'offset'.

  • formula (str) – The model as a formula, for labels.

Parameters:
formula: str#
function: collections.abc.Callable[..., numpy.typing.NDArray[numpy.float64]]#
names: tuple[str, ...]#
polyphys.analyze.correlations.Topology#

The polymer topologies the bond vectors are defined for.

alias of Literal[‘linear’, ‘ring’]

polyphys.analyze.correlations.acf(x, nlags, alpha=0.05)[source]#

Estimate the autocorrelation function of a time series by the FFT.

Parameters:
  • x (numpy.typing.ArrayLike) – The time series, shape (n_frames,) or (n_frames, n_series), with the frames along axis 0 and one series per column; every value finite, and no series constant.

  • nlags (int) – The largest lag, in frames, from 0 to n_frames - 1.

  • alpha (float | None) – The significance level of the confidence band, in (0, 1); 0.05 gives a 95 % band. None skips the band.

Returns:

The autocorrelation at lags 0 to nlags and its band.

Return type:

polyphys.analyze.correlations.AcfResult

Raises:

ValueError – If x is not one- or two-dimensional or has fewer than two frames, holds a non-finite value, or has a constant series; if nlags is outside [0, n_frames - 1]; or if alpha is outside (0, 1).

Notes

With \(\bar{x}\) the mean of the series, the autocovariance is \(c_k = \frac{1}{n} \sum_{t=0}^{n-k-1} (x_t - \bar{x}) (x_{t+k} - \bar{x})\) and \(r_k = c_k / c_0\) (NIST/SEMATECH e-Handbook, section 1.3.3.1). The sum over \(t\) for every \(k\) is one linear correlation, computed as the inverse transform of the squared magnitude of the series’ Fourier transform after zero-padding to at least \(2n - 1\) points so that no wrap-around enters (Wiener-Khinchin).

The band is Bartlett’s (1946) large-sample approximation under the hypothesis that the autocorrelation vanishes beyond lag \(k - 1\): \(\mathrm{var}(r_k) \approx (1 + 2 \sum_{j=1}^{k-1} r_j^2) / n\), so the band widens with the lag, and the half-width is \(z_{1-\alpha/2} \sqrt{\mathrm{var}(r_k)}\) with \(z\) the standard normal quantile. For a white-noise series it reduces to \(\pm z_{1-\alpha/2} / \sqrt{n}\). The approximation needs \(n \gg\) nlags.

Time is \(O(m\, n \log n)\) and space \(O(m\, n)\), with \(n\) the frames and \(m\) the series.

Examples

The alternating series 1, -1, 1, -1, ... of 8 frames has mean 0, \(c_k = (-1)^k (8 - k) / 8\), and so \(r_1 = -7/8\) and \(r_2 = 6/8\):

>>> import numpy as np
>>> from polyphys.analyze.correlations import acf
>>> x = np.tile([1.0, -1.0], 4)
>>> result = acf(x, nlags=2)
>>> np.round(result.values, 6)
array([ 1.   , -0.875,  0.75 ])

The 95 % band is 0 at lag 0, \(1.96 / \sqrt{8}\) at lag 1, and wider at lag 2 because \(r_1^2\) enters:

>>> np.round(result.half_width, 4)
array([0.    , 0.693 , 1.1025])

Added in version 0.6.0.

polyphys.analyze.correlations.bartlett_half_width(values, n_frames, alpha=0.05)[source]#

The half-width of Bartlett’s confidence band around an autocorrelation.

Parameters:
  • values (numpy.typing.ArrayLike) – The sample autocorrelation \(r_k\) for lags 0 to nlags, shape (nlags + 1,) or (nlags + 1, n_series), as acf() returns it.

  • n_frames (int) – The number of frames the autocorrelation was estimated from, at least 2.

  • alpha (float) – The significance level of the band, in (0, 1); 0.05 gives a 95 % band.

Returns:

The half-width at each lag, the same shape as values, float64; 0 at lag 0. The band is values - half_width to values + half_width.

Return type:

numpy.typing.NDArray[numpy.double]

Raises:

ValueError – If values is not one- or two-dimensional, n_frames is below 2, or alpha is outside (0, 1).

Notes

Bartlett’s (1946) large-sample variance under the hypothesis that the autocorrelation vanishes beyond lag \(k - 1\) is \(\mathrm{var}(r_k) \approx (1 + 2 \sum_{j=1}^{k-1} r_j^2) / n\), and the half-width is \(z_{1-\alpha/2} \sqrt{\mathrm{var}(r_k)}\). The band depends on the stored \(r_k\), n_frames, and alpha only, so a pipeline need store the autocorrelation alone and draw the band at any level later. Time and space are \(O(m\, k)\), with \(k\) the lags and \(m\) the series.

Examples

The band of the alternating series of acf()’s example, rebuilt from its autocorrelation:

>>> import numpy as np
>>> from polyphys.analyze.correlations import bartlett_half_width
>>> np.round(bartlett_half_width([1.0, -0.875, 0.75], 8), 4)
array([0.    , 0.693 , 1.1025])

Added in version 0.6.0.

polyphys.analyze.correlations.bond_cosine_correlation(positions, topology)[source]#

Average the cosine between bond vectors at every lag along a chain.

Parameters:
  • positions (numpy.typing.ArrayLike) – The monomer coordinates in chain order, shape (..., n_atoms, n_dims), unwrapped; no two bonded monomers coincide.

  • topology (typing.Literal['linear', 'ring']) – Whether the last monomer is bonded back to the first.

Returns:

\(C(k)\) for lags 0 to n_bonds - 1, shape (..., n_bonds), dimensionless in [-1, 1] with C(0) = 1. Leading axes are kept, so the average over frames is result.mean(axis=0) for a (n_frames, n_atoms, n_dims) input.

Return type:

numpy.typing.NDArray[numpy.double]

Raises:

ValueError – If topology is unknown, positions has a bad shape or a chain too short for its topology, or a bond has zero length.

Notes

With \(\hat{\mathbf{u}}_i\) the unit bond vectors, on a linear chain \(C(k) = \frac{1}{N - k} \sum_{i=0}^{N-k-1} \hat{\mathbf{u}}_i \cdot \hat{\mathbf{u}}_{i+k}\) over the \(N - k\) pairs at lag \(k\); on a ring \(C(k) = \frac{1}{N} \sum_{i=0}^{N-1} \hat{\mathbf{u}}_i \cdot \hat{\mathbf{u}}_{(i+k) \bmod N}\) over all \(N\) pairs, so that the result does not depend on which monomer is labeled first and \(C(k) = C(N - k)\). The sums over \(i\) are linear (zero-padded) or circular autocorrelations, computed by the FFT of each Cartesian component and summed over components.

For a worm-like chain \(C(k) = \exp(-k\, b / \ell_p)\) with \(b\) the bond length and \(\ell_p\) the persistence length (Rubinstein and Colby 2003, section 2.3), so fitting 'exp' with fit_decay() on lags up to n_bonds // 2 for a ring gives \(\tau = \ell_p / b\), and \(\ell_p\) is \(\tau\) times the mean bond length. On a linear chain this is the quantity and the fit of MDAnalysis.analysis.polymer.PersistenceLength (results.bond_autocorrelation, results.lb, results.lp), which averages over chains and frames and does not treat rings.

Time is \(O(F\, N \log N)\) and space \(O(F\, N)\) for \(F\) frames of \(N\) bonds, against the \(O(F\, N^2)\) of the pairwise matrix.

Examples

The bonds of a square ring turn by a right angle each, so \(C(k) = \cos(\pi k / 2)\):

>>> import numpy as np
>>> from polyphys.analyze.correlations import bond_cosine_correlation
>>> square = np.array([[0.0, 0.0], [1.0, 0.0], [1.0, 1.0], [0.0, 1.0]])
>>> np.round(bond_cosine_correlation(square, 'ring'), 12)
array([ 1.,  0., -1.,  0.])

A straight rod has every bond parallel:

>>> rod = np.outer(np.arange(4.0), [1.0, 2.0, 2.0])
>>> np.round(bond_cosine_correlation(rod, 'linear'), 12)
array([1., 1., 1.])

Added in version 0.6.0.

polyphys.analyze.correlations.bond_vectors(positions, topology)[source]#

Difference the positions of a chain into its bond vectors.

Parameters:
  • positions (numpy.typing.ArrayLike) – The monomer coordinates in chain order, shape (..., n_atoms, n_dims), unwrapped, in the caller’s length unit; n_atoms at least 2 on a linear chain and 3 on a ring.

  • topology (typing.Literal['linear', 'ring']) – Whether the last monomer is bonded back to the first.

Returns:

The bond vectors \(\mathbf{r}_{i+1} - \mathbf{r}_i\), shape (..., n_bonds, n_dims), with n_bonds = n_atoms - 1 on a linear chain and n_atoms on a ring, whose last bond is \(\mathbf{r}_0 - \mathbf{r}_{n-1}\). Bond lengths are numpy.linalg.norm(bonds, axis=-1).

Return type:

numpy.typing.NDArray[numpy.double]

Raises:

ValueError – If topology is unknown, positions has fewer than two dimensions, or the chain is too short for its topology.

Notes

Positions must be unwrapped: a periodic image jump inside a bond gives a bond of the box length. Time and space are \(O(F\, n_\mathrm{atoms}\, n_\mathrm{dims})\).

Examples

>>> import numpy as np
>>> from polyphys.analyze.correlations import bond_vectors
>>> square = np.array([[0.0, 0.0], [1.0, 0.0], [1.0, 1.0], [0.0, 1.0]])
>>> bond_vectors(square, 'linear')
array([[ 1.,  0.],
       [ 0.,  1.],
       [-1.,  0.]])
>>> bond_vectors(square, 'ring')[-1]
array([ 0., -1.])

Added in version 0.6.0.

polyphys.analyze.correlations.fit_decay(x, y, model, p0=None, **kwargs)[source]#

Fit a decay model to a curve by nonlinear least squares.

Parameters:
Returns:

The parameters, their standard errors, the covariance, the starting point used, and whether the fit converged.

Return type:

polyphys.analyze.correlations.DecayFit

Raises:

ValueError – If model is unknown; x and y are not finite one-dimensional arrays of one length; there are fewer points than parameters; or p0 has the wrong length. scipy.optimize.curve_fit() also raises ValueError for an invalid option in kwargs, such as an unknown method or a p0 outside bounds.

Notes

scipy.optimize.curve_fit() minimizes the sum of squared residuals from p0, by default with the Levenberg-Marquardt method ('lm'), or with 'trf' when bounds are given, as its documentation states; method in kwargs chooses another. A nonlinear model can have several local minima, so p0 matters, and the one used is recorded. A fit fails when the model is not finite at the start, when the optimizer stops without a solution, or when it does not move from the start and the covariance cannot be estimated, which is how the 'lm' method signals a Jacobian that is zero or rank-deficient (the scipy.optimize.curve_fit() documentation, under pcov). A failed fit is reported with converged=False, NaN parameters, and a warning in the logging log, not an exception; invalid inputs or options still raise (see Raises). When the fit moves but the covariance cannot be estimated (SciPy emits an OptimizeWarning), it is converged and the standard errors of the affected parameters are NaN. A start at the exact solution of noise-free data does not move either, but its covariance is finite, so it is converged. The standard error is the square root of the covariance diagonal, as in SciPy’s documentation.

The models are the stretched exponential of Williams and Watts (1970), Trans. Faraday Soc. 66, 80, https://doi.org/10.1039/TF9706600080, with \(\beta = 1\) for the pure exponential.

Time is \(O(n_\mathrm{iter}\, n_\mathrm{points}\, n_\mathrm{params})\) and space \(O(n_\mathrm{points}\, n_\mathrm{params})\).

Examples

>>> import numpy as np
>>> from polyphys.analyze.correlations import fit_decay
>>> x = np.arange(0.0, 40.0)
>>> y = 2.0 * np.exp(-((x / 5.0) ** 0.8)) + 0.3
>>> fit = fit_decay(x, y, 'stretched_amp_offset')
>>> fit.converged
True
>>> fit.names
('tau', 'beta', 'amp', 'offset')
>>> np.round(fit.params, 6)
array([5. , 0.8, 2. , 0.3])

Added in version 0.6.0.

Changed in version 0.9.0: A start where the model is not finite, and a fit that does not move and has no finite covariance, are failed fits; method='trf' no longer raises on a non-finite start.

polyphys.analyze.correlations.initial_guess(x, y, model)[source]#

Read a starting point for each parameter of a decay model off the data.

Parameters:
Returns:

One value per parameter of the model, in its order.

Return type:

numpy.typing.NDArray[numpy.double]

Raises:

ValueError – If model is unknown, or x and y are not finite one-dimensional arrays of one length.

Notes

The offset \(B\) is the mean of the last tenth of y (at least one point); the amplitude \(A\) is y[0] - B, or 1 when that is 0; the decay time \(\tau\) is the first x at which \((y - B) / A\) is at or below \(1/e\), which is exact for a pure exponential sampled at \(\tau\), or the last x when the curve never falls that far; and \(\beta\) is 1. A model without an offset takes \(B = 0\); one without an amplitude takes \(A = 1\). A \(\tau\) of 0 (the curve is already at \(1/e\) at x[0] = 0) is replaced by the first positive x, or 1.

Time and space are \(O(n_\mathrm{points})\).

Examples

A pure exponential of decay time 4 crosses \(1/e\) at x = 4:

>>> import numpy as np
>>> from polyphys.analyze.correlations import initial_guess
>>> x = np.arange(0.0, 20.0)
>>> initial_guess(x, np.exp(-x / 4.0), 'exp')
array([4.])

The offset is read off the tail, so it is a starting point, not the answer:

>>> y = 2.0 * np.exp(-x / 4.0) + 0.5
>>> np.round(initial_guess(x, y, 'stretched_amp_offset'), 2)
array([4.  , 1.  , 1.98, 0.52])

Added in version 0.6.0.