"""\
==========================================================
:mod:`polyphys.manage.organizer`
==========================================================
The :mod:`polyphys.manage.organizer` module provides tools for organizing and
aggregating simulation data based on filename patterns, file structures, and
processing stages. It facilitates handling segmented data, grouping experiments
into ensembles, and consolidating ensembles into spaces.
The module is tailored for use with LAMMPS and similar simulation tools, where
it automates data consolidation across hierarchical levels. It is particularly
useful for organizing data generated in high-throughput simulations, enabling
efficient post-processing, ensemble averaging, and preparation of data for
statistical and comparative analysis.
Terminology
===========
The following terms are used throughout the module to structure and organize
simulation output data:
- **Experiment**: A complete simulation run, typically using tools like LAMMPS,
representing a specific set of conditions.
- **Lineage**: The level of an artifact in the hierarchy
``segment → whole → ensemble_long → ensemble → space``, the values of
:py:obj:`~.LineageT`; each level aggregates the one
before it:
- **segment**: a chunk of one simulation run.
- **whole**: one run, its segments joined.
- **ensemble_long**: one state point, every input parameter in the name.
- **ensemble**: one state point, its replicates (wholes with different
random seeds) collapsed into a short name.
- **space**: the parameter sweep, the state points of one study.
- **Attributes**: Macroscopic characteristics such as box size or polymer
topology that remain constant within a space.
- **Properties**: Physical features measured during the simulation, like
density or radius of gyration, with optional associated measures (e.g.,
autocorrelation).
- **Species**: The particle or molecular entity type, such as 'Mon' for
monomers, each with specific behaviors or attributes as needed.
- **Phases**: Directories organized by processing phase: `simsAll` and
`simsCont` (simulation inputs), `logs`, `trjs` (trajectories), `probes`
(probing data), `analysis` (post-processing), `viz` (visualization-ready
data), `galaxy`, and `allInOne`; the values of
:py:obj:`~.PhaseT`.
- **Stages**: Directories inside a phase, each holding the artifacts of one
lineage level or one step of the reduction: `segment`, `wholeSim`, `ens`,
`ensAvg`, `space`, and `galaxy`, the values of
:py:obj:`~.StageT`. The organizer writes them in the
`analysis` phase.
The words `segment` and `space` are both lineage levels and stages, and
`galaxy` is both a phase and a stage; which one a word is follows from where
it appears: a lineage argument, a phase directory, or the last field of a
stage directory's name. `galaxy` is not a lineage level.
Naming Conventions
==================
The :doc:`/lineage` page is the reference; in short:
- A stage directory is ``<root>/<phase>/<prefix>-<group>-<stage>``, as
:func:`make_database` builds it.
- A file is ``<lineage name>-<group>-<property>``, as
:func:`create_fullname` builds it, with ``-ensAvg`` or ``-space`` added
where two stages hold files of one level (``ens`` and ``ensAvg``) or the
level is ``space``, and then the extension. A stamps table is
``<space>-<group>-stamps-<level>.csv``, with ``<level>`` the lineage of
its rows.
- Fields are joined by ``-``. A lineage name never contains ``-``, and a
group cannot; a property may, to join a measure (``gyrTMon-acf``), so it
is everything between the group and the stage suffix or the extension.
Dependencies
============
- `numpy`: For numerical operations on arrays, such as summing or concatenating
segments.
- `pandas`: For handling tabular data and DataFrames used in data aggregation.
Contents
========
Naming and persistence:
:func:`create_fullname`, :func:`save_artifact`, and :func:`make_database`.
Building wholes from their parts:
:func:`whole_from_segments` and :func:`whole_from_file`.
Grouping wholes into ensembles:
:func:`ens_from_bin_edge`, :func:`ens_from_vec`, :func:`ens_from_mat_t`,
:func:`ens_from_df`, and :func:`ensemble`.
Averaging over an ensemble:
:func:`ens_avg_from_bin_edge`, :func:`ens_avg_from_ndarray`,
:func:`ens_avg_from_df`, and :func:`ensemble_avg`.
Space-level collection and bookkeeping:
:func:`children_stamps`, :func:`parents_stamps`, :func:`space_tseries`, and
:func:`space_hists`.
Reading tables the research code wrote:
:data:`BULK_NAME_MAP` and :func:`rename_bulk_columns`.
The density sum rule on the ``ensAvg`` profiles these reducers build is in
:mod:`polyphys.analyze.sumrule`.
Usage ===== The :mod:`~polyphys.manage.organizer` module streamlines
organization and aggregation of simulation data. Users can assemble segmented
data, organize data into ensembles, and generate ensemble-averaged data at
various aggregation stages. Its multi-level organization supports comparative
analysis across varied conditions within spaces and galaxies.
Examples
========
Two replicates (``ens1`` and ``ens2``) of one state point, each probed in
two segments (``j01`` and ``j02``), reduce to one ensemble-averaged time
series. Segment ``s`` of replicate ``e`` holds two frames of the value
``10 * e + s``:
>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> root = Path(tmp.name)
>>> state = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> paths = []
>>> for e in (1, 2):
... for s in (1, 2):
... path = root / f'{state}ens{e}.j0{s}.bug.gyrTMon.npy'
... np.save(path, np.full(2, 10.0 * e + s))
... paths.append(path)
Join each replicate's segments into a whole, collect the wholes of the
state point into an ensemble, and average over the replicates:
>>> wholes = whole_from_segments('gyrTMon', paths, ChainCyl, 'bug',
... 'tseries')
>>> wholes[state + 'ens1']
array([11., 11., 12., 12.])
>>> ensembles = ensemble('gyrTMon', wholes, ChainCyl, 'bug', 'vector')
>>> averages = ensemble_avg('gyrTMon', ensembles, 'bug', 'dataframe')
>>> averages[state]
gyrTMon-mean gyrTMon-var gyrTMon-sem
0 16.0 50.0 5.0
1 16.0 50.0 5.0
2 17.0 50.0 5.0
3 17.0 50.0 5.0
>>> tmp.cleanup()
Notes
=====
- Filenames must follow consistent patterns for the `organizer` module to
accurately parse and organize data.
- This module supports only cylindrical, slit, or cubic geometries.
- Functions in this module rely on consistent naming conventions for groups,
properties, species, and stages.
"""
import warnings
import logging
from typing import (
Any, Callable, Literal, Mapping, Sequence, Iterator, TYPE_CHECKING,
get_args,
)
import os
import pathlib
from pathlib import Path
from glob import glob
from types import MappingProxyType
import numpy as np
import pandas as pd
from .types import (
Pathish,
PathGroup,
AnyPathIter,
GroupT,
PropertyT,
ParserClassT,
StageT,
PhaseT,
WholeRelationT,
PrimitiveLineageT,
)
from .utils import invalid_keyword, sort_filenames, round_to_multiple
if TYPE_CHECKING:
# Unused here, but sphinx_autodoc_typehints runs this block to resolve
# the "ParserBase" forward reference inside ParserClassT.
from .parser import ParserBase # noqa: F401
__all__ = [
'BULK_NAME_MAP',
'children_stamps',
'create_fullname',
'ens_avg_from_bin_edge',
'ens_avg_from_df',
'ens_avg_from_ndarray',
'ens_from_bin_edge',
'ens_from_df',
'ens_from_mat_t',
'ens_from_vec',
'ensemble',
'ensemble_avg',
'make_database',
'parents_stamps',
'rename_bulk_columns',
'save_artifact',
'space_hists',
'space_tseries',
'whole_from_file',
'whole_from_segments',
]
logger = logging.getLogger(__name__)
[docs]
def create_fullname(name: str, group: GroupT, prop: PropertyT) -> str:
"""
Join a base name, a particle group, and a property into an artifact's
full name.
Parameters
----------
name : str
The base name.
group : :py:obj:`~.GroupT`
Particle group (e.g., 'bug' or 'nucleoid').
prop : :py:obj:`~.PropertyT`
Physical property name (e.g., 'density').
Returns
-------
str
A string that combines `name`, `group`, and `prop` in a
hyphen-separated format.
Examples
--------
>>> create_fullname('sample', 'bug', 'density')
'sample-bug-density'
"""
return "-".join([name, group, prop])
[docs]
def save_artifact(
filename: str,
data: np.ndarray | pd.DataFrame | dict[str, np.ndarray],
save_to: Pathish,
) -> None:
"""
Save `data` as ``<save_to>/<filename>``, a ``.npy`` or ``.csv`` file by
its type.
An existing file of the same name is overwritten.
Parameters
----------
filename : str
The output file name.
data : np.ndarray | pd.DataFrame | dict[str, np.ndarray]
Data to be saved. Accepted types:
- `np.ndarray`: Saved as ``<filename>.npy``.
- `pd.DataFrame`: Saved as ``<filename>.csv``, without the index.
- `dict` of `np.ndarray`: Each key must have the form
``<prop>-<measure>``, with exactly one hyphen, and each entry is
saved as ``<filename>-<measure>.npy``.
save_to : Pathish
Path to an existing directory where the file is saved.
Notes
-----
:func:`numpy.save` and :meth:`pandas.DataFrame.to_csv` overwrite an
existing file of the same name.
Raises
------
TypeError
If `data` type does not match any of the supported formats.
ValueError
If `data` is a dict and one of its keys does not contain exactly one
hyphen.
Examples
--------
>>> import tempfile
>>> tmp = tempfile.TemporaryDirectory()
Save a NumPy array to a ``.npy`` file:
>>> save_artifact('w-bug-rhoMon', np.array([1.0, 2.0]), tmp.name)
>>> np.load(Path(tmp.name) / 'w-bug-rhoMon.npy')
array([1., 2.])
Save a DataFrame to a ``.csv`` file:
>>> save_artifact('w-bug-rhoMon', pd.DataFrame({'A': [1, 2]}), tmp.name)
>>> pd.read_csv(Path(tmp.name) / 'w-bug-rhoMon.csv')
A
0 1
1 2
Save each entry of a dict to its own file, named by its measure:
>>> save_artifact('w-bug-rhoMon', {'rhoMon-mean': np.array([1.5])},
... tmp.name)
>>> sorted(path.name for path in Path(tmp.name).iterdir())
['w-bug-rhoMon-mean.npy', 'w-bug-rhoMon.csv', 'w-bug-rhoMon.npy']
>>> tmp.cleanup()
"""
file_path = os.path.join(save_to, filename)
# Save data based on type
if isinstance(data, pd.DataFrame):
data.to_csv(f"{file_path}.csv", index=False)
elif isinstance(data, np.ndarray):
np.save(f"{file_path}.npy", data)
elif isinstance(data, dict):
for prop_key, prop_data in data.items():
_, prop_measure = prop_key.split("-")
np.save(f"{file_path}-{prop_measure}.npy", prop_data)
else:
raise TypeError(
f"Unsupported data type {type(data).__name__}."
+ "Expected pd.DataFrame, np.ndarray, or dict of np.ndarray."
)
[docs]
def make_database(
old_database: Pathish,
phase: PhaseT,
stage: StageT,
group: GroupT | None,
) -> str:
"""
Create, if needed, the stage directory
``<root>/<phase>/<prefix>-<group>-<stage>/`` of the sweep that
`old_database` belongs to.
`old_database` is a directory of the same sweep that sits directly in a
phase directory, ``<root>/<old_phase>/<old_dir>``. The new directory
keeps ``<root>``, goes into `phase`, and is named
``<prefix>-<group>-<stage>``, with ``<prefix>`` the part of
``<old_dir>`` before its first hyphen (or ``*``) and ``<group>`` left
out when `group` is None. The :doc:`/lineage` page defines phases and
stages.
Parameters
----------
old_database : Pathish
Path to a directory of the sweep, directly inside a phase directory;
it need not exist.
phase : :py:obj:`~.PhaseT`
The new phase name for the directory, specifying its role in the
workflow: one of ``'simsAll'``, ``'simsCont'``, ``'logs'``,
``'trjs'``, ``'probes'``, ``'analysis'``, ``'viz'``, ``'galaxy'``,
or ``'allInOne'``.
stage : :py:obj:`~.StageT`
The stage of processing or data type represented in the new
directory: one of ``'segment'``, ``'wholeSim'``, ``'ens'``,
``'ensAvg'``, ``'space'``, or ``'galaxy'``.
group : GroupT or None
Particle group related to the data in the directory; left out of the
name when None. It cannot contain ``-``, the field separator of the
name.
Returns
-------
str
The path to the new directory, ending with ``/``. If the directory
already exists, the same path is returned and an INFO record naming
it is logged to the module's logger.
Raises
------
ValueError
If `phase` is not a value of `PhaseT`, `stage` not a value of
`StageT`, `group` contains ``-``, or the parent of `old_database` is
not named for a value of `PhaseT`; no directory is made.
Examples
--------
The new directory sits next to the old phase directory:
- Input: ``root/parent1/.../parentN/old_phase/old_dir``
- Result: ``root/parent1/.../parentN/phase/new_dir``
>>> import tempfile
>>> tmp = tempfile.TemporaryDirectory()
>>> old = Path(tmp.name) / 'probes' / 'N20-bug-segment'
>>> new = make_database(old, 'analysis', 'ens', 'bug')
>>> Path(new).relative_to(tmp.name).as_posix()
'analysis/N20-bug-ens'
>>> Path(new).is_dir()
True
>>> tmp.cleanup()
A name outside the phase vocabulary is refused:
>>> make_database(old, 'simAll', 'ens', 'bug')
Traceback (most recent call last):
...
ValueError: 'simAll' is an invalid option. Please select one of: \
simsAll, simsCont, logs, trjs, probes, analysis, viz, galaxy, allInOne.
"""
# The type aliases are the one declaration of both vocabularies.
invalid_keyword(phase, get_args(PhaseT))
invalid_keyword(stage, get_args(StageT))
if group is not None and "-" in group:
raise ValueError(
f"group {group!r} contains '-', the field separator of a "
"directory name.")
old_path = pathlib.Path(old_database)
old_phase = old_path.parent.name
if old_phase not in get_args(PhaseT):
raise ValueError(
f"{old_phase!r} is not a phase: the parent of {str(old_path)!r} "
"must be a phase directory, one of "
f"{', '.join(get_args(PhaseT))}.")
# Common prefix derived from old_database to create new directory name
prefix = old_path.parts[-1].split("*")[0].split("-")[0]
new_directory = "-".join([part for part in [prefix, group, stage] if part])
# Construct the new database path in the same parent directory level as
# the old one
new_database_parts = list(old_path.parts[:-2]) + [phase, new_directory]
new_database = pathlib.Path(*new_database_parts)
# Avoid double slashes at the start of the path
if str(new_database).startswith("//"):
new_database = pathlib.Path(str(new_database)[1:])
# Create the new directory if it doesn't already exist
try:
new_database.mkdir(parents=True, exist_ok=False)
except FileExistsError:
logger.info("Reusing the existing directory %s.", new_database)
return str(new_database) + "/"
def _as_path(item: Pathish | PathGroup) -> Path:
"""Normalize a single item into a pathlib.Path.
Accepts either:
- a bare pathish (str, Path, os.PathLike)
- a sequence of pathish values (we take the first element)
"""
# Bare pathish?
if isinstance(item, (str, os.PathLike)):
return Path(item)
# Sequence of pathish (e.g., tuple[Path, ...] from sort_filenames)
if isinstance(item, Sequence) and item:
first = item[0]
if isinstance(first, (str, os.PathLike)):
return Path(first)
raise TypeError(f"Unsupported path item: {type(item)!r} -> {item!r}")
def _normalize_paths(paths: AnyPathIter) -> Iterator[Path]:
"""Yield normalized Path objects from either a flat or grouped iterable."""
for item in paths:
# item is either Pathish or PathGroup; _as_path handles both
yield _as_path(item)
def _assert_equal_bin_edges(arrays: Sequence[np.ndarray]) -> np.ndarray:
"""
Ensure all input arrays of bin edges are identical.
Parameters
----------
arrays : Sequence[np.ndarray]
Sequence of 1D arrays representing bin edges from different wholes
or segments.
Returns
-------
np.ndarray
A copy of the first array if all arrays are identical.
Raises
------
ValueError
If any array differs from the first one.
"""
ref = arrays[0]
if not all(np.array_equal(ref, a) for a in arrays[1:]):
raise ValueError("Bin edges are not identical across all arrays.")
return ref.copy()
def _assert_equal_lengths(ensemble_name: str, lengths: dict[str, int]) -> None:
"""
Raise ``ValueError`` unless every whole of `ensemble_name` has the same
length; `lengths` maps each whole's name to its number of rows or
values. Replicates of one state point run the same number of steps, so
a shorter whole is an incomplete run, not one to average over. The cost
is :math:`O(n)` in the number of wholes.
"""
if len(set(lengths.values())) > 1:
listed = ", ".join(f"{name} {n}" for name, n in lengths.items())
raise ValueError(
f"The wholes of ensemble '{ensemble_name}' have unequal "
f"lengths: {listed}.")
[docs]
def whole_from_segments(
prop: PropertyT,
segments: AnyPathIter,
parser: ParserClassT,
group: GroupT,
relation: WholeRelationT,
save_to: Pathish | None = None,
) -> dict[str, np.ndarray]:
"""
Combine the segment files of one property into one array per whole.
How the segments of a whole combine is set by `relation`.
Parameters
----------
prop : :py:obj:`~.PropertyT`
The property as it appears in file names, such as ``'gyrTMon'``.
segments : :py:obj:`~.AnyPathIter`
An iterable of paths to segment ``.npy`` files for `prop`, either
bare paths or 1-tuples of a path, as
:func:`~polyphys.manage.utils.sort_filenames` returns them. Each path
points to a single segment file. Segments of different wholes may be
interleaved, but the segments of each whole must come in
consecutive ``segment_id`` order (the ``j<NN>`` keyword of their
names), with no id skipped.
parser : :py:obj:`~.ParserClassT`
A parser class from :mod:`polyphys.manage.parser` that interprets
file paths and extracts attributes such as the `whole` name and the
``segment_id``.
group : :py:obj:`~.GroupT`
Type of the particle group.
relation : :py:obj:`~.WholeRelationT`
Specifies how to combine segments into a whole. Accepted values:
- 'histogram'
'segment' is an N-dimensional histogram-like file, so the segments
of a 'whole' are summed element-wise. In the two-dimensional
space, a histogram has a (x_nbins, y_nbins) shape.
- 'tseries'
'segment' is a time-series-like file, so the segments of a 'whole'
are concatenated along axis 0, in ``segment_id`` order.
- 'bin_edge'
'segment' is a bin-edge file. All the segments of a whole must
hold identical edges, and one copy of them is the whole's
bin-edge array.
save_to : Pathish, optional
Directory path where the output files are saved, if specified. Each
whole is saved as ``<whole>-<group>-<prop>.npy``, overwriting a file
of the same name.
Returns
-------
dict[str, np.ndarray]
Dictionary with whole names as keys and the combined whole data as
NumPy arrays, in the order in which the wholes first appear in
`segments`.
Raises
------
ValueError
If `relation` is not one of the accepted values; if the segments of
a whole do not arrive in strictly increasing ``segment_id`` order,
which includes a segment passed twice; if they skip a
``segment_id``, such as ``j01`` then ``j03``, which would shift the
time of every later frame or leave a part out of a sum; or, for
'bin_edge', if the segments of a whole hold different edges.
TypeError
If an item of `segments` is neither a path nor a sequence whose first
element is a path.
Notes
-----
The order of a whole's segments is part of the data's meaning for
'tseries', and a duplicated segment would be counted twice by
'histogram'. This function checks the order and that no id is skipped;
it does not require the first id to be 1. Sort the paths with
:func:`~polyphys.manage.utils.sort_filenames` first, which orders the
``j<NN>`` keyword naturally.
Let n_segments be the number of segment files and m the number of
values in one segment. Time is O(n_segments * m). Every segment of
every whole is loaded and held in memory before any whole is reduced,
so peak memory is O(n_segments * m), not the size of one whole; the
returned wholes add O(n_wholes * m) for 'histogram' and 'bin_edge' and
O(n_segments * m) for 'tseries', where n_wholes is the number of
wholes.
Examples
--------
Two segments of one whole, written as the probe phase names them, join
into one time series in ``segment_id`` order:
>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> whole = ('N20r5.5lz25.0ac1.0nc100dt0.005bdump1000'
... 'adump5000ens1')
>>> paths = []
>>> for seg_id in (1, 2):
... path = Path(tmp.name) / f'{whole}.j0{seg_id}.bug.gyrTMon.npy'
... np.save(path, np.array([10.0, 11.0]) * seg_id)
... paths.append(path)
>>> wholes = whole_from_segments('gyrTMon', paths, ChainCyl, 'bug',
... 'tseries')
>>> wholes[whole]
array([10., 11., 20., 22.])
'histogram' sums the same segments element-wise:
>>> whole_from_segments('gyrTMon', paths, ChainCyl, 'bug',
... 'histogram')[whole]
array([30., 33.])
Segments out of order raise rather than giving a reversed series:
>>> whole_from_segments('gyrTMon', paths[::-1], ChainCyl, 'bug',
... 'tseries') # doctest: +ELLIPSIS
Traceback (most recent call last):
...
ValueError: The segments of whole '...ens1' are not in ... [2, 1]...
>>> tmp.cleanup()
"""
invalid_keyword(relation, ["histogram", "tseries", "bin_edge"])
# Combine segments into wholes based on segment names and relation type,
# checking that each whole's segment_id values strictly increase
grouped_segments: dict[str, list[np.ndarray]] = {}
segment_ids: dict[str, list[int]] = {}
for seg_path in _normalize_paths(segments):
seg_info = parser(seg_path, "segment", group, ispath=True)
whole_name = getattr(seg_info, "whole")
seg_id = getattr(seg_info, "segment_id")
whole_ids = segment_ids.setdefault(whole_name, [])
if whole_ids and seg_id <= whole_ids[-1]:
raise ValueError(
f"The segments of whole '{whole_name}' are not in strictly "
f"increasing segment_id order: {whole_ids + [seg_id]}. Sort "
"them with sort_filenames first."
)
if whole_ids and seg_id != whole_ids[-1] + 1:
raise ValueError(
f"The segments of whole '{whole_name}' skip a segment_id: "
f"{whole_ids + [seg_id]}."
)
whole_ids.append(seg_id)
grouped_segments.setdefault(whole_name, []).append(np.load(seg_path))
# Define reduction strategies for each relation type
reducers: dict[str, Callable[[Sequence[np.ndarray]], np.ndarray]] = {
"histogram": lambda arrs: np.sum(arrs, axis=0),
"tseries": lambda arrs: np.concatenate(arrs, axis=0),
"bin_edge": _assert_equal_bin_edges,
}
reducer = reducers[relation]
# Apply the appropriate mapping function to merge each group of segments
# into a whole.
wholes: dict[str, np.ndarray] = \
{name: reducer(vals) for name, vals in grouped_segments.items()}
if save_to is not None:
for name, data in wholes.items():
save_artifact(create_fullname(name, group, prop), data, save_to)
return wholes
[docs]
def whole_from_file(
whole_paths: AnyPathIter,
parser: ParserClassT,
group: GroupT,
) -> dict[str, np.ndarray]:
"""
Load the whole ``.npy`` files of one property of a particle `group`.
Each path in `whole_paths` points to one whole file.
Parameters
----------
whole_paths : AnyPathIter
Paths to whole files, either bare paths or 1-tuples of a path, as
:func:`~polyphys.manage.utils.sort_filenames` returns them. Each file
is loaded independently.
parser : ParserClassT
A parser class from :mod:`polyphys.manage.parser` that interprets
file paths and extracts attributes such as the `whole` name.
group : GroupT
The particle group to which the data pertains.
Returns
-------
dict[str, np.ndarray]
Dictionary where keys are the `whole` names (derived from each file
path) and values are NumPy arrays containing the loaded data.
Raises
------
TypeError
If an item of `whole_paths` is neither a path nor a sequence whose
first element is a path.
ValueError
If two files parse to the same whole name; the message names both.
Notes
-----
This function assumes that each file in `whole_paths` contains data
related to a complete simulation, referred to as a "whole." For further
information on terminology, see the `organizer` module documentation.
Let n_wholes be the number of files and m the number of values in one
file. Time and memory are O(n_wholes * m): every whole is loaded and
held in memory at once.
Examples
--------
>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> whole = ('N20r5.5lz25.0ac1.0nc100dt0.005bdump1000'
... 'adump5000ens1')
>>> path = Path(tmp.name) / f'{whole}-bug-gyrTMon.npy'
>>> np.save(path, np.array([1.0, 2.0, 3.0]))
>>> whole_from_file([path], ChainCyl, 'bug')[whole]
array([1., 2., 3.])
>>> tmp.cleanup()
"""
wholes: dict[str, np.ndarray] = {}
sources: dict[str, Path] = {}
for whole_path in _normalize_paths(whole_paths):
whole_info = parser(whole_path, "whole", group, ispath=True)
name = getattr(whole_info, "whole")
if name in sources:
raise ValueError(
f"'{sources[name]}' and '{whole_path}' are both whole "
f"'{name}'.")
sources[name] = whole_path
wholes[name] = np.load(whole_path)
return wholes
[docs]
def ens_from_bin_edge(
ens_data: tuple[str, dict[str, np.ndarray]],
) -> tuple[str, np.ndarray]:
"""
Reduce the bin edges of the 'whole' datasets in an 'ensemble' to the
single array they share.
Every 'whole' in an ensemble must be binned identically, so that the
histograms paired with these edges stay aligned bin by bin. The edges are
checked for equality with ``_assert_equal_bin_edges``, the check
:func:`whole_from_segments` applies to segments, rather than merged.
Parameters
----------
ens_data : tuple[str, dict[str, np.ndarray]]
A tuple where the first element is the ensemble name (str) and the
second element is a non-empty dictionary of bin edge arrays from
different 'whole' datasets within that ensemble. Within the
dictionary, keys are whole names while values are 1D arrays of whole
bin edges, each of shape ``(n_bins + 1,)``.
Returns
-------
tuple[str, np.ndarray]
A tuple where the first element is the ensemble name, and the second
is a copy of the shared 1D array of bin edges, of shape
``(n_bins + 1,)``.
Raises
------
ValueError
If the bin edges of any 'whole' differ from those of the first one,
in length or in value.
Notes
-----
Time and space complexity are O(n_wholes * n_edges) and O(n_edges).
Examples
--------
>>> ens_from_bin_edge(
... ('ens1', {'w1': np.array([0., 1., 2.]),
... 'w2': np.array([0., 1., 2.])})
... )
('ens1', array([0., 1., 2.]))
Differing edges raise instead of being merged:
>>> ens_from_bin_edge(
... ('ens1', {'w1': np.array([0., 1., 2.]),
... 'w2': np.array([0., 1., 3.])})
... )
Traceback (most recent call last):
...
ValueError: Bin edges are not identical across all arrays.
"""
ensemble_name, whole_edges = ens_data
shared_bin_edges = _assert_equal_bin_edges(list(whole_edges.values()))
return ensemble_name, shared_bin_edges
[docs]
def ens_from_vec(
ens_data: tuple[str, dict[str, np.ndarray]],
) -> tuple[str, pd.DataFrame]:
"""
Merge the 1D arrays of the wholes of one ensemble into a DataFrame.
Each whole becomes one column, such as a time series or a histogram of
the whole.
Parameters
----------
ens_data : tuple[str, dict[str, np.ndarray]]
A tuple where the first element is the ensemble name, and the second
is a dictionary mapping each 'whole' dataset name to a 1D NumPy array
of measurements.
Returns
-------
tuple[str, pd.DataFrame]
A tuple where the first element is the ensemble name, and the second
is a pandas DataFrame. Each column represents a 'whole' dataset, in
the order of the dictionary, and rows contain corresponding vector
values from each dataset.
Raises
------
ValueError
If the vectors have unequal lengths; the message names the ensemble
and each whole's length.
Notes
-----
Let n_wholes be the number of wholes and m the length of one vector.
Time and extra memory are O(n_wholes * m): the vectors are copied into
the DataFrame.
Examples
--------
>>> ens_name, ensemble_df = ens_from_vec(
... ('ensemble1', {'whole1': np.array([0.1, 0.2, 0.3]),
... 'whole2': np.array([0.4, 0.5, 0.6])})
... )
>>> ens_name
'ensemble1'
>>> ensemble_df
whole1 whole2
0 0.1 0.4
1 0.2 0.5
2 0.3 0.6
"""
ensemble_name, whole_vectors = ens_data
_assert_equal_lengths(
ensemble_name, {name: len(v) for name, v in whole_vectors.items()})
ensemble_df = pd.DataFrame.from_dict(whole_vectors, orient="columns")
return ensemble_name, ensemble_df
[docs]
def ens_from_mat_t(
ens_data: tuple[str, dict[str, np.ndarray]],
) -> tuple[str, np.ndarray]:
"""
Stack the 2D arrays of the wholes of one ensemble into a 3D array.
Each whole becomes one slice along the first axis.
Parameters
----------
ens_data : tuple[str, dict[str, np.ndarray]]
A tuple where the first element is the ensemble name (str) and the
second element is a dictionary with 'whole' names as keys and 2D NumPy
arrays as values.
Returns
-------
tuple[str, np.ndarray]
A tuple where the first element is the ensemble name and the second
is a 3D NumPy array of shape ``(n_wholes, n_rows, n_cols)``, formed
by stacking the 2D arrays along the first axis in the order of the
dictionary.
Notes
-----
Let n_wholes be the number of wholes and m the number of elements of one
matrix. Time and extra memory are O(n_wholes * m): the matrices are
copied into the stacked array.
Examples
--------
>>> ens_name, combined = ens_from_mat_t(
... ('ensemble1', {'whole1': np.array([[1, 2], [3, 4]]),
... 'whole2': np.array([[5, 6], [7, 8]])})
... )
>>> combined.shape
(2, 2, 2)
>>> combined[1]
array([[5, 6],
[7, 8]])
"""
ensemble_name, whole_matrices = ens_data
combined_matrix = np.stack(list(whole_matrices.values()), axis=0)
return ensemble_name, combined_matrix
[docs]
def ens_from_df(
ens_data: tuple[str, dict[str, pd.DataFrame]],
) -> tuple[str, pd.DataFrame]:
"""
Average the DataFrames of the wholes of one ensemble into one
DataFrame.
Each DataFrame is one whole of the ensemble.
In a 'whole' DataFrame, headers are 'elements' of a 'matrix' or 2D
quantity and columns are the values of a given property.
In each 'whole' DataFrame, headers represent 'elements' of a 2D quantity
(matrix-like structure), and columns contain the values for a specified
property. Each 'whole' DataFrame may be either a 'histogram' or a
'time-series':
- In a 'histogram' DataFrame, an additional column, `bin_center`, holds
bin center values that are invariant under averaging and are retained
in the output DataFrame. The length of histogram DataFrame is equal to
the number of bins.
- In a 'time-series' DataFrame, headers (columns) are elements of a matrix,
and rows represent the values of each element over time. The length of
timeseries DataFrame is equal to the number of time frames.
Parameters
----------
ens_data : tuple[str, dict[str, pd.DataFrame]]
A tuple where the first element is the ensemble name and the second
is a dictionary with 'whole' names as keys and DataFrames, sharing
their columns and index labels, as values.
Returns
-------
tuple[str, pd.DataFrame]
A tuple with:
- The ensemble name (str).
- A DataFrame with the averaged values across all 'whole'
DataFrames.
Raises
------
ValueError
If the DataFrames have unequal numbers of rows; the message names
the ensemble and each whole's length. A shorter whole is not
averaged over the rows it has.
Notes
-----
A clear distinction exists between ensembles of 'vector' or 'matrix' types
and ensembles of 'dataframe' type. In vector or matrix types, each column
in the resulting ensemble DataFrame is named after a 'whole' and represents
that whole's values for a specific property. In contrast, for DataFrame
type ensembles, the columns are elements of a 2D quantity (e.g., matrix or
time-series values) and hold averaged values across all whole DataFrames.
This function assumes that averaging non-element headers, such as
'bin_center,' does not change their values, and these are carried forward
without modification.
Let n_wholes be the number of wholes, n_rows the rows and n_cols the
columns of one DataFrame. Time and extra memory are
O(n_wholes * n_rows * n_cols): the DataFrames are concatenated into one
before they are grouped by index label and averaged.
Examples
--------
>>> df1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
>>> df2 = pd.DataFrame({'A': [2, 3, 4], 'B': [5, 6, 7]})
>>> ens_name, averaged_df = ens_from_df(
... ('ensemble1', {'whole1': df1, 'whole2': df2})
... )
>>> averaged_df
A B
0 1.5 4.5
1 2.5 5.5
2 3.5 6.5
"""
ensemble_name, whole_data = ens_data
_assert_equal_lengths(
ensemble_name, {name: len(df) for name, df in whole_data.items()})
ensemble_df = pd.concat(list(whole_data.values())).groupby(level=0).mean()
return ensemble_name, ensemble_df
[docs]
def ensemble(
prop: PropertyT,
wholes: dict[str, np.ndarray] | dict[str, pd.DataFrame],
parser: ParserClassT,
group: GroupT,
whole_type: Literal["vector", "matrix", "dataframe", "bin_edge"],
whole_edges: dict[str, np.ndarray] | None = None,
save_to: Pathish | None = None,
) -> dict[str, np.ndarray | pd.DataFrame]:
"""
Merge the wholes of one property into one ensemble per state point.
`whole_type` sets the structure of each whole and how the wholes of one
ensemble merge.
Parameters
----------
prop : :py:obj:`~.PropertyT`
The property as it appears in file names, such as ``'rhoMon'``.
wholes : dict[str, np.ndarray] | dict[str, pd.DataFrame]
Dictionary containing 'whole' names as keys and their corresponding
data as values, represented as NumPy arrays or DataFrames.
parser : ParserClassT
Parser class used to interpret whole filenames, such as the ensemble
name of each 'whole'.
group : GroupT
Type of the particle group being processed.
whole_type : {'vector', 'matrix', 'dataframe', 'bin_edge'}
Specifies the type of each 'whole' and defines the merging method:
- 'vector': A 1D NumPy array, such as a time series or histogram.
- 'matrix': A 2D NumPy array, such as a gyration tensor.
- 'dataframe': A DataFrame representing either a time series (with
headers as matrix elements and rows as values over time) or a
histogram (headers as elements, rows as counts or frequencies,
and an additional 'bin_center' column for bin centers).
- 'bin_edge': A 1D array representing bin edges. Assumes all bin edges
are identical and selects one to represent the ensemble.
whole_edges : dict[str, np.ndarray], optional
Dictionary with 'whole' names as keys and 1D bin edge arrays, each of
shape ``(n_bins + 1,)``, as values. Only valid when `whole_type` is
'vector'. The wholes of an ensemble share their bin edges by design,
so the edges of the first whole of each ensemble are used and the
others are not compared. Each merged ensemble gets a 'bin_center'
column of length ``n_bins`` holding the midpoints of those edges.
Must contain every key of `wholes`.
save_to : Pathish, optional
Directory path where output files are saved, if specified; see
Notes.
Returns
-------
dict[str, np.ndarray | pd.DataFrame]
The ensembles, keyed by their ``ensemble_long`` names, as NumPy
arrays or DataFrames.
Raises
------
ValueError
If any parameter has an invalid value; if `whole_edges` is given and
`whole_type` is not 'vector'; or if the number of bins differs from
the number of rows of the merged ensemble.
Notes
-----
Different averaging methods apply based on `whole_type`. When `whole_type`
is "DataFrame," the resulting DataFrame averages the values of each matrix
element across wholes, retaining headers that are mean-invariant (e.g.,
'bin_center'). For 'vector' and 'matrix' types, columns represent
individual wholes in the ensemble DataFrame.
The bin center of bin ``i`` is the midpoint of its two edges,
``0.5 * (edges[i] + edges[i + 1])``. A center is invariant under
averaging over wholes, which is why :func:`ensemble_avg` excludes
'bin_center' from its statistics by default.
Let n_wholes be the number of wholes, m the number of values in one
whole, n_ensembles the number of ensembles, and n_edges the length of
one bin-edge array. Grouping parses each whole name once and merging
copies every whole once, so time and extra memory are
O(n_wholes * m): the caller's `wholes` and the merged ensembles are both
held in memory. With `whole_edges`, computing the centers adds
O(n_ensembles * n_edges) time and memory.
With `save_to`, each ensemble is saved by :func:`save_artifact` under
``<ensemble_long>-<group>-<prop>``, as ``.csv`` for a DataFrame and
``.npy`` for an array, overwriting a file of the same name.
Examples
--------
Two replicates of one state point, with shared edges, merge into one
ensemble that carries the edge midpoints as 'bin_center':
>>> from polyphys.tests.synthetic import ChainCyl
>>> base = ('N2000r10.5lz300.0ac1.0nc117000dt0.005bdump5000'
... 'adump10000ens')
>>> wholes = {base + '1': np.array([1., 2., 3.]),
... base + '2': np.array([3., 4., 5.])}
>>> edges = {name: np.array([0., 1., 2., 4.]) for name in wholes}
>>> ens = ensemble('rhoMon', wholes, ChainCyl, 'bug', 'vector',
... whole_edges=edges)
>>> ens_df = next(iter(ens.values()))
>>> ens_df['bin_center'].tolist()
[0.5, 1.5, 3.0]
Without `whole_edges`, the ensemble has one column per whole only:
>>> ens = ensemble('rhoMon', wholes, ChainCyl, 'bug', 'vector')
>>> 'bin_center' in next(iter(ens.values())).columns
False
"""
invalid_keyword(whole_type, ["vector", "matrix", "dataframe", "bin_edge"])
if whole_edges is not None and whole_type != "vector":
raise ValueError(
"'whole_edges' is only valid with whole_type 'vector', not "
f"'{whole_type}'."
)
merging_func: dict[
str, Callable[..., tuple[str, np.ndarray | pd.DataFrame]]
] = {
"vector": ens_from_vec,
"matrix": ens_from_mat_t,
"dataframe": ens_from_df,
"bin_edge": ens_from_bin_edge,
}
# Collecting wholes for each ensemble; the wholes of an ensemble share
# their bin edges by design, so the centers come from its first whole
ens_wholes: dict[str, dict[str, np.ndarray | pd.DataFrame]] = {}
bin_centers: dict[str, np.ndarray] = {}
for whole_name, whole_data in wholes.items():
whole_info = parser(whole_name, "whole", group)
ens_name = getattr(whole_info, "ensemble_long")
ens_wholes.setdefault(ens_name, {})[whole_name] = whole_data
if whole_edges is not None and ens_name not in bin_centers:
edges = np.asarray(whole_edges[whole_name])
bin_centers[ens_name] = 0.5 * (edges[:-1] + edges[1:])
# Merging wholes into a single ensemble data object
ensembles: dict[str, np.ndarray | pd.DataFrame] = dict(
map(merging_func[whole_type], ens_wholes.items())
)
for ens_name, centers in bin_centers.items():
ensembles[ens_name]["bin_center"] = centers
# Save ensemble data object if save_to is specified
if save_to:
for ens_name, data in ensembles.items():
ens_fullname = create_fullname(ens_name, group, prop)
save_artifact(ens_fullname, data, save_to)
return ensembles
[docs]
def ens_avg_from_bin_edge(
ens_data: np.ndarray | Sequence[np.ndarray],
) -> np.ndarray:
"""
Return the bin edges of an 'ensemble' for its ensemble-averaged
artifact.
Bin edges are not averaged: every 'whole' in the ensemble must share
them. A single 1D array, the form :func:`ens_from_bin_edge` produces and
:func:`ensemble_avg` passes here, is returned as a copy. A sequence of
arrays, one per 'whole', is checked for equality with
``_assert_equal_bin_edges``, and a copy of the shared array is
returned.
Parameters
----------
ens_data : np.ndarray or Sequence[np.ndarray]
Either the ensemble's 1D array of bin edges, of shape
``(n_bins + 1,)``, or a non-empty sequence of such arrays, one per
'whole'. A 2D array of shape ``(n_wholes, n_bins + 1)`` counts as a
sequence of its rows.
Returns
-------
np.ndarray
A copy of the shared 1D array of bin edges, of shape
``(n_bins + 1,)``.
Raises
------
ValueError
If `ens_data` is a sequence and any array in it differs from the
first one, in length or in value.
Notes
-----
This function performs no averaging or statistical calculation. Time and
space complexity are O(n_wholes * n_edges) and O(n_edges).
Examples
--------
>>> ens_avg_from_bin_edge(np.array([0.0, 0.1, 0.2]))
array([0. , 0.1, 0.2])
>>> ens_avg_from_bin_edge(
... [np.array([0.0, 0.1, 0.2]), np.array([0.0, 0.1, 0.2])]
... )
array([0. , 0.1, 0.2])
>>> ens_avg_from_bin_edge(
... [np.array([0.0, 0.1, 0.2]), np.array([0.0, 0.1, 0.3])]
... )
Traceback (most recent call last):
...
ValueError: Bin edges are not identical across all arrays.
"""
if isinstance(ens_data, np.ndarray) and ens_data.ndim == 1:
return ens_data.copy()
return _assert_equal_bin_edges([np.asarray(edges) for edges in ens_data])
[docs]
def ens_avg_from_ndarray(
ens_prop: PropertyT, ens_data: np.ndarray
) -> dict[str, np.ndarray]:
"""
Compute the mean, variance, and standard error of the mean (SEM)
across the wholes of an ensemble 3D array.
Each slice along the first axis (axis=0) is one whole's matrix, and the
statistics are taken along that axis.
Parameters
----------
ens_prop : PropertyT
The name of the property to which this 'ensemble' data belongs. Used
as a prefix for the computed statistics.
ens_data : np.ndarray
A 3D NumPy array where each matrix slice along axis=0 represents a
'whole' dataset.
Returns
-------
dict[str, np.ndarray]
A dictionary where keys are statistical measures (mean, variance, SEM)
with `ens_prop` as a prefix while values are NumPy arrays of the
computed statistics across 'whole' matrices.
Notes
-----
- The SEM is the sample standard deviation with ``ddof=1`` divided by
the square root of the number of wholes, which are assumed
independent. Bessel's correction makes the variance unbiased; the
standard deviation is still biased, though less (the
:func:`numpy.std` documentation, Notes), and so is the SEM.
- This function performs averaging along axis=0 for each element across
the 'whole' data matrices.
- Let n_wholes be the length of axis 0 and m the number of elements of
one whole. Time is O(n_wholes * m). NumPy's ``var`` and ``std`` build
temporaries of the size of `ens_data`, so peak extra memory is
O(n_wholes * m); the three results take O(m).
Examples
--------
>>> ens_data = np.array([
... [[1.0, 2.0], [3.0, 4.0]],
... [[2.0, 3.0], [4.0, 5.0]],
... ])
>>> ens_avg = ens_avg_from_ndarray('gyrTMon', ens_data)
>>> list(ens_avg)
['gyrTMon-mean', 'gyrTMon-var', 'gyrTMon-sem']
>>> ens_avg['gyrTMon-mean']
array([[1.5, 2.5],
[3.5, 4.5]])
>>> ens_avg['gyrTMon-sem']
array([[0.5, 0.5],
[0.5, 0.5]])
"""
n_wholes_sqrt = ens_data.shape[0] ** 0.5
ens_avg = {
f"{ens_prop}-mean": np.mean(ens_data, axis=0),
f"{ens_prop}-var": np.var(ens_data, axis=0, ddof=1),
f"{ens_prop}-sem": np.std(ens_data, axis=0, ddof=1) / n_wholes_sqrt,
}
return ens_avg
[docs]
def ens_avg_from_df(
ens_prop: PropertyT, ens_data: pd.DataFrame, exclude: list[str]
) -> pd.DataFrame:
"""
Compute the mean, variance, and standard error of the mean (SEM) across
the whole columns of an ensemble DataFrame.
The result keeps the excluded columns, such as ``bin_center``, and
replaces the whole columns with the statistics.
Parameters
----------
ens_prop : PropertyT
The name of the property to which this 'ensemble' data belongs, used
as a prefix for the generated statistics columns.
ens_data : pd.DataFrame
A DataFrame containing ensemble data. Each 'whole' dataset is in its
own column, and any columns specified in `exclude` remain unaffected.
exclude : list of str
A list of column names to exclude from averaging. For example,
`exclude` might contain 'bin_center' for histogram-type ensembles.
Returns
-------
pd.DataFrame
A new DataFrame with the excluded columns of `ens_data`, followed by
the columns ``<ens_prop>-mean``, ``<ens_prop>-var``, and
``<ens_prop>-sem``: the mean, variance, and SEM of each row across
the 'whole' data columns. It has the same index as `ens_data`.
Notes
-----
- `ens_data` is not modified; the statistics are written to a new
DataFrame.
- The function averages only the 'whole' columns, excluding any specified
in `exclude`.
- The standard error of the mean (SEM) is computed from the sample
standard deviation with ``ddof=1``, the wholes assumed independent.
Bessel's correction makes the variance unbiased; the standard
deviation is still biased, though less (the :func:`numpy.std`
documentation, Notes), and so is the SEM.
- Let n_rows be the number of rows of `ens_data` and n_wholes the number
of 'whole' columns. Time is O(n_rows * n_wholes). Each statistic
selects the 'whole' columns as a new DataFrame, so peak extra memory
is O(n_rows * n_wholes), not O(n_rows); the result takes O(n_rows).
Examples
--------
>>> ens_data = pd.DataFrame({
... 'bin_center': [1, 2, 3],
... 'whole1': [4.0, 5.0, 6.0],
... 'whole2': [5.0, 6.0, 7.0],
... })
>>> ens_avg_from_df('density', ens_data, ['bin_center'])
bin_center density-mean density-var density-sem
0 1 4.5 0.5 0.5
1 2 5.5 0.5 0.5
2 3 6.5 0.5 0.5
>>> list(ens_data.columns)
['bin_center', 'whole1', 'whole2']
"""
wholes = [col for col in ens_data.columns if col not in exclude]
ens_avg = ens_data.drop(columns=wholes)
ens_avg[ens_prop + "-mean"] = ens_data[wholes].mean(axis=1)
ens_avg[ens_prop + "-var"] = ens_data[wholes].var(axis=1, ddof=1)
ens_avg[ens_prop + "-sem"] = ens_data[wholes].sem(axis=1, ddof=1)
return ens_avg
[docs]
def ensemble_avg(
prop: PropertyT,
ensembles: dict[str, np.ndarray | pd.DataFrame],
group: GroupT,
ens_type: Literal["dataframe", "ndarray", "bin_edge"],
exclude: list[str] | None = None,
save_to: Pathish | None = None,
) -> dict[str, np.ndarray] | dict[str, pd.DataFrame]:
"""
Average the wholes of each ensemble of one property.
Columns listed in `exclude` are left out of the averages.
Parameters
----------
prop : :py:obj:`~.PropertyT`
The property as it appears in file names, such as ``'rhoMon'``.
ensembles : dict[str, np.ndarray | pd.DataFrame]
A dictionary where each key is an ensemble name and each value is
either a DataFrame or ndarray representing 'whole' data.
group : GroupT
Type of the particle group.
ens_type : {'dataframe', 'ndarray', 'bin_edge'}
The data format for each ensemble:
- 'dataframe': DataFrame format, with columns as 'whole' simulations
or items in `exclude`.
- 'ndarray': 3D ndarray format, with each slice along axis=0
representing a 'whole' matrix.
- 'bin_edge': 1D ndarray format for bin edges, with bin edges assumed
to be identical across wholes.
exclude : list[str], default None
List of column names or items not included in ensemble averaging,
such as 'bin_center' for histograms.
save_to : Pathish, optional
Path to the directory where the averaged ensemble data is saved, if
specified; see Notes.
Returns
-------
dict[str, np.ndarray] | dict[str, pd.DataFrame]
A dictionary where each key is an ensemble name, and each value is a
DataFrame or ndarray of ensemble-averaged data.
Raises
------
ValueError
If `ens_type` is not one of the accepted values.
Notes
-----
With `save_to`, each result is saved by :func:`save_artifact` under
``<ensemble>-<group>-<prop>-ensAvg``: a DataFrame as ``.csv``, bin edges
as ``.npy``, and the dictionary of an 'ndarray' ensemble as one
``...-ensAvg-<measure>.npy`` file per statistic, each overwriting a file
of the same name.
Let n_ensembles be the number of ensembles. Ensembles are averaged one
after another, with the time and peak extra memory of
:func:`ens_avg_from_df`, :func:`ens_avg_from_ndarray`, or
:func:`ens_avg_from_bin_edge` for the largest one; the results for all
n_ensembles ensembles are held in memory together.
Examples
--------
>>> ensembles = {
... 'ensemble1': pd.DataFrame({'whole1': [1.0, 2.0],
... 'whole2': [3.0, 4.0]}),
... 'ensemble2': pd.DataFrame({'whole1': [5.0, 6.0],
... 'whole2': [5.0, 8.0]}),
... }
>>> avg = ensemble_avg('rhoMon', ensembles, 'bug', 'dataframe')
>>> avg['ensemble2']
rhoMon-mean rhoMon-var rhoMon-sem
0 5.0 0.0 0.0
1 7.0 2.0 1.0
"""
if exclude is None:
exclude = ["bin_center"]
# Compute ensemble averages
ens_avgs: dict[str, np.ndarray] | dict[str, pd.DataFrame] = {}
if ens_type == "bin_edge":
for ens_name, ens_data in ensembles.items():
ens_avg = ens_avg_from_bin_edge(ens_data) # type: ignore
ens_avgs[ens_name] = ens_avg # type: ignore
elif ens_type == "ndarray":
for ens_name, ens_data in ensembles.items():
ens_avg = ens_avg_from_ndarray(prop, ens_data) # type: ignore
ens_avgs[ens_name] = ens_avg # type: ignore
elif ens_type == "dataframe":
for ens_name, ens_data in ensembles.items():
ens_avg = ens_avg_from_df(prop, ens_data, exclude) # type: ignore
ens_avgs[ens_name] = ens_avg # type: ignore
else:
invalid_keyword(ens_type, ["dataframe", "ndarray", "bin_edge"])
# Save averaged ensemble data if save_to path is provided
if save_to is not None:
save_property = prop + "-ensAvg"
for ens_name, data in ens_avgs.items():
ens_fullname = create_fullname(ens_name, group, save_property)
save_artifact(ens_fullname, data, save_to)
return ens_avgs
def _group_from_stamp_names(stamps: list[tuple[str]]) -> str:
"""
The particle group in the names of stamp files,
``<lineage name>-<group>-stamps.csv``, which must agree.
Parameters
----------
stamps : list[tuple[str]]
List of 1-tuples, each holding the path to a 'stamp' CSV file.
Returns
-------
str
The group the file names share.
Raises
------
ValueError
If a file name is not ``<name>-<group>-stamps.csv``, or the names
give more than one group.
"""
groups: set[str] = set()
for (path,) in stamps:
name = Path(path).name
parts = name.removesuffix(".csv").rsplit("-", 2)
if len(parts) != 3 or parts[2] != "stamps":
raise ValueError(
f"'{name}' is not a '<name>-<group>-stamps.csv' stamp file, "
"and the stamps have no 'group' column.")
groups.add(parts[1])
if len(groups) != 1:
raise ValueError(
f"The stamp file names give {sorted(groups)}, not one group.")
return groups.pop()
[docs]
def children_stamps(
stamps: list[tuple[str]],
child_lineage: PrimitiveLineageT,
save_to: Pathish | None = None,
) -> pd.DataFrame:
"""
Combine the stamp files of the artifacts of one lineage, `child_lineage`,
into one DataFrame for a *space*.
With `save_to`, the DataFrame is also saved as a CSV file.
Parameters
----------
stamps : list[tuple[str]]
List of 1-tuples, as :func:`~polyphys.manage.utils.sort_filenames`
returns them, each holding the path to a 'stamp' CSV file. Rows are
concatenated in this order.
child_lineage : {'segment', 'whole'}
Lineage of the stamp files, and so of the rows of the result.
save_to : Pathish, optional
Directory path where the combined DataFrame is saved, if specified,
as ``<space>-<group>-stamps-<child_lineage>.csv``, with ``space`` and
``group`` read from the first row; a file of the same name is
overwritten.
Returns
-------
pd.DataFrame
Combined DataFrame containing data from all the 'stamp' files, with
a fresh ``0..n-1`` index, and a ``group`` column (see Notes).
Raises
------
ValueError
If `child_lineage` is not one of the accepted values; or if the
stamps have no ``group`` column and a file name is not
``<name>-<group>-stamps.csv``, or the names give more than one
group.
Warns
-----
UserWarning
If `child_lineage` is 'whole' and the 'segment' and 'segment_id'
columns are dropped.
Notes
-----
- It is assumed that all the stamp files have belong to the same *group* in
a given *space*.
- If `child_lineage` is 'whole', columns 'segment' and 'segment_id' are
removed, as they are redundant for whole-lineage data, with a
`UserWarning`.
Either may be absent, as in stamps the probe phase writes for a whole;
if neither is present, nothing is dropped and a DEBUG record is
logged.
- The stamps :func:`~polyphys.probe.core.write_stamps` writes carry a
``group`` column; the research code's stamps did not, and held the
group in the file name, ``<lineage name>-<group>-stamps.csv``. When
the column is missing it is added from the file names, after
``space`` when that column is present and last otherwise, so the
result and the parent stamps built from it carry the group either
way.
- Let n_files be the number of stamp files, n_rows the total number of
rows, and n_cols the number of columns. Time is O(n_rows * n_cols).
Every file is read before the frames are concatenated, so peak memory
is about twice the combined data, O(n_rows * n_cols).
Examples
--------
>>> import tempfile
>>> tmp = tempfile.TemporaryDirectory()
>>> paths = []
>>> for seg_id in (1, 2):
... path = Path(tmp.name) / f'w1.j0{seg_id}-bug-stamps.csv'
... pd.DataFrame({'segment': [f'w1.j0{seg_id}'], 'whole': ['w1'],
... 'segment_id': [seg_id], 'n_frames': [10]}
... ).to_csv(path, index=False)
... paths.append((str(path),))
>>> children_stamps(paths, 'segment')
segment whole segment_id n_frames group
0 w1.j01 w1 1 10 bug
1 w1.j02 w1 2 10 bug
>>> tmp.cleanup()
The stamps have no ``group`` column, so it comes from the file names
and, with no ``space`` column either, goes last.
"""
invalid_keyword(child_lineage, ["segment", "whole"])
# Load and concatenate all stamps
space_stamps = pd.concat(
[pd.read_csv(stamp[0]) for stamp in stamps], ignore_index=True
)
# The research code's stamps hold the group in the file name only.
if "group" not in space_stamps.columns:
group = _group_from_stamp_names(stamps)
columns = list(space_stamps.columns)
position = (columns.index("space") + 1 if "space" in columns
else len(columns))
space_stamps.insert(position, "group", group)
# Drop specific columns for 'whole' lineage
cols_to_drop = ["segment", "segment_id"]
if child_lineage == "whole":
dropped = [col for col in cols_to_drop if col in space_stamps.columns]
if dropped:
space_stamps = space_stamps.drop(columns=dropped)
warnings.warn(
"'segment' and 'segment_id' columns are dropped when"
" individual 'whole' stamps combined to create a single"
" dataframe of 'whole' stamps by 'children_stamps'.",
UserWarning,
)
else:
logger.debug("No %s columns to drop from the whole stamps.",
cols_to_drop)
# Save the resulting DataFrame to a CSV if specified
if save_to is not None:
space_name = str(space_stamps.loc[0, "space"])
group = str(space_stamps.loc[0, "group"])
filename = f"{space_name}-{group}-stamps-{child_lineage}.csv"
space_stamps.to_csv(os.path.join(save_to, filename), index=False)
return space_stamps
[docs]
def parents_stamps(
stamps: pd.DataFrame,
child_lineage: PrimitiveLineageT,
properties: dict[str, Callable[..., Any]] | None = None,
save_to: Pathish | None = None,
) -> pd.DataFrame:
"""
Aggregate the child stamps of a *space* into parent stamps.
The aggregation functions are given by `properties`; with `save_to`, the
result is also saved as a CSV file.
Parameters
----------
stamps : pd.DataFrame
DataFrame containing child stamp data, the lineage of whose rows is
`child_lineage`. It must have a 'whole' and a 'segment_id' column for
'segment' stamps, and an 'ensemble_long' and an 'ensemble_id' column
for 'whole' stamps, and an 'n_frames' column for both.
child_lineage : {'segment', 'whole'}
Lineage of the child stamps, the rows of `stamps`; the parents are
one level up, 'whole' or 'ensemble_long'.
properties : dict of str -> Callable, optional
Dictionary specifying aggregation functions for specific columns,
replacing the default 'last'. It cannot name the parent column
('whole' or 'ensemble_long'), the counted column ('segment_id' or
'ensemble_id'), or 'n_frames', which the function aggregates itself.
save_to : Pathish, optional
Directory path where the parent DataFrame is saved, if specified,
as ``<space>-<group>-stamps-<level>.csv``, with ``<level>`` the
lineage of the parent rows, ``whole`` for 'segment' stamps and
``ensemble_long`` for 'whole' stamps, and with ``space`` and
``group`` read from the first row; a file of the same name is
overwritten.
Returns
-------
pd.DataFrame
Aggregated parent DataFrame, one row per parent:
- 'segment': one row per 'whole', with 'n_segments' (the number of
segments) and 'n_frames' (their sum).
- 'whole': one row per 'ensemble_long', with 'n_ensembles' (the
number of wholes) and 'n_frames' (the last whole's); 'segment',
'segment_id', and 'n_segments' are dropped if present.
Every other column takes its last value in each group unless
`properties` says otherwise; 'lineage_name' and the `child_lineage`
column are dropped.
Raises
------
ValueError
If `child_lineage` is not one of the accepted values; if a row has no
parent name (NaN in 'whole' for 'segment' stamps, in
'ensemble_long' for 'whole' stamps), which the group-by would drop;
if `properties` names a column the function aggregates itself; or
if `save_to` is given and the stamps lack a 'space' or a 'group'
column, which the file name needs.
Notes
-----
- It is assumed that all the stamp files have belong to the same *group* in
a given *space*.
- If `child_lineage` is 'segment', stamps correspond to individual
segments, each with a unique `segment_id`.
- If `child_lineage` is 'whole', stamps correspond to whole simulations,
each identified by an `ensemble_id`.
- The 'lineage_name' column and the `child_lineage` column are not
carried into the parent stamps. Either may be absent, as
'lineage_name' is from the stamps the probe phase writes; an absent
one is skipped with a DEBUG record.
- Let n_rows be the number of child stamps and n_cols the number of
columns. The group-by is hash-based, so expected time is
O(n_rows * n_cols); memory is one copy of the parent stamps.
Examples
--------
>>> stamps = pd.DataFrame({
... 'lineage_name': ['w1.j01', 'w1.j02', 'w2.j01'],
... 'segment': ['w1.j01', 'w1.j02', 'w2.j01'],
... 'whole': ['w1', 'w1', 'w2'],
... 'segment_id': [1, 2, 1],
... 'n_frames': [10, 20, 30],
... })
>>> parents_stamps(stamps, 'segment')
whole n_segments n_frames
0 w1 2 30
1 w2 1 30
"""
invalid_keyword(child_lineage, ["segment", "whole"])
# Base aggregation functions for columns
base_columns = list(stamps.columns)
for column in ("lineage_name", child_lineage):
if column in base_columns:
base_columns.remove(column)
else:
logger.debug("Column %r is not among the stamps columns.", column)
agg_funcs: dict[str, str | Callable[..., Any]] = {
col: "last" for col in base_columns
}
# Define grouping column and lineage-specific aggregations
if child_lineage == "whole":
parent_groupby = "ensemble_long"
own_aggs: dict[str, str] = {"ensemble_id": "count",
"n_frames": "last"}
else:
parent_groupby = "whole"
own_aggs = {"segment_id": "count", "n_frames": "sum"}
for column in (properties or {}):
if column == parent_groupby or column in own_aggs:
raise ValueError(
f"properties cannot set {column!r}, which parents_stamps "
"aggregates itself.")
missing = int(stamps[parent_groupby].isna().sum())
if missing:
rows = "row has" if missing == 1 else "rows have"
raise ValueError(
f"{missing} stamps {rows} no {parent_groupby!r}; the group-by "
"would drop them.")
if properties is not None:
agg_funcs.update(properties)
agg_funcs.update(own_aggs)
# Perform the groupby and aggregate
agg_funcs.pop(parent_groupby)
parent_stamps = stamps.groupby(parent_groupby).agg(agg_funcs).reset_index()
# Rename and drop columns based on lineage
if child_lineage == "whole":
parent_stamps = parent_stamps.rename(
columns={"ensemble_id": "n_ensembles"}
)
# If the 'whole' stamps are generated directly in the 'probe' phase,
# then 'segment' and 'segment_id' columns are "N/A" and are removed
# from the list of stamps columns that are added to the parents
# stamps.
# There is no need to have the 'n_segment' column in the parent
# stamps, so it is removed. The 'whole' stamps directly generated
# in the 'probe' phase do not have such a column, but those generated
# from 'segment' stamps have.
# Dropping redundant columns silently:
parent_stamps = parent_stamps.drop(
columns=["n_segments", "segment_id", "segment"],
errors="ignore",
)
else:
parent_stamps = parent_stamps.rename(
columns={"segment_id": "n_segments"}
)
if save_to is not None:
for column in ("space", "group"):
if column not in parent_stamps.columns:
raise ValueError(
f"Saving the parent stamps needs a {column!r} column "
"for the file name.")
space_name = str(parent_stamps.loc[0, "space"])
group = str(parent_stamps.loc[0, "group"])
level = parent_groupby
filename = f"{space_name}-{group}-stamps-{level}.csv"
parent_stamps.to_csv(os.path.join(save_to, filename), index=False)
return parent_stamps
[docs]
def space_tseries(
input_database: str,
prop: PropertyT,
parser: ParserClassT,
hierarchy: str,
physical_attrs: list[str],
group: GroupT,
*,
divisor: float = 0.025,
round_to: int = 3,
is_save: bool | None = False,
crowder_attr: str = "phi_bulk_c",
) -> pd.DataFrame:
"""
Stack the ensemble-averaged time series of one property across the
state points of a space into one DataFrame.
The physical attributes in `physical_attrs` are added as columns.
Parameters
----------
input_database : str
Path to the directory containing the ensemble-averaged time-series
files, named ``<ensemble_long>-<group>-<prop>-ensAvg.csv``, with or
without a trailing path separator.
prop : str
Name of the physical property of interest.
parser : ParserClassT
Parser class to infer file-specific information from filenames or
paths.
hierarchy : str
Pattern prefix for the time-series filenames (e.g., `"N*"`).
physical_attrs : list[str]
List of physical attributes to add as new columns in the output
DataFrame.
group : GroupT
Particle group type. Its trajectory's dump period, in time steps,
is the `parser` attribute named by ``dump_keyword``.
divisor : float, default 0.025
Rounding step for the crowder volume fraction named by
`crowder_attr`.
round_to : int, default 3
Number of decimal places for rounding the crowder volume fraction.
is_save : bool, default False
If True, saves the concatenated DataFrame to a CSV file named
``<space>-<group>-<prop>-space.csv`` in the ``analysis`` phase's
``space`` directory, which :func:`make_database` creates next to
`input_database`; a file of the same name is overwritten.
crowder_attr : str, default 'phi_bulk_c'
Name of the `parser` attribute that holds the bulk volume fraction
of crowders. Its value is added as a column of that name, whether or
not it is listed in `physical_attrs`, and rounded into the
``phi_bulk_c_round`` column.
Returns
-------
pd.DataFrame
Concatenated DataFrame of all time-series with added physical
attributes. Besides the columns of each file, it has ``t_index``
(the frame index times the dump period of the `group` trajectory),
``time`` (``t_index`` times the parser's ``dt``), one column per
entry in `physical_attrs`, a `crowder_attr` column, and
``phi_bulk_c_round``.
Examples
--------
One ensemble-averaged time series of two frames of the ``bug``
trajectory, dumped every 1000 steps (``bdump``) of 0.005 time units:
>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> database = Path(tmp.name) / 'analysis' / 'N20-bug-ensAvg'
>>> database.mkdir(parents=True)
>>> state = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> pd.DataFrame({'gyrTMon-mean': [1.0, 2.0]}).to_csv(
... database / f'{state}-bug-gyrTMon-ensAvg.csv', index=False)
>>> space = space_tseries(str(database) + '/', 'gyrTMon', ChainCyl,
... 'N*', ['nmon'], 'bug')
>>> space[['t_index', 'time', 'gyrTMon-mean', 'nmon']]
t_index time gyrTMon-mean nmon
0 0 0.0 1.0 20
1 1000 5.0 2.0 20
>>> tmp.cleanup()
Notes
-----
- Files are selected by globbing
``os.path.join(input_database, hierarchy + '-<prop>-ensAvg.csv')``
and sorted alphanumerically by their full names.
- Requires a parser class with methods to retrieve attribute information
for each file.
- Let n_files be the number of matched files, n_rows the total number
of rows across them, and n_cols the number of columns of the result.
Time is O(n_rows * n_cols + n_files log n_files). Every file is read
and kept before the frames are concatenated, so peak memory is about
twice the result, O(n_rows * n_cols).
"""
prop_ext = f"-{prop}-ensAvg.csv"
ens_avg_csvs = sort_filenames(
glob(os.path.join(input_database, hierarchy + prop_ext)), ["csv"]
)
prop_csvs = []
for ens_avg_csv in ens_avg_csvs:
ens_avg = pd.read_csv(ens_avg_csv[0], header=0)
prop_info = parser(
ens_avg_csv[0], "ensemble_long", group, ispath=True
)
ens_avg = ens_avg.reset_index()
ens_avg = ens_avg.rename(columns={"index": "t_index"})
# Scale frame indices by the dump period of the group's trajectory
ens_avg["t_index"] *= getattr(prop_info, prop_info.dump_keyword)
ens_avg["time"] = ens_avg["t_index"] * getattr(prop_info, "dt")
# Add physical attributes
for attr_name in physical_attrs:
ens_avg[attr_name] = getattr(prop_info, attr_name)
# Add and round the bulk volume fraction of crowders
ens_avg[crowder_attr] = getattr(prop_info, crowder_attr)
ens_avg["phi_bulk_c_round"] = ens_avg[crowder_attr].apply(
lambda x: round_to_multiple(x, divisor, round_to)
)
prop_csvs.append(ens_avg)
# Concatenate all time-series DataFrames
prop_db = pd.concat(prop_csvs, axis=0)
prop_db = prop_db.reset_index(drop=True)
# Optionally save to file
if is_save:
save_to_space = make_database(
input_database, "analysis", stage="space", group=group
)
space = save_to_space.split("/")[-2].split("-")[0]
filepath = (
save_to_space + f"{create_fullname(space, group, prop)}-space.csv"
)
prop_db.to_csv(filepath, index=False)
return prop_db
[docs]
def space_hists(
input_database: str,
prop: PropertyT,
parser: ParserClassT,
hierarchy: str,
physical_attrs: list[str],
group: GroupT,
bin_center: np.ndarray | None = None,
normalize: bool | None = False,
divisor: float = 0.025,
round_to: int = 3,
is_save: bool | None = False,
crowder_attr: str = "phi_bulk_c",
) -> pd.DataFrame:
"""
Stack the ensemble-averaged histograms of one property across the state
points of a space into one DataFrame.
The histograms are normalized if `normalize` asks for it.
Parameters
----------
input_database : str
Path to the directory containing the ensemble-averaged histogram
files, named ``<ensemble_long>-<group>-<prop>-ensAvg.csv``, with or
without a trailing path separator.
prop : :py:obj:`~.PropertyT`
Name of the physical property of interest.
parser : ParserClassT
Parser class to infer file-specific information from filenames or
paths.
hierarchy : str
Pattern prefix for the histogram filenames (e.g., `"N*"`).
physical_attrs : list[str]
List of physical attributes to add as new columns in the output
DataFrame.
group : GroupT
Particle group type.
bin_center : np.ndarray, optional
Array of bin centers. If not provided, must be present in the
DataFrames.
normalize : bool, default False
If True, normalizes the histogram data.
divisor : float, default 0.025
Rounding step for the crowder volume fraction named by
`crowder_attr`.
round_to : int, default 3
Number of decimal places for rounding the crowder volume fraction.
is_save : bool, default False
If True, saves the concatenated DataFrame to a CSV file named
``<space>-<group>-<prop>-space.csv`` in the ``analysis`` phase's
``space`` directory, which :func:`make_database` creates next to
`input_database`; a file of the same name is overwritten.
crowder_attr : str, default 'phi_bulk_c'
Name of the `parser` attribute that holds the bulk volume fraction
of crowders. Its value is added as a column of that name, whether or
not it is listed in `physical_attrs`, and rounded into the
``phi_bulk_c_round`` column.
Returns
-------
pd.DataFrame
Concatenated DataFrame of all histograms with added physical
attributes: one column per entry in `physical_attrs`, a
`crowder_attr` column, and ``phi_bulk_c_round``.
Warns
-----
UserWarning
If `normalize` is True and the ``<prop>-mean`` column of a file sums
to zero; its ``<prop>-norm`` column is then set to zero.
Examples
--------
One ensemble-averaged histogram of three bins, normalized to sum to one:
>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> database = Path(tmp.name) / 'analysis' / 'N20-bug-ensAvg'
>>> database.mkdir(parents=True)
>>> state = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> pd.DataFrame({'rhoMon-mean': [1.0, 2.0, 1.0]}).to_csv(
... database / f'{state}-bug-rhoMon-ensAvg.csv', index=False)
>>> space = space_hists(str(database) + '/', 'rhoMon', ChainCyl, 'N*',
... ['nmon'], 'bug',
... bin_center=np.array([0.5, 1.5, 2.5]),
... normalize=True)
>>> space[['bin_center', 'rhoMon-mean', 'rhoMon-norm', 'nmon']]
bin_center rhoMon-mean rhoMon-norm nmon
0 0.5 1.0 0.25 20
1 1.5 2.0 0.50 20
2 2.5 1.0 0.25 20
>>> tmp.cleanup()
Notes
-----
- Files are selected by globbing
``os.path.join(input_database, hierarchy + '-<prop>-ensAvg.csv')``
and sorted alphanumerically by their full names.
- If `normalize` is True, histogram values will be scaled to sum to 1.
- Let n_files be the number of matched files, n_rows the total number
of rows across them, and n_cols the number of columns of the result.
Time is O(n_rows * n_cols + n_files log n_files). Every file is read
and kept before the frames are concatenated, so peak memory is about
twice the result, O(n_rows * n_cols).
- The `bin_center` should be provided if it is not available in the input
data. When all the ensemble-averaged DataFrames have the same bin
center, like 'clustersHistTFoci' or 'bondsHistTFoci' properties, the
`bin_center` may not be provided.
"""
prop_ext = f"-{prop}-ensAvg.csv"
ens_avg_csvs_ungrouped = glob(
os.path.join(input_database, hierarchy + prop_ext))
ens_avg_csvs = sort_filenames(ens_avg_csvs_ungrouped, formats=["csv"])
prop_csvs = []
for ens_avg_csv in ens_avg_csvs:
ens_avg = pd.read_csv(ens_avg_csv[0], header=0)
prop_info = parser(
ens_avg_csv[0], "ensemble_long", group, ispath=True
)
# Handle bin_center if provided
if bin_center is not None:
ens_avg["bin_center"] = bin_center.tolist()
ens_avg["bin_center-norm"] = (
ens_avg["bin_center"] / ens_avg["bin_center"].max()
)
# Normalize data if specified
if normalize:
normalizer = ens_avg[prop + "-mean"].sum()
if normalizer != 0:
ens_avg[prop + "-norm"] = ens_avg[prop + "-mean"] / normalizer
else:
warnings.warn(
"All values are zero; normalized values set to zero.",
UserWarning,
)
ens_avg[prop + "-norm"] = 0
# Add physical attributes
for attr_name in physical_attrs:
ens_avg[attr_name] = getattr(prop_info, attr_name)
# Add and round the bulk volume fraction of crowders
ens_avg[crowder_attr] = getattr(prop_info, crowder_attr)
ens_avg["phi_bulk_c_round"] = ens_avg[crowder_attr].apply(
lambda x: round_to_multiple(x, divisor, round_to)
)
prop_csvs.append(ens_avg)
# Concatenate all histograms
prop_db = pd.concat(prop_csvs, axis=0)
prop_db = prop_db.reset_index(drop=True)
# Optionally save to file
if is_save:
save_to_space = make_database(
input_database, "analysis", stage="space", group=group
)
space = save_to_space.split("/")[-2].split("-")[0]
filepath = (
save_to_space + f"{create_fullname(space, group, prop)}-space.csv"
)
prop_db.to_csv(filepath, index=False)
return prop_db
BULK_NAME_MAP: Mapping[str, str] = MappingProxyType({
"phi_m_bulk": "phi_bulk_m",
"rho_m_bulk": "rho_bulk_m",
"phi_c_bulk": "phi_bulk_c",
"rho_c_bulk": "rho_bulk_c",
"phi_c_bulk_one": "phi_bulk_c_one",
"phi_c_bulk_two": "phi_bulk_c_two",
"rho_c_bulk_one": "rho_bulk_c_one",
"rho_c_bulk_two": "rho_bulk_c_two",
"phi_hns_bulk": "phi_bulk_hns",
"rho_hns_bulk": "rho_bulk_hns",
"phi_c_bulk_round": "phi_bulk_c_round",
})
"""
Each bulk-quantity name of the research code mapped to its PolyPhys name.
The research code named a bulk quantity ``<quantity>_<species>_bulk``, such
as ``phi_c_bulk``; PolyPhys names it ``<quantity>_bulk_<species>``, such as
``phi_bulk_c``. The research code wrote every parser attribute as a column
of its ``*-stamps.csv`` files, and those columns carry into its
ensemble-average and ``space`` tables, so a table it wrote carries the old
names. The map is read-only; :func:`rename_bulk_columns` applies it.
.. versionadded:: 0.7.0
"""
[docs]
def rename_bulk_columns(table: pd.DataFrame) -> pd.DataFrame:
"""
Return a copy of `table` whose bulk-quantity columns carry the PolyPhys
names.
Each column named by a key of :data:`BULK_NAME_MAP` is renamed to that
key's value; every other column, every value, and the column order are
unchanged. A table that already carries the PolyPhys names comes back
equal to `table`.
Parameters
----------
table : pd.DataFrame
A table read from a stamp, ensemble-average, or ``space`` file that
the research code wrote.
Returns
-------
pd.DataFrame
A new table; `table` is not modified.
Notes
-----
Runs in :math:`O(n_\\text{col})` time for the renaming and
:math:`O(n_\\text{row} n_\\text{col})` space for the copy.
Examples
--------
A row of a research-code stamp file:
>>> import pandas as pd
>>> old = pd.DataFrame({'nmon': [20], 'phi_c_bulk': [0.1],
... 'rho_c_bulk': [0.191], 'n_frames': [100]})
>>> new = rename_bulk_columns(old)
>>> list(new.columns)
['nmon', 'phi_bulk_c', 'rho_bulk_c', 'n_frames']
>>> list(old.columns)
['nmon', 'phi_c_bulk', 'rho_c_bulk', 'n_frames']
.. versionadded:: 0.7.0
"""
return table.rename(columns=dict(BULK_NAME_MAP))