polyphys.manage.organizer#

polyphys.manage.organizer#

The polyphys.manage.organizer module provides tools for organizing and aggregating simulation data based on filename patterns, file structures, and processing stages. It facilitates handling segmented data, grouping experiments into ensembles, and consolidating ensembles into spaces.

The module is tailored for use with LAMMPS and similar simulation tools, where it automates data consolidation across hierarchical levels. It is particularly useful for organizing data generated in high-throughput simulations, enabling efficient post-processing, ensemble averaging, and preparation of data for statistical and comparative analysis.

Terminology#

The following terms are used throughout the module to structure and organize simulation output data:

  • Experiment: A complete simulation run, typically using tools like LAMMPS, representing a specific set of conditions.

  • Lineage: The level of an artifact in the hierarchy segment → whole → ensemble_long → ensemble → space, the values of LineageT; each level aggregates the one before it:

    • segment: a chunk of one simulation run.

    • whole: one run, its segments joined.

    • ensemble_long: one state point, every input parameter in the name.

    • ensemble: one state point, its replicates (wholes with different random seeds) collapsed into a short name.

    • space: the parameter sweep, the state points of one study.

  • Attributes: Macroscopic characteristics such as box size or polymer topology that remain constant within a space.

  • Properties: Physical features measured during the simulation, like density or radius of gyration, with optional associated measures (e.g., autocorrelation).

  • Species: The particle or molecular entity type, such as ‘Mon’ for monomers, each with specific behaviors or attributes as needed.

  • Phases: Directories organized by processing phase: simsAll and simsCont (simulation inputs), logs, trjs (trajectories), probes (probing data), analysis (post-processing), viz (visualization-ready data), galaxy, and allInOne; the values of PhaseT.

  • Stages: Directories inside a phase, each holding the artifacts of one lineage level or one step of the reduction: segment, wholeSim, ens, ensAvg, space, and galaxy, the values of StageT. The organizer writes them in the analysis phase.

The words segment and space are both lineage levels and stages, and galaxy is both a phase and a stage; which one a word is follows from where it appears: a lineage argument, a phase directory, or the last field of a stage directory’s name. galaxy is not a lineage level.

Naming Conventions#

The Lineage, phases, and stages page is the reference; in short:

  • A stage directory is <root>/<phase>/<prefix>-<group>-<stage>, as make_database() builds it.

  • A file is <lineage name>-<group>-<property>, as create_fullname() builds it, with -ensAvg or -space added where two stages hold files of one level (ens and ensAvg) or the level is space, and then the extension. A stamps table is <space>-<group>-stamps-<level>.csv, with <level> the lineage of its rows.

  • Fields are joined by -. A lineage name never contains -, and a group cannot; a property may, to join a measure (gyrTMon-acf), so it is everything between the group and the stage suffix or the extension.

Dependencies#

  • numpy: For numerical operations on arrays, such as summing or concatenating segments.

  • pandas: For handling tabular data and DataFrames used in data aggregation.

Contents#

Naming and persistence: create_fullname(), save_artifact(), and make_database().

Building wholes from their parts: whole_from_segments() and whole_from_file().

Grouping wholes into ensembles: ens_from_bin_edge(), ens_from_vec(), ens_from_mat_t(), ens_from_df(), and ensemble().

Averaging over an ensemble: ens_avg_from_bin_edge(), ens_avg_from_ndarray(), ens_avg_from_df(), and ensemble_avg().

Space-level collection and bookkeeping: children_stamps(), parents_stamps(), space_tseries(), and space_hists().

Reading tables the research code wrote: BULK_NAME_MAP and rename_bulk_columns().

The density sum rule on the ensAvg profiles these reducers build is in polyphys.analyze.sumrule.

Usage ===== The organizer module streamlines organization and aggregation of simulation data. Users can assemble segmented data, organize data into ensembles, and generate ensemble-averaged data at various aggregation stages. Its multi-level organization supports comparative analysis across varied conditions within spaces and galaxies.

Examples

Two replicates (ens1 and ens2) of one state point, each probed in two segments (j01 and j02), reduce to one ensemble-averaged time series. Segment s of replicate e holds two frames of the value 10 * e + s:

>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> root = Path(tmp.name)
>>> state = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> paths = []
>>> for e in (1, 2):
...     for s in (1, 2):
...         path = root / f'{state}ens{e}.j0{s}.bug.gyrTMon.npy'
...         np.save(path, np.full(2, 10.0 * e + s))
...         paths.append(path)

Join each replicate’s segments into a whole, collect the wholes of the state point into an ensemble, and average over the replicates:

>>> wholes = whole_from_segments('gyrTMon', paths, ChainCyl, 'bug',
...                              'tseries')
>>> wholes[state + 'ens1']
array([11., 11., 12., 12.])
>>> ensembles = ensemble('gyrTMon', wholes, ChainCyl, 'bug', 'vector')
>>> averages = ensemble_avg('gyrTMon', ensembles, 'bug', 'dataframe')
>>> averages[state]
   gyrTMon-mean  gyrTMon-var  gyrTMon-sem
0          16.0         50.0          5.0
1          16.0         50.0          5.0
2          17.0         50.0          5.0
3          17.0         50.0          5.0
>>> tmp.cleanup()

Notes

  • Filenames must follow consistent patterns for the organizer module to accurately parse and organize data.

  • This module supports only cylindrical, slit, or cubic geometries.

  • Functions in this module rely on consistent naming conventions for groups, properties, species, and stages.

Module Attributes

BULK_NAME_MAP

Each bulk-quantity name of the research code mapped to its PolyPhys name.

Functions

children_stamps(stamps, child_lineage[, save_to])

Combine the stamp files of the artifacts of one lineage, child_lineage, into one DataFrame for a space.

create_fullname(name, group, prop)

Join a base name, a particle group, and a property into an artifact's full name.

ens_avg_from_bin_edge(ens_data)

Return the bin edges of an 'ensemble' for its ensemble-averaged artifact.

ens_avg_from_df(ens_prop, ens_data, exclude)

Compute the mean, variance, and standard error of the mean (SEM) across the whole columns of an ensemble DataFrame.

ens_avg_from_ndarray(ens_prop, ens_data)

Compute the mean, variance, and standard error of the mean (SEM) across the wholes of an ensemble 3D array.

ens_from_bin_edge(ens_data)

Reduce the bin edges of the 'whole' datasets in an 'ensemble' to the single array they share.

ens_from_df(ens_data)

Average the DataFrames of the wholes of one ensemble into one DataFrame.

ens_from_mat_t(ens_data)

Stack the 2D arrays of the wholes of one ensemble into a 3D array.

ens_from_vec(ens_data)

Merge the 1D arrays of the wholes of one ensemble into a DataFrame.

ensemble(prop, wholes, parser, group, whole_type)

Merge the wholes of one property into one ensemble per state point.

ensemble_avg(prop, ensembles, group, ens_type)

Average the wholes of each ensemble of one property.

make_database(old_database, phase, stage, group)

Create, if needed, the stage directory <root>/<phase>/<prefix>-<group>-<stage>/ of the sweep that old_database belongs to.

parents_stamps(stamps, child_lineage[, ...])

Aggregate the child stamps of a space into parent stamps.

rename_bulk_columns(table)

Return a copy of table whose bulk-quantity columns carry the PolyPhys names.

save_artifact(filename, data, save_to)

Save data as <save_to>/<filename>, a .npy or .csv file by its type.

space_hists(input_database, prop, parser, ...)

Stack the ensemble-averaged histograms of one property across the state points of a space into one DataFrame.

space_tseries(input_database, prop, parser, ...)

Stack the ensemble-averaged time series of one property across the state points of a space into one DataFrame.

whole_from_file(whole_paths, parser, group)

Load the whole .npy files of one property of a particle group.

whole_from_segments(prop, segments, parser, ...)

Combine the segment files of one property into one array per whole.

polyphys.manage.organizer.BULK_NAME_MAP: Mapping[str, str] = mappingproxy({'phi_m_bulk': 'phi_bulk_m', 'rho_m_bulk': 'rho_bulk_m', 'phi_c_bulk': 'phi_bulk_c', 'rho_c_bulk': 'rho_bulk_c', 'phi_c_bulk_one': 'phi_bulk_c_one', 'phi_c_bulk_two': 'phi_bulk_c_two', 'rho_c_bulk_one': 'rho_bulk_c_one', 'rho_c_bulk_two': 'rho_bulk_c_two', 'phi_hns_bulk': 'phi_bulk_hns', 'rho_hns_bulk': 'rho_bulk_hns', 'phi_c_bulk_round': 'phi_bulk_c_round'})#

Each bulk-quantity name of the research code mapped to its PolyPhys name.

The research code named a bulk quantity <quantity>_<species>_bulk, such as phi_c_bulk; PolyPhys names it <quantity>_bulk_<species>, such as phi_bulk_c. The research code wrote every parser attribute as a column of its *-stamps.csv files, and those columns carry into its ensemble-average and space tables, so a table it wrote carries the old names. The map is read-only; rename_bulk_columns() applies it.

Added in version 0.7.0.

polyphys.manage.organizer.children_stamps(stamps, child_lineage, save_to=None)[source]#

Combine the stamp files of the artifacts of one lineage, child_lineage, into one DataFrame for a space.

With save_to, the DataFrame is also saved as a CSV file.

Parameters:
  • stamps (list[tuple[str]]) – List of 1-tuples, as sort_filenames() returns them, each holding the path to a ‘stamp’ CSV file. Rows are concatenated in this order.

  • child_lineage (typing.Literal['segment', 'whole']) – Lineage of the stamp files, and so of the rows of the result.

  • save_to (str | os.PathLike[str] | None) – Directory path where the combined DataFrame is saved, if specified, as <space>-<group>-stamps-<child_lineage>.csv, with space and group read from the first row; a file of the same name is overwritten.

Returns:

Combined DataFrame containing data from all the ‘stamp’ files, with a fresh 0..n-1 index, and a group column (see Notes).

Return type:

pandas.DataFrame

Raises:

ValueError – If child_lineage is not one of the accepted values; or if the stamps have no group column and a file name is not <name>-<group>-stamps.csv, or the names give more than one group.

Warns:

UserWarning – If child_lineage is ‘whole’ and the ‘segment’ and ‘segment_id’ columns are dropped.

Notes

  • It is assumed that all the stamp files have belong to the same group in a given space.

  • If child_lineage is ‘whole’, columns ‘segment’ and ‘segment_id’ are removed, as they are redundant for whole-lineage data, with a UserWarning. Either may be absent, as in stamps the probe phase writes for a whole; if neither is present, nothing is dropped and a DEBUG record is logged.

  • The stamps write_stamps() writes carry a group column; the research code’s stamps did not, and held the group in the file name, <lineage name>-<group>-stamps.csv. When the column is missing it is added from the file names, after space when that column is present and last otherwise, so the result and the parent stamps built from it carry the group either way.

  • Let n_files be the number of stamp files, n_rows the total number of rows, and n_cols the number of columns. Time is O(n_rows * n_cols). Every file is read before the frames are concatenated, so peak memory is about twice the combined data, O(n_rows * n_cols).

Examples

>>> import tempfile
>>> tmp = tempfile.TemporaryDirectory()
>>> paths = []
>>> for seg_id in (1, 2):
...     path = Path(tmp.name) / f'w1.j0{seg_id}-bug-stamps.csv'
...     pd.DataFrame({'segment': [f'w1.j0{seg_id}'], 'whole': ['w1'],
...                   'segment_id': [seg_id], 'n_frames': [10]}
...                  ).to_csv(path, index=False)
...     paths.append((str(path),))
>>> children_stamps(paths, 'segment')
  segment whole  segment_id  n_frames group
0  w1.j01    w1           1        10   bug
1  w1.j02    w1           2        10   bug
>>> tmp.cleanup()

The stamps have no group column, so it comes from the file names and, with no space column either, goes last.

polyphys.manage.organizer.create_fullname(name, group, prop)[source]#

Join a base name, a particle group, and a property into an artifact’s full name.

Parameters:
  • name (str) – The base name.

  • group (str) – Particle group (e.g., ‘bug’ or ‘nucleoid’).

  • prop (str) – Physical property name (e.g., ‘density’).

Returns:

A string that combines name, group, and prop in a hyphen-separated format.

Return type:

str

Examples

>>> create_fullname('sample', 'bug', 'density')
'sample-bug-density'
polyphys.manage.organizer.ens_avg_from_bin_edge(ens_data)[source]#

Return the bin edges of an ‘ensemble’ for its ensemble-averaged artifact.

Bin edges are not averaged: every ‘whole’ in the ensemble must share them. A single 1D array, the form ens_from_bin_edge() produces and ensemble_avg() passes here, is returned as a copy. A sequence of arrays, one per ‘whole’, is checked for equality with _assert_equal_bin_edges, and a copy of the shared array is returned.

Parameters:

ens_data (typing.Union[numpy.ndarray, typing.Sequence[numpy.ndarray]]) – Either the ensemble’s 1D array of bin edges, of shape (n_bins + 1,), or a non-empty sequence of such arrays, one per ‘whole’. A 2D array of shape (n_wholes, n_bins + 1) counts as a sequence of its rows.

Returns:

A copy of the shared 1D array of bin edges, of shape (n_bins + 1,).

Return type:

numpy.ndarray

Raises:

ValueError – If ens_data is a sequence and any array in it differs from the first one, in length or in value.

Notes

This function performs no averaging or statistical calculation. Time and space complexity are O(n_wholes * n_edges) and O(n_edges).

Examples

>>> ens_avg_from_bin_edge(np.array([0.0, 0.1, 0.2]))
array([0. , 0.1, 0.2])
>>> ens_avg_from_bin_edge(
...     [np.array([0.0, 0.1, 0.2]), np.array([0.0, 0.1, 0.2])]
... )
array([0. , 0.1, 0.2])
>>> ens_avg_from_bin_edge(
...     [np.array([0.0, 0.1, 0.2]), np.array([0.0, 0.1, 0.3])]
... )
Traceback (most recent call last):
    ...
ValueError: Bin edges are not identical across all arrays.
polyphys.manage.organizer.ens_avg_from_df(ens_prop, ens_data, exclude)[source]#

Compute the mean, variance, and standard error of the mean (SEM) across the whole columns of an ensemble DataFrame.

The result keeps the excluded columns, such as bin_center, and replaces the whole columns with the statistics.

Parameters:
  • ens_prop (str) – The name of the property to which this ‘ensemble’ data belongs, used as a prefix for the generated statistics columns.

  • ens_data (pandas.DataFrame) – A DataFrame containing ensemble data. Each ‘whole’ dataset is in its own column, and any columns specified in exclude remain unaffected.

  • exclude (list[str]) – A list of column names to exclude from averaging. For example, exclude might contain ‘bin_center’ for histogram-type ensembles.

Returns:

A new DataFrame with the excluded columns of ens_data, followed by the columns <ens_prop>-mean, <ens_prop>-var, and <ens_prop>-sem: the mean, variance, and SEM of each row across the ‘whole’ data columns. It has the same index as ens_data.

Return type:

pandas.DataFrame

Notes

  • ens_data is not modified; the statistics are written to a new DataFrame.

  • The function averages only the ‘whole’ columns, excluding any specified in exclude.

  • The standard error of the mean (SEM) is computed from the sample standard deviation with ddof=1, the wholes assumed independent. Bessel’s correction makes the variance unbiased; the standard deviation is still biased, though less (the numpy.std() documentation, Notes), and so is the SEM.

  • Let n_rows be the number of rows of ens_data and n_wholes the number of ‘whole’ columns. Time is O(n_rows * n_wholes). Each statistic selects the ‘whole’ columns as a new DataFrame, so peak extra memory is O(n_rows * n_wholes), not O(n_rows); the result takes O(n_rows).

Examples

>>> ens_data = pd.DataFrame({
...     'bin_center': [1, 2, 3],
...     'whole1': [4.0, 5.0, 6.0],
...     'whole2': [5.0, 6.0, 7.0],
... })
>>> ens_avg_from_df('density', ens_data, ['bin_center'])
   bin_center  density-mean  density-var  density-sem
0           1           4.5          0.5          0.5
1           2           5.5          0.5          0.5
2           3           6.5          0.5          0.5
>>> list(ens_data.columns)
['bin_center', 'whole1', 'whole2']
polyphys.manage.organizer.ens_avg_from_ndarray(ens_prop, ens_data)[source]#

Compute the mean, variance, and standard error of the mean (SEM) across the wholes of an ensemble 3D array.

Each slice along the first axis (axis=0) is one whole’s matrix, and the statistics are taken along that axis.

Parameters:
  • ens_prop (str) – The name of the property to which this ‘ensemble’ data belongs. Used as a prefix for the computed statistics.

  • ens_data (numpy.ndarray) – A 3D NumPy array where each matrix slice along axis=0 represents a ‘whole’ dataset.

Returns:

A dictionary where keys are statistical measures (mean, variance, SEM) with ens_prop as a prefix while values are NumPy arrays of the computed statistics across ‘whole’ matrices.

Return type:

dict[str, numpy.ndarray]

Notes

  • The SEM is the sample standard deviation with ddof=1 divided by the square root of the number of wholes, which are assumed independent. Bessel’s correction makes the variance unbiased; the standard deviation is still biased, though less (the numpy.std() documentation, Notes), and so is the SEM.

  • This function performs averaging along axis=0 for each element across the ‘whole’ data matrices.

  • Let n_wholes be the length of axis 0 and m the number of elements of one whole. Time is O(n_wholes * m). NumPy’s var and std build temporaries of the size of ens_data, so peak extra memory is O(n_wholes * m); the three results take O(m).

Examples

>>> ens_data = np.array([
...     [[1.0, 2.0], [3.0, 4.0]],
...     [[2.0, 3.0], [4.0, 5.0]],
... ])
>>> ens_avg = ens_avg_from_ndarray('gyrTMon', ens_data)
>>> list(ens_avg)
['gyrTMon-mean', 'gyrTMon-var', 'gyrTMon-sem']
>>> ens_avg['gyrTMon-mean']
array([[1.5, 2.5],
       [3.5, 4.5]])
>>> ens_avg['gyrTMon-sem']
array([[0.5, 0.5],
       [0.5, 0.5]])
polyphys.manage.organizer.ens_from_bin_edge(ens_data)[source]#

Reduce the bin edges of the ‘whole’ datasets in an ‘ensemble’ to the single array they share.

Every ‘whole’ in an ensemble must be binned identically, so that the histograms paired with these edges stay aligned bin by bin. The edges are checked for equality with _assert_equal_bin_edges, the check whole_from_segments() applies to segments, rather than merged.

Parameters:

ens_data (tuple[str, dict[str, numpy.ndarray]]) – A tuple where the first element is the ensemble name (str) and the second element is a non-empty dictionary of bin edge arrays from different ‘whole’ datasets within that ensemble. Within the dictionary, keys are whole names while values are 1D arrays of whole bin edges, each of shape (n_bins + 1,).

Returns:

A tuple where the first element is the ensemble name, and the second is a copy of the shared 1D array of bin edges, of shape (n_bins + 1,).

Return type:

tuple[str, numpy.ndarray]

Raises:

ValueError – If the bin edges of any ‘whole’ differ from those of the first one, in length or in value.

Notes

Time and space complexity are O(n_wholes * n_edges) and O(n_edges).

Examples

>>> ens_from_bin_edge(
...     ('ens1', {'w1': np.array([0., 1., 2.]),
...               'w2': np.array([0., 1., 2.])})
... )
('ens1', array([0., 1., 2.]))

Differing edges raise instead of being merged:

>>> ens_from_bin_edge(
...     ('ens1', {'w1': np.array([0., 1., 2.]),
...               'w2': np.array([0., 1., 3.])})
... )
Traceback (most recent call last):
    ...
ValueError: Bin edges are not identical across all arrays.
polyphys.manage.organizer.ens_from_df(ens_data)[source]#

Average the DataFrames of the wholes of one ensemble into one DataFrame.

Each DataFrame is one whole of the ensemble.

In a ‘whole’ DataFrame, headers are ‘elements’ of a ‘matrix’ or 2D quantity and columns are the values of a given property.

In each ‘whole’ DataFrame, headers represent ‘elements’ of a 2D quantity (matrix-like structure), and columns contain the values for a specified property. Each ‘whole’ DataFrame may be either a ‘histogram’ or a ‘time-series’:

  • In a ‘histogram’ DataFrame, an additional column, bin_center, holds bin center values that are invariant under averaging and are retained in the output DataFrame. The length of histogram DataFrame is equal to the number of bins.

  • In a ‘time-series’ DataFrame, headers (columns) are elements of a matrix, and rows represent the values of each element over time. The length of timeseries DataFrame is equal to the number of time frames.

Parameters:

ens_data (tuple[str, dict[str, pandas.DataFrame]]) – A tuple where the first element is the ensemble name and the second is a dictionary with ‘whole’ names as keys and DataFrames, sharing their columns and index labels, as values.

Returns:

A tuple with:

  • The ensemble name (str).

  • A DataFrame with the averaged values across all ‘whole’ DataFrames.

Return type:

tuple[str, pandas.DataFrame]

Raises:

ValueError – If the DataFrames have unequal numbers of rows; the message names the ensemble and each whole’s length. A shorter whole is not averaged over the rows it has.

Notes

A clear distinction exists between ensembles of ‘vector’ or ‘matrix’ types and ensembles of ‘dataframe’ type. In vector or matrix types, each column in the resulting ensemble DataFrame is named after a ‘whole’ and represents that whole’s values for a specific property. In contrast, for DataFrame type ensembles, the columns are elements of a 2D quantity (e.g., matrix or time-series values) and hold averaged values across all whole DataFrames.

This function assumes that averaging non-element headers, such as ‘bin_center,’ does not change their values, and these are carried forward without modification.

Let n_wholes be the number of wholes, n_rows the rows and n_cols the columns of one DataFrame. Time and extra memory are O(n_wholes * n_rows * n_cols): the DataFrames are concatenated into one before they are grouped by index label and averaged.

Examples

>>> df1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
>>> df2 = pd.DataFrame({'A': [2, 3, 4], 'B': [5, 6, 7]})
>>> ens_name, averaged_df = ens_from_df(
...     ('ensemble1', {'whole1': df1, 'whole2': df2})
... )
>>> averaged_df
     A    B
0  1.5  4.5
1  2.5  5.5
2  3.5  6.5
polyphys.manage.organizer.ens_from_mat_t(ens_data)[source]#

Stack the 2D arrays of the wholes of one ensemble into a 3D array.

Each whole becomes one slice along the first axis.

Parameters:

ens_data (tuple[str, dict[str, numpy.ndarray]]) – A tuple where the first element is the ensemble name (str) and the second element is a dictionary with ‘whole’ names as keys and 2D NumPy arrays as values.

Returns:

A tuple where the first element is the ensemble name and the second is a 3D NumPy array of shape (n_wholes, n_rows, n_cols), formed by stacking the 2D arrays along the first axis in the order of the dictionary.

Return type:

tuple[str, numpy.ndarray]

Notes

Let n_wholes be the number of wholes and m the number of elements of one matrix. Time and extra memory are O(n_wholes * m): the matrices are copied into the stacked array.

Examples

>>> ens_name, combined = ens_from_mat_t(
...     ('ensemble1', {'whole1': np.array([[1, 2], [3, 4]]),
...                    'whole2': np.array([[5, 6], [7, 8]])})
... )
>>> combined.shape
(2, 2, 2)
>>> combined[1]
array([[5, 6],
       [7, 8]])
polyphys.manage.organizer.ens_from_vec(ens_data)[source]#

Merge the 1D arrays of the wholes of one ensemble into a DataFrame.

Each whole becomes one column, such as a time series or a histogram of the whole.

Parameters:

ens_data (tuple[str, dict[str, numpy.ndarray]]) – A tuple where the first element is the ensemble name, and the second is a dictionary mapping each ‘whole’ dataset name to a 1D NumPy array of measurements.

Returns:

A tuple where the first element is the ensemble name, and the second is a pandas DataFrame. Each column represents a ‘whole’ dataset, in the order of the dictionary, and rows contain corresponding vector values from each dataset.

Return type:

tuple[str, pandas.DataFrame]

Raises:

ValueError – If the vectors have unequal lengths; the message names the ensemble and each whole’s length.

Notes

Let n_wholes be the number of wholes and m the length of one vector. Time and extra memory are O(n_wholes * m): the vectors are copied into the DataFrame.

Examples

>>> ens_name, ensemble_df = ens_from_vec(
...     ('ensemble1', {'whole1': np.array([0.1, 0.2, 0.3]),
...                    'whole2': np.array([0.4, 0.5, 0.6])})
... )
>>> ens_name
'ensemble1'
>>> ensemble_df
   whole1  whole2
0     0.1     0.4
1     0.2     0.5
2     0.3     0.6
polyphys.manage.organizer.ensemble(prop, wholes, parser, group, whole_type, whole_edges=None, save_to=None)[source]#

Merge the wholes of one property into one ensemble per state point.

whole_type sets the structure of each whole and how the wholes of one ensemble merge.

Parameters:
  • prop (str) – The property as it appears in file names, such as 'rhoMon'.

  • wholes (dict[str, numpy.ndarray] | dict[str, pandas.DataFrame]) – Dictionary containing ‘whole’ names as keys and their corresponding data as values, represented as NumPy arrays or DataFrames.

  • parser (type[polyphys.manage.parser.ParserBase]) – Parser class used to interpret whole filenames, such as the ensemble name of each ‘whole’.

  • group (str) – Type of the particle group being processed.

  • whole_type (typing.Literal['vector', 'matrix', 'dataframe', 'bin_edge']) –

    Specifies the type of each ‘whole’ and defines the merging method:

    • ’vector’: A 1D NumPy array, such as a time series or histogram.

    • ’matrix’: A 2D NumPy array, such as a gyration tensor.

    • ’dataframe’: A DataFrame representing either a time series (with headers as matrix elements and rows as values over time) or a histogram (headers as elements, rows as counts or frequencies, and an additional ‘bin_center’ column for bin centers).

    • ’bin_edge’: A 1D array representing bin edges. Assumes all bin edges are identical and selects one to represent the ensemble.

  • whole_edges (dict[str, numpy.ndarray] | None) – Dictionary with ‘whole’ names as keys and 1D bin edge arrays, each of shape (n_bins + 1,), as values. Only valid when whole_type is ‘vector’. The wholes of an ensemble share their bin edges by design, so the edges of the first whole of each ensemble are used and the others are not compared. Each merged ensemble gets a ‘bin_center’ column of length n_bins holding the midpoints of those edges. Must contain every key of wholes.

  • save_to (str | os.PathLike[str] | None) – Directory path where output files are saved, if specified; see Notes.

Returns:

The ensembles, keyed by their ensemble_long names, as NumPy arrays or DataFrames.

Return type:

dict[str, numpy.ndarray | pandas.DataFrame]

Raises:

ValueError – If any parameter has an invalid value; if whole_edges is given and whole_type is not ‘vector’; or if the number of bins differs from the number of rows of the merged ensemble.

Notes

Different averaging methods apply based on whole_type. When whole_type is “DataFrame,” the resulting DataFrame averages the values of each matrix element across wholes, retaining headers that are mean-invariant (e.g., ‘bin_center’). For ‘vector’ and ‘matrix’ types, columns represent individual wholes in the ensemble DataFrame.

The bin center of bin i is the midpoint of its two edges, 0.5 * (edges[i] + edges[i + 1]). A center is invariant under averaging over wholes, which is why ensemble_avg() excludes ‘bin_center’ from its statistics by default.

Let n_wholes be the number of wholes, m the number of values in one whole, n_ensembles the number of ensembles, and n_edges the length of one bin-edge array. Grouping parses each whole name once and merging copies every whole once, so time and extra memory are O(n_wholes * m): the caller’s wholes and the merged ensembles are both held in memory. With whole_edges, computing the centers adds O(n_ensembles * n_edges) time and memory.

With save_to, each ensemble is saved by save_artifact() under <ensemble_long>-<group>-<prop>, as .csv for a DataFrame and .npy for an array, overwriting a file of the same name.

Examples

Two replicates of one state point, with shared edges, merge into one ensemble that carries the edge midpoints as ‘bin_center’:

>>> from polyphys.tests.synthetic import ChainCyl
>>> base = ('N2000r10.5lz300.0ac1.0nc117000dt0.005bdump5000'
...         'adump10000ens')
>>> wholes = {base + '1': np.array([1., 2., 3.]),
...           base + '2': np.array([3., 4., 5.])}
>>> edges = {name: np.array([0., 1., 2., 4.]) for name in wholes}
>>> ens = ensemble('rhoMon', wholes, ChainCyl, 'bug', 'vector',
...                whole_edges=edges)
>>> ens_df = next(iter(ens.values()))
>>> ens_df['bin_center'].tolist()
[0.5, 1.5, 3.0]

Without whole_edges, the ensemble has one column per whole only:

>>> ens = ensemble('rhoMon', wholes, ChainCyl, 'bug', 'vector')
>>> 'bin_center' in next(iter(ens.values())).columns
False
polyphys.manage.organizer.ensemble_avg(prop, ensembles, group, ens_type, exclude=None, save_to=None)[source]#

Average the wholes of each ensemble of one property.

Columns listed in exclude are left out of the averages.

Parameters:
  • prop (str) – The property as it appears in file names, such as 'rhoMon'.

  • ensembles (dict[str, numpy.ndarray | pandas.DataFrame]) – A dictionary where each key is an ensemble name and each value is either a DataFrame or ndarray representing ‘whole’ data.

  • group (str) – Type of the particle group.

  • ens_type (typing.Literal['dataframe', 'ndarray', 'bin_edge']) –

    The data format for each ensemble:

    • ’dataframe’: DataFrame format, with columns as ‘whole’ simulations or items in exclude.

    • ’ndarray’: 3D ndarray format, with each slice along axis=0 representing a ‘whole’ matrix.

    • ’bin_edge’: 1D ndarray format for bin edges, with bin edges assumed to be identical across wholes.

  • exclude (list[str] | None) – List of column names or items not included in ensemble averaging, such as ‘bin_center’ for histograms.

  • save_to (str | os.PathLike[str] | None) – Path to the directory where the averaged ensemble data is saved, if specified; see Notes.

Returns:

A dictionary where each key is an ensemble name, and each value is a DataFrame or ndarray of ensemble-averaged data.

Return type:

dict[str, numpy.ndarray] | dict[str, pandas.DataFrame]

Raises:

ValueError – If ens_type is not one of the accepted values.

Notes

With save_to, each result is saved by save_artifact() under <ensemble>-<group>-<prop>-ensAvg: a DataFrame as .csv, bin edges as .npy, and the dictionary of an ‘ndarray’ ensemble as one ...-ensAvg-<measure>.npy file per statistic, each overwriting a file of the same name.

Let n_ensembles be the number of ensembles. Ensembles are averaged one after another, with the time and peak extra memory of ens_avg_from_df(), ens_avg_from_ndarray(), or ens_avg_from_bin_edge() for the largest one; the results for all n_ensembles ensembles are held in memory together.

Examples

>>> ensembles = {
...     'ensemble1': pd.DataFrame({'whole1': [1.0, 2.0],
...                                'whole2': [3.0, 4.0]}),
...     'ensemble2': pd.DataFrame({'whole1': [5.0, 6.0],
...                                'whole2': [5.0, 8.0]}),
... }
>>> avg = ensemble_avg('rhoMon', ensembles, 'bug', 'dataframe')
>>> avg['ensemble2']
   rhoMon-mean  rhoMon-var  rhoMon-sem
0          5.0         0.0         0.0
1          7.0         2.0         1.0
polyphys.manage.organizer.make_database(old_database, phase, stage, group)[source]#

Create, if needed, the stage directory <root>/<phase>/<prefix>-<group>-<stage>/ of the sweep that old_database belongs to.

old_database is a directory of the same sweep that sits directly in a phase directory, <root>/<old_phase>/<old_dir>. The new directory keeps <root>, goes into phase, and is named <prefix>-<group>-<stage>, with <prefix> the part of <old_dir> before its first hyphen (or *) and <group> left out when group is None. The Lineage, phases, and stages page defines phases and stages.

Parameters:
  • old_database (str | os.PathLike[str]) – Path to a directory of the sweep, directly inside a phase directory; it need not exist.

  • phase (typing.Literal['simsAll', 'simsCont', 'logs', 'trjs', 'probes', 'analysis', 'viz', 'galaxy', 'allInOne']) – The new phase name for the directory, specifying its role in the workflow: one of 'simsAll', 'simsCont', 'logs', 'trjs', 'probes', 'analysis', 'viz', 'galaxy', or 'allInOne'.

  • stage (typing.Literal['segment', 'wholeSim', 'ens', 'ensAvg', 'space', 'galaxy']) – The stage of processing or data type represented in the new directory: one of 'segment', 'wholeSim', 'ens', 'ensAvg', 'space', or 'galaxy'.

  • group (str | None) – Particle group related to the data in the directory; left out of the name when None. It cannot contain -, the field separator of the name.

Returns:

The path to the new directory, ending with /. If the directory already exists, the same path is returned and an INFO record naming it is logged to the module’s logger.

Return type:

str

Raises:

ValueError – If phase is not a value of PhaseT, stage not a value of StageT, group contains -, or the parent of old_database is not named for a value of PhaseT; no directory is made.

Examples

The new directory sits next to the old phase directory:

  • Input: root/parent1/.../parentN/old_phase/old_dir

  • Result: root/parent1/.../parentN/phase/new_dir

>>> import tempfile
>>> tmp = tempfile.TemporaryDirectory()
>>> old = Path(tmp.name) / 'probes' / 'N20-bug-segment'
>>> new = make_database(old, 'analysis', 'ens', 'bug')
>>> Path(new).relative_to(tmp.name).as_posix()
'analysis/N20-bug-ens'
>>> Path(new).is_dir()
True
>>> tmp.cleanup()

A name outside the phase vocabulary is refused:

>>> make_database(old, 'simAll', 'ens', 'bug')
Traceback (most recent call last):
...
ValueError: 'simAll' is an invalid option. Please select one of: simsAll, simsCont, logs, trjs, probes, analysis, viz, galaxy, allInOne.
polyphys.manage.organizer.parents_stamps(stamps, child_lineage, properties=None, save_to=None)[source]#

Aggregate the child stamps of a space into parent stamps.

The aggregation functions are given by properties; with save_to, the result is also saved as a CSV file.

Parameters:
  • stamps (pandas.DataFrame) – DataFrame containing child stamp data, the lineage of whose rows is child_lineage. It must have a ‘whole’ and a ‘segment_id’ column for ‘segment’ stamps, and an ‘ensemble_long’ and an ‘ensemble_id’ column for ‘whole’ stamps, and an ‘n_frames’ column for both.

  • child_lineage (typing.Literal['segment', 'whole']) – Lineage of the child stamps, the rows of stamps; the parents are one level up, ‘whole’ or ‘ensemble_long’.

  • properties (dict[str, typing.Callable[..., typing.Any]] | None) – Dictionary specifying aggregation functions for specific columns, replacing the default ‘last’. It cannot name the parent column (‘whole’ or ‘ensemble_long’), the counted column (‘segment_id’ or ‘ensemble_id’), or ‘n_frames’, which the function aggregates itself.

  • save_to (str | os.PathLike[str] | None) – Directory path where the parent DataFrame is saved, if specified, as <space>-<group>-stamps-<level>.csv, with <level> the lineage of the parent rows, whole for ‘segment’ stamps and ensemble_long for ‘whole’ stamps, and with space and group read from the first row; a file of the same name is overwritten.

Returns:

Aggregated parent DataFrame, one row per parent:

  • ’segment’: one row per ‘whole’, with ‘n_segments’ (the number of segments) and ‘n_frames’ (their sum).

  • ’whole’: one row per ‘ensemble_long’, with ‘n_ensembles’ (the number of wholes) and ‘n_frames’ (the last whole’s); ‘segment’, ‘segment_id’, and ‘n_segments’ are dropped if present.

Every other column takes its last value in each group unless properties says otherwise; ‘lineage_name’ and the child_lineage column are dropped.

Return type:

pandas.DataFrame

Raises:

ValueError – If child_lineage is not one of the accepted values; if a row has no parent name (NaN in ‘whole’ for ‘segment’ stamps, in ‘ensemble_long’ for ‘whole’ stamps), which the group-by would drop; if properties names a column the function aggregates itself; or if save_to is given and the stamps lack a ‘space’ or a ‘group’ column, which the file name needs.

Notes

  • It is assumed that all the stamp files have belong to the same group in a given space.

  • If child_lineage is ‘segment’, stamps correspond to individual segments, each with a unique segment_id.

  • If child_lineage is ‘whole’, stamps correspond to whole simulations, each identified by an ensemble_id.

  • The ‘lineage_name’ column and the child_lineage column are not carried into the parent stamps. Either may be absent, as ‘lineage_name’ is from the stamps the probe phase writes; an absent one is skipped with a DEBUG record.

  • Let n_rows be the number of child stamps and n_cols the number of columns. The group-by is hash-based, so expected time is O(n_rows * n_cols); memory is one copy of the parent stamps.

Examples

>>> stamps = pd.DataFrame({
...     'lineage_name': ['w1.j01', 'w1.j02', 'w2.j01'],
...     'segment': ['w1.j01', 'w1.j02', 'w2.j01'],
...     'whole': ['w1', 'w1', 'w2'],
...     'segment_id': [1, 2, 1],
...     'n_frames': [10, 20, 30],
... })
>>> parents_stamps(stamps, 'segment')
  whole  n_segments  n_frames
0    w1           2        30
1    w2           1        30
polyphys.manage.organizer.rename_bulk_columns(table)[source]#

Return a copy of table whose bulk-quantity columns carry the PolyPhys names.

Each column named by a key of BULK_NAME_MAP is renamed to that key’s value; every other column, every value, and the column order are unchanged. A table that already carries the PolyPhys names comes back equal to table.

Parameters:

table (pandas.DataFrame) – A table read from a stamp, ensemble-average, or space file that the research code wrote.

Returns:

A new table; table is not modified.

Return type:

pandas.DataFrame

Notes

Runs in \(O(n_\text{col})\) time for the renaming and \(O(n_\text{row} n_\text{col})\) space for the copy.

Examples

A row of a research-code stamp file:

>>> import pandas as pd
>>> old = pd.DataFrame({'nmon': [20], 'phi_c_bulk': [0.1],
...                     'rho_c_bulk': [0.191], 'n_frames': [100]})
>>> new = rename_bulk_columns(old)
>>> list(new.columns)
['nmon', 'phi_bulk_c', 'rho_bulk_c', 'n_frames']
>>> list(old.columns)
['nmon', 'phi_c_bulk', 'rho_c_bulk', 'n_frames']

Added in version 0.7.0.

polyphys.manage.organizer.save_artifact(filename, data, save_to)[source]#

Save data as <save_to>/<filename>, a .npy or .csv file by its type.

An existing file of the same name is overwritten.

Parameters:
  • filename (str) – The output file name.

  • data (numpy.ndarray | pandas.DataFrame | dict[str, numpy.ndarray]) –

    Data to be saved. Accepted types:

    • np.ndarray: Saved as <filename>.npy.

    • pd.DataFrame: Saved as <filename>.csv, without the index.

    • dict of np.ndarray: Each key must have the form <prop>-<measure>, with exactly one hyphen, and each entry is saved as <filename>-<measure>.npy.

  • save_to (str | os.PathLike[str]) – Path to an existing directory where the file is saved.

Return type:

None

Notes

numpy.save() and pandas.DataFrame.to_csv() overwrite an existing file of the same name.

Raises:
  • TypeError – If data type does not match any of the supported formats.

  • ValueError – If data is a dict and one of its keys does not contain exactly one hyphen.

Parameters:
Return type:

None

Examples

>>> import tempfile
>>> tmp = tempfile.TemporaryDirectory()

Save a NumPy array to a .npy file:

>>> save_artifact('w-bug-rhoMon', np.array([1.0, 2.0]), tmp.name)
>>> np.load(Path(tmp.name) / 'w-bug-rhoMon.npy')
array([1., 2.])

Save a DataFrame to a .csv file:

>>> save_artifact('w-bug-rhoMon', pd.DataFrame({'A': [1, 2]}), tmp.name)
>>> pd.read_csv(Path(tmp.name) / 'w-bug-rhoMon.csv')
   A
0  1
1  2

Save each entry of a dict to its own file, named by its measure:

>>> save_artifact('w-bug-rhoMon', {'rhoMon-mean': np.array([1.5])},
...               tmp.name)
>>> sorted(path.name for path in Path(tmp.name).iterdir())
['w-bug-rhoMon-mean.npy', 'w-bug-rhoMon.csv', 'w-bug-rhoMon.npy']
>>> tmp.cleanup()
polyphys.manage.organizer.space_hists(input_database, prop, parser, hierarchy, physical_attrs, group, bin_center=None, normalize=False, divisor=0.025, round_to=3, is_save=False, crowder_attr='phi_bulk_c')[source]#

Stack the ensemble-averaged histograms of one property across the state points of a space into one DataFrame.

The histograms are normalized if normalize asks for it.

Parameters:
  • input_database (str) – Path to the directory containing the ensemble-averaged histogram files, named <ensemble_long>-<group>-<prop>-ensAvg.csv, with or without a trailing path separator.

  • prop (str) – Name of the physical property of interest.

  • parser (type[polyphys.manage.parser.ParserBase]) – Parser class to infer file-specific information from filenames or paths.

  • hierarchy (str) – Pattern prefix for the histogram filenames (e.g., “N*”).

  • physical_attrs (list[str]) – List of physical attributes to add as new columns in the output DataFrame.

  • group (str) – Particle group type.

  • bin_center (numpy.ndarray | None) – Array of bin centers. If not provided, must be present in the DataFrames.

  • normalize (bool | None) – If True, normalizes the histogram data.

  • divisor (float) – Rounding step for the crowder volume fraction named by crowder_attr.

  • round_to (int) – Number of decimal places for rounding the crowder volume fraction.

  • is_save (bool | None) – If True, saves the concatenated DataFrame to a CSV file named <space>-<group>-<prop>-space.csv in the analysis phase’s space directory, which make_database() creates next to input_database; a file of the same name is overwritten.

  • crowder_attr (str) – Name of the parser attribute that holds the bulk volume fraction of crowders. Its value is added as a column of that name, whether or not it is listed in physical_attrs, and rounded into the phi_bulk_c_round column.

Returns:

Concatenated DataFrame of all histograms with added physical attributes: one column per entry in physical_attrs, a crowder_attr column, and phi_bulk_c_round.

Return type:

pandas.DataFrame

Warns:

UserWarning – If normalize is True and the <prop>-mean column of a file sums to zero; its <prop>-norm column is then set to zero.

Examples

One ensemble-averaged histogram of three bins, normalized to sum to one:

>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> database = Path(tmp.name) / 'analysis' / 'N20-bug-ensAvg'
>>> database.mkdir(parents=True)
>>> state = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> pd.DataFrame({'rhoMon-mean': [1.0, 2.0, 1.0]}).to_csv(
...     database / f'{state}-bug-rhoMon-ensAvg.csv', index=False)
>>> space = space_hists(str(database) + '/', 'rhoMon', ChainCyl, 'N*',
...                     ['nmon'], 'bug',
...                     bin_center=np.array([0.5, 1.5, 2.5]),
...                     normalize=True)
>>> space[['bin_center', 'rhoMon-mean', 'rhoMon-norm', 'nmon']]
   bin_center  rhoMon-mean  rhoMon-norm  nmon
0         0.5          1.0         0.25    20
1         1.5          2.0         0.50    20
2         2.5          1.0         0.25    20
>>> tmp.cleanup()

Notes

  • Files are selected by globbing os.path.join(input_database, hierarchy + '-<prop>-ensAvg.csv') and sorted alphanumerically by their full names.

  • If normalize is True, histogram values will be scaled to sum to 1.

  • Let n_files be the number of matched files, n_rows the total number of rows across them, and n_cols the number of columns of the result. Time is O(n_rows * n_cols + n_files log n_files). Every file is read and kept before the frames are concatenated, so peak memory is about twice the result, O(n_rows * n_cols).

  • The bin_center should be provided if it is not available in the input data. When all the ensemble-averaged DataFrames have the same bin center, like ‘clustersHistTFoci’ or ‘bondsHistTFoci’ properties, the bin_center may not be provided.

polyphys.manage.organizer.space_tseries(input_database, prop, parser, hierarchy, physical_attrs, group, *, divisor=0.025, round_to=3, is_save=False, crowder_attr='phi_bulk_c')[source]#

Stack the ensemble-averaged time series of one property across the state points of a space into one DataFrame.

The physical attributes in physical_attrs are added as columns.

Parameters:
  • input_database (str) – Path to the directory containing the ensemble-averaged time-series files, named <ensemble_long>-<group>-<prop>-ensAvg.csv, with or without a trailing path separator.

  • prop (str) – Name of the physical property of interest.

  • parser (type[polyphys.manage.parser.ParserBase]) – Parser class to infer file-specific information from filenames or paths.

  • hierarchy (str) – Pattern prefix for the time-series filenames (e.g., “N*”).

  • physical_attrs (list[str]) – List of physical attributes to add as new columns in the output DataFrame.

  • group (str) – Particle group type. Its trajectory’s dump period, in time steps, is the parser attribute named by dump_keyword.

  • divisor (float) – Rounding step for the crowder volume fraction named by crowder_attr.

  • round_to (int) – Number of decimal places for rounding the crowder volume fraction.

  • is_save (bool | None) – If True, saves the concatenated DataFrame to a CSV file named <space>-<group>-<prop>-space.csv in the analysis phase’s space directory, which make_database() creates next to input_database; a file of the same name is overwritten.

  • crowder_attr (str) – Name of the parser attribute that holds the bulk volume fraction of crowders. Its value is added as a column of that name, whether or not it is listed in physical_attrs, and rounded into the phi_bulk_c_round column.

Returns:

Concatenated DataFrame of all time-series with added physical attributes. Besides the columns of each file, it has t_index (the frame index times the dump period of the group trajectory), time (t_index times the parser’s dt), one column per entry in physical_attrs, a crowder_attr column, and phi_bulk_c_round.

Return type:

pandas.DataFrame

Examples

One ensemble-averaged time series of two frames of the bug trajectory, dumped every 1000 steps (bdump) of 0.005 time units:

>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> database = Path(tmp.name) / 'analysis' / 'N20-bug-ensAvg'
>>> database.mkdir(parents=True)
>>> state = 'N20r5.5lz25.0ac1.0nc100dt0.005bdump1000adump5000'
>>> pd.DataFrame({'gyrTMon-mean': [1.0, 2.0]}).to_csv(
...     database / f'{state}-bug-gyrTMon-ensAvg.csv', index=False)
>>> space = space_tseries(str(database) + '/', 'gyrTMon', ChainCyl,
...                       'N*', ['nmon'], 'bug')
>>> space[['t_index', 'time', 'gyrTMon-mean', 'nmon']]
   t_index  time  gyrTMon-mean  nmon
0        0   0.0           1.0    20
1     1000   5.0           2.0    20
>>> tmp.cleanup()

Notes

  • Files are selected by globbing os.path.join(input_database, hierarchy + '-<prop>-ensAvg.csv') and sorted alphanumerically by their full names.

  • Requires a parser class with methods to retrieve attribute information for each file.

  • Let n_files be the number of matched files, n_rows the total number of rows across them, and n_cols the number of columns of the result. Time is O(n_rows * n_cols + n_files log n_files). Every file is read and kept before the frames are concatenated, so peak memory is about twice the result, O(n_rows * n_cols).

polyphys.manage.organizer.whole_from_file(whole_paths, parser, group)[source]#

Load the whole .npy files of one property of a particle group.

Each path in whole_paths points to one whole file.

Parameters:
Returns:

Dictionary where keys are the whole names (derived from each file path) and values are NumPy arrays containing the loaded data.

Return type:

dict[str, numpy.ndarray]

Raises:
  • TypeError – If an item of whole_paths is neither a path nor a sequence whose first element is a path.

  • ValueError – If two files parse to the same whole name; the message names both.

Notes

This function assumes that each file in whole_paths contains data related to a complete simulation, referred to as a “whole.” For further information on terminology, see the organizer module documentation.

Let n_wholes be the number of files and m the number of values in one file. Time and memory are O(n_wholes * m): every whole is loaded and held in memory at once.

Examples

>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> whole = ('N20r5.5lz25.0ac1.0nc100dt0.005bdump1000'
...          'adump5000ens1')
>>> path = Path(tmp.name) / f'{whole}-bug-gyrTMon.npy'
>>> np.save(path, np.array([1.0, 2.0, 3.0]))
>>> whole_from_file([path], ChainCyl, 'bug')[whole]
array([1., 2., 3.])
>>> tmp.cleanup()
polyphys.manage.organizer.whole_from_segments(prop, segments, parser, group, relation, save_to=None)[source]#

Combine the segment files of one property into one array per whole.

How the segments of a whole combine is set by relation.

Parameters:
  • prop (str) – The property as it appears in file names, such as 'gyrTMon'.

  • segments (typing.Union[typing.Iterable[str | os.PathLike[str]], typing.Iterable[typing.Sequence[str | os.PathLike[str]]]]) – An iterable of paths to segment .npy files for prop, either bare paths or 1-tuples of a path, as sort_filenames() returns them. Each path points to a single segment file. Segments of different wholes may be interleaved, but the segments of each whole must come in consecutive segment_id order (the j<NN> keyword of their names), with no id skipped.

  • parser (type[polyphys.manage.parser.ParserBase]) – A parser class from polyphys.manage.parser that interprets file paths and extracts attributes such as the whole name and the segment_id.

  • group (str) – Type of the particle group.

  • relation (typing.Literal['histogram', 'tseries', 'bin_edge']) –

    Specifies how to combine segments into a whole. Accepted values:

    • ’histogram’

      ’segment’ is an N-dimensional histogram-like file, so the segments of a ‘whole’ are summed element-wise. In the two-dimensional space, a histogram has a (x_nbins, y_nbins) shape.

    • ’tseries’

      ’segment’ is a time-series-like file, so the segments of a ‘whole’ are concatenated along axis 0, in segment_id order.

    • ’bin_edge’

      ’segment’ is a bin-edge file. All the segments of a whole must hold identical edges, and one copy of them is the whole’s bin-edge array.

  • save_to (str | os.PathLike[str] | None) – Directory path where the output files are saved, if specified. Each whole is saved as <whole>-<group>-<prop>.npy, overwriting a file of the same name.

Returns:

Dictionary with whole names as keys and the combined whole data as NumPy arrays, in the order in which the wholes first appear in segments.

Return type:

dict[str, numpy.ndarray]

Raises:
  • ValueError – If relation is not one of the accepted values; if the segments of a whole do not arrive in strictly increasing segment_id order, which includes a segment passed twice; if they skip a segment_id, such as j01 then j03, which would shift the time of every later frame or leave a part out of a sum; or, for ‘bin_edge’, if the segments of a whole hold different edges.

  • TypeError – If an item of segments is neither a path nor a sequence whose first element is a path.

Notes

The order of a whole’s segments is part of the data’s meaning for ‘tseries’, and a duplicated segment would be counted twice by ‘histogram’. This function checks the order and that no id is skipped; it does not require the first id to be 1. Sort the paths with sort_filenames() first, which orders the j<NN> keyword naturally.

Let n_segments be the number of segment files and m the number of values in one segment. Time is O(n_segments * m). Every segment of every whole is loaded and held in memory before any whole is reduced, so peak memory is O(n_segments * m), not the size of one whole; the returned wholes add O(n_wholes * m) for ‘histogram’ and ‘bin_edge’ and O(n_segments * m) for ‘tseries’, where n_wholes is the number of wholes.

Examples

Two segments of one whole, written as the probe phase names them, join into one time series in segment_id order:

>>> import tempfile
>>> from polyphys.tests.synthetic import ChainCyl
>>> tmp = tempfile.TemporaryDirectory()
>>> whole = ('N20r5.5lz25.0ac1.0nc100dt0.005bdump1000'
...          'adump5000ens1')
>>> paths = []
>>> for seg_id in (1, 2):
...     path = Path(tmp.name) / f'{whole}.j0{seg_id}.bug.gyrTMon.npy'
...     np.save(path, np.array([10.0, 11.0]) * seg_id)
...     paths.append(path)
>>> wholes = whole_from_segments('gyrTMon', paths, ChainCyl, 'bug',
...                              'tseries')
>>> wholes[whole]
array([10., 11., 20., 22.])

‘histogram’ sums the same segments element-wise:

>>> whole_from_segments('gyrTMon', paths, ChainCyl, 'bug',
...                     'histogram')[whole]
array([30., 33.])

Segments out of order raise rather than giving a reversed series:

>>> whole_from_segments('gyrTMon', paths[::-1], ChainCyl, 'bug',
...                     'tseries')
Traceback (most recent call last):
    ...
ValueError: The segments of whole '...ens1' are not in ... [2, 1]...
>>> tmp.cleanup()