# Studies: adding a study This page shows how to add a study to PolyPhys: the parser class that reads its file names, the specs that describe its probe and its analysis, the folder that holds them, and the checks that run on them. It also shows how to read the tables that the research code before PolyPhys wrote. ## Notation *study* : One research project: one simulation model, swept over its parameters, such as `HnsCub`, a ring polymer with H-NS proteins among crowders in a periodic cube. *keyword* : A short label in a file name followed by its value, such as `N200` for a chain of 200 monomers or `ens1` for the first replicate. *parser class* : A subclass of {class}`~polyphys.manage.parser.ParserBase` that declares a study's keywords and reads a name into typed attributes and lineage names. *spec* : A record of a study's choices: the {class}`~polyphys.spec.ProjectSpec` of its analysis, the {class}`~polyphys.spec.ProbeSpec` of its probe, and one {class}`~polyphys.spec.AnalysisTable` per particle group. *species* : A kind of particle, labeled as in the property names: `Mon` for the monomers, `Crd` for the crowders, `Hns` for the H-NS cores. $a_i$ : The diameter of the particles of species $i$, in the length unit of the simulation. The [Lineage](lineage.md) page defines the lineage levels and the groups. ## The steps Adding a study changes nothing in `polyphys/`, which ships no study's parser: a study declares its parser in its own folder, as each of the maintainer's published studies does in [polyphys-lab](https://github.com/amirhs1/polyphys-lab). 1. Declare the study's parser class, as in "Declaring the parser" below. 2. Make a folder for the study as a Python package. Its `spec.py` builds the study's `ProjectSpec` (as `SPEC`), its `ProbeSpec` (as `PROBE`), and a tuple of its `AnalysisTable` records, one per particle group (as `ANALYSIS`); its `__init__.py` exports the three. The [Probe](probe.md) page, "A study's probe as data", and the [Pipeline](pipeline.md) page, "A study's analysis as data", describe the last two. 3. Build the specs. Each attribute a `ProjectSpec` names is checked against the parser when the spec is built, so a typo raises when `spec.py` is imported, not when the analysis runs. 4. Test the study: check its specs against its parser and against the values its analysis used before, as polyphys-lab's `tests/test_studies.py` does for the maintainer's studies. 5. Run the study's data through the phases, as the [Workflow](workflow.md) page describes. The maintainer's published studies are in the companion repository [polyphys-lab](https://github.com/amirhs1/polyphys-lab), one folder per study, each installed as `polyphys_lab.`. PolyPhys does not depend on it; a study elsewhere uses the same layout in its own repository. In PolyPhys 1.0.0, importing one of these parsers from `polyphys.manage.parser` raises an `ImportError` that names `projects//parser.py` in the PolyPhys repository; that folder is now `projects//python/` in polyphys-lab. The [Design decisions](decisions.md) page records why each study is a folder outside the package. ## Declaring the parser A parser class is a declaration: it lists the study's filename keywords, the order in which each lineage writes them, any fixed values, the particle species whose bulk number density and volume fraction to compute, and the attribute that holds each group's dump period (here fixed in `_defaults`). `ParserBase` does the parsing. Each keyword gets a `KeywordSpec` naming the attribute that stores its value and the value's type; a `float` keyword may also convert the value as `scale * value + offset`, here `lcube = 2 l` from the half-side `l`. The conversion is exact decimal arithmetic on the value's text, rounded once to a float, so the cylinder size `D = 2 r - 1` of `r4.1` is `7.2` and prints as `D7.2` in the short names. A bulk volume fraction is nominal, the species' bead volume over the volume its centers can reach, and can exceed one for a species nearly as wide as the box; a species with no room for its centers raises `ValueError` naming the state point: ```python >>> from polyphys.manage.parser import KeywordSpec, ParserBase, Species >>> class ChainCub(ParserBase): ... _geometry = 'cubic' ... _topology = 'linear' ... _groups = ['bug', 'all'] ... _dump_keywords = {'bug': 'bdump', 'all': 'adump'} ... _keywords = { ... 'N': KeywordSpec('nmon', int), ... 'ac': KeywordSpec('dcrowd', float), ... 'nc': KeywordSpec('ncrowd', int), ... 'l': KeywordSpec('lcube', float, scale=2.0), ... 'ens': KeywordSpec('ensemble_id', int), ... 'j': KeywordSpec('segment_id', int), ... } ... _patterns = { ... 'segment': ('N', 'ac', 'nc', 'l', 'ens', 'j'), ... 'whole': ('N', 'ac', 'nc', 'l', 'ens'), ... 'ensemble_long': ('N', 'ac', 'nc', 'l'), ... 'ensemble': ('N', 'ac', 'nc'), ... 'space': ('N', 'ac'), ... } ... _defaults = {'dmon': 1.0, 'bdump': 1000, 'adump': 5000} ... _species = (Species('m', 'nmon', 'dmon'), ... Species('c', 'ncrowd', 'dcrowd')) ... _project_attributes = { ... 'segment': ['dmon', 'rho_bulk_m', 'rho_bulk_c'], ... 'whole': ['dmon', 'rho_bulk_m', 'rho_bulk_c'], ... 'ensemble_long': ['dmon', 'rho_bulk_m', 'rho_bulk_c'], ... 'ensemble': ['dmon'], ... 'space': ['dmon'], ... } >>> artifact = ChainCub('N100ac2.0nc50l10.0ens1.bug', 'whole', 'bug') >>> artifact.lcube, artifact.ensemble, artifact.space (20.0, 'N100ac2nc50', 'N100ac2') >>> artifact.dump_keyword, artifact.bdump ('bdump', 1000) ``` The declaration is checked when the class is created: a pattern keyword without a `KeywordSpec`, an attribute that appears twice in one pattern, a species whose size or number no long-form name sets, or a group without a dump keyword raises `TypeError`. A name that lacks a keyword of its lineage's pattern, or whose value is not a number, raises `ValueError` naming the keyword, the lineage, and the name. A value that is a function of parsed values but fits none of the tables, such as a complementary percentage `pct_two = 100 - pct_one` of two crowder species, goes in an override of `_derived_attributes`, which runs at every lineage after the bulk values. ## Declaring the ProjectSpec A parser says how a study's names read; a `ProjectSpec` from `polyphys.spec` says how its analysis reads them. It holds what the parser does not; the geometry, topology, and glob patterns come from the parser. `parser` : The study's `ParserBase` subclass. `group` : The particle group of the polymer, such as `'bug'`. `divisor` : The step to which the bulk volume fraction of crowders is rounded in the `phi_bulk_c_round` column; a study choice with no default. `crowder_attr` : The parser attribute that holds that volume fraction, such as `phi_bulk_c`; a study choice with no default, keyword-only. `physical_attrs` : The parser attributes that become columns of the `space` tables. `species_sizes` : Each density-profile species label mapped to the parser attribute that holds that species' diameter. `tseries_properties` : The study's time series, such as `'gyrTMon'`. The spec for the cubic parser `ChainCub` declared above: ```python >>> from polyphys.spec import ProjectSpec >>> chain = ProjectSpec( ... parser=ChainCub, group='bug', divisor=0.025, ... crowder_attr='phi_bulk_c', ... physical_attrs=('space', 'ensemble_long', 'nmon', 'dcrowd'), ... species_sizes={'Mon': 'dmon', 'Crd': 'dcrowd'}, ... tseries_properties=('gyrTMon',)) >>> chain.geometry, chain.directions ('cubic', ('r',)) >>> chain.space_pattern, chain.space_hierarchy, chain.hierarchy ('N*ac*', 'N*', 'N*') ``` `space_pattern` matches the study's `space` names, and `space_hierarchy` and `hierarchy` are the glob prefixes of its `space` and `ensemble_long` names, which the stage functions take. Each attribute a spec names is checked against the attributes the parser sets at `ensemble_long`, so a typo or a missing declaration raises `ValueError` when the spec is built. The space stage reads its fields from the spec, as in `space_tseries_from_spec(database, chain, 'gyrTMon')` from `polyphys.analyze.pipeline`. A study that probes its trajectories with `polyphys.probe` builds a `ProbeSpec` beside its `ProjectSpec`: the MDAnalysis selection of each species, the label of the chain, the chain-size observables to measure, whether the chain is unwrapped for its asphericity, and the settings of its spatial histograms, pair distances, and binders; see [Probe](probe.md). The study's drivers and figure code will join its specs in its folder. ## Reading tables the research code wrote PolyPhys names a bulk quantity `_bulk_`; the research code it replaces named it `__bulk`, and wrote every parser attribute as a column of its `*-stamps.csv` files, from where the names carry into its ensemble-average and `space` tables. `BULK_NAME_MAP` in `polyphys.manage.organizer` maps each old name to its new one, and `rename_bulk_columns` applies the map to a table read from such a file, leaving every other column unchanged: | Research code | PolyPhys | | ------------------ | ------------------ | | `phi_m_bulk` | `phi_bulk_m` | | `rho_m_bulk` | `rho_bulk_m` | | `phi_c_bulk` | `phi_bulk_c` | | `rho_c_bulk` | `rho_bulk_c` | | `phi_c_bulk_one` | `phi_bulk_c_one` | | `phi_c_bulk_two` | `phi_bulk_c_two` | | `rho_c_bulk_one` | `rho_bulk_c_one` | | `rho_c_bulk_two` | `rho_bulk_c_two` | | `phi_hns_bulk` | `phi_bulk_hns` | | `rho_hns_bulk` | `rho_bulk_hns` | | `phi_c_bulk_round` | `phi_bulk_c_round` | ```python >>> import pandas as pd >>> from polyphys.manage.organizer import rename_bulk_columns >>> old = pd.DataFrame({'nmon': [20], 'phi_c_bulk': [0.1], 'n_frames': [100]}) >>> list(rename_bulk_columns(old).columns) ['nmon', 'phi_bulk_c', 'n_frames'] ``` The parser classes carry no alias for an old name: a name reaches a file only when a parser lists it in `attributes`, and an alias there would give every new table both columns.