MD-EXAFS ecosystem integration

MD-EXAFS ecosystem integration#

This document outlines how md-exafs connects with AiiDA-FEFF and AiiDAlab-EXAFS.

1. Ecosystem topology#

                  ┌───────────────────────────────┐
                  │       MD-EXAFS Core           │
                  │        (`md_exafs`)           │
                  │  (Pure Python, zero-AiiDA)    │
                  └───────────────┬───────────────┘
                                  │
         ┌────────────────────────┴────────────────────────┐
         ▼                                                 ▼
┌─────────────────────────────────┐       ┌─────────────────────────────────┐
│           AiiDA-FEFF            │       │         AiiDAlab-EXAFS          │
│        (`aiida-feff`)           │◄──────┤        (`aiidalab-feff`)        │
│ • Slurm/PBS HPC Daemon Jobs     │       │ • Jupyter / AiiDAlab Web App    │
│ • ExafsArchiveData Node         │       │ • In-Memory DW Pre-screening    │
│ • Remote Batch Sharding         │       │ • Interactive Paths Explorer    │
│ • Graph Provenance Tracking     │       │ • Workflow Submission Wizard    │
└─────────────────────────────────┘       └─────────────────────────────────┘

The division of responsibilities is strict:

  1. md_exafs contains all physics, statistical estimators, card generation rules, and HDF5 serialisation routines. It has no knowledge of AiiDA or web frameworks.

  2. aiida-feff handles HPC scheduler submission, remote data staging, daemon monitoring, and relational graph provenance. This is the actual aiida plugin.

  3. aiidalab-exafs provides interactive user interface widgets within Jupyter and AiiDAlab.

2. Shared data architecture: ExafsArchiveData#

ExafsArchiveData (implemented in aiida_feff.data.archive) wraps either a batch shard (batch_shard.h5) or an ensemble archive (ensemble_results.h5). It delegates array and path queries directly to md_exafs.hdf5.ArchiveReader:

# Accessing properties on an ExafsArchiveData node
archive_node: ExafsArchiveData = workchain.outputs.archive

# Primary spectra (delegates to md_exafs.hdf5.ArchiveReader)
k = archive_node.k
chi = archive_node.chi
r = archive_node.r
chir_mag = archive_node.chir_mag

# Querying path contributions
paths = archive_node.iter_paths()

# Legacy converter for external tools expecting XasData
xas_node = archive_node.to_xas_data()

3. Remote cluster execution#

When EnsembleExafsWorkChain executes calculations on a remote cluster:

  1. Remote execution: The batch runner script _run_batch.py runs on the cluster node, delegating directly to md_exafs.execution.BatchExecutor.

  2. Local sharding: FEFF processes run concurrently on the allocated cores. Outputs are streamed into a single batch_shard.h5 archive on node scratch.

  3. Network transfer reduction: Rather than retrieving thousands of individual xmu.dat, chi.dat, and feffNNNN.dat files across SFTP, AiiDA retrieves only batch_shard.h5, reducing the file transfer count by more than 95%.

  4. Ensemble aggregation: The workflow invokes the calcfunction merge_exafs_shards(*shards). This calls md_exafs.execution.merge_shards, which computes ensemble averages, calculates Fourier transforms, and emits a single consolidated ExafsArchiveData node.

4. Parameter delegation#

FeffParameters in AiiDA-FEFF is implemented as a thin Dict adapter. Rather than maintaining separate validation and card serialisation rules, it delegates directly to md_exafs.FeffConfig:

from aiida_feff.data.parameters import FeffParameters

params = FeffParameters(dict={
    "edge": "K",
    "radius": 5.5,
    "s02": 0.9,
    "scf": "4.0 0 30 0.2 1",
})

# Under the hood, delegates validation and tag normalisation to md_exafs
feff_config = params.to_feff_config()
user_tags = params.to_pymatgen_user_tags()

5. In-memory trajectory screening in AiiDAlab#

Debye–Waller and MSRD analysis involves only minimum-image geometric distances and coordinate moments; it requires no electronic structure calculations and negligible compute time. Storing intermediate MSRD results as provenance-tracked AiiDA nodes adds database overhead without scientific benefit.

In aiidalab-exafs, trajectory screening is provided by aiidalab_feff.dw_widget.DebyeWallerScreeningWidget. When a trajectory is uploaded, the widget invokes md_exafs.debye_waller in-memory. It computes \(\sigma^2(R_\text{eff})\) and isotropic thermal \(B\)-factors which can be used to help constrain FEFF fits or just to inform the user about the expected thermal disorder in the system.

6. Shared visualisation engine#

Both the Marimo notebook (notebooks/paths_explorer.py) and the AiiDAlab widget (aiidalab_feff.widgets.paths_explorer) consume md_exafs.viz:

  • group_path_results() ensures consistent path clustering and distance binning across both interfaces.

  • build_chi_chart() and build_chir_chart() construct identical Altair spectral charts.

  • calculate_path_vectors() supplies 3D polyline vectors to WEAS-widget.