Is this crystal a metal?¶
metallicity.is_metal.cgcnn — a crystal graph network that answers one
question: does DFT give this crystal a zero band gap.
It matters because metals need denser k-point sampling than insulators, and they need smearing, which insulators do not. Get it wrong towards "insulator" and the calculation still runs, still converges, and quietly returns a number you cannot trust.
On 10 625 held-out structures it catches 97.2% of metals. What that costs, and why it is tuned that way rather than for accuracy, is the second half of this notebook.
Setup¶
Two things: the model, and the atomic embedding table it was built against.
The table comes from the published PSDI record. It is part of the model, not a detail — swap it and the graphs change, so the answers change, silently. That is why its checksum is pinned in the model's record and checked on load.
Run this from a clone of the repository. The weights are not published yet, so they are copied from local_data/; when the record goes live they will download like the table does.
import shutil
from pathlib import Path
from urllib.request import urlretrieve
PSDI = "https://data-collections.psdi.ac.uk/api/records"
work = Path("goldilocks-metallicity")
work.mkdir(exist_ok=True)
# The atomic embedding table lives in its own published record. It is part of
# the model rather than a setting: a different table changes the graphs, so the
# answers change, silently. Its digest is pinned and checked on load.
atom_init = work / "atom_init.json"
if not atom_init.exists():
urlretrieve(f"{PSDI}/m742g-g0k14/files/atom_init.json/content", atom_init)
# The weights and the record. Once this model is deposited both come from a
# PSDI record like the table above; until then they come from the repository.
DEPOSIT = Path("../../deposits/metallicity/is_metal/cgcnn")
WEIGHTS = Path("../../local_data/models/metallicity/is_metal/cgcnn/is_metal.pt")
shutil.copy(DEPOSIT / "model.json", work / "model.json")
shutil.copy(WEIGHTS, work / "is_metal.pt")
for path in sorted(work.iterdir()):
print(f"{path.name:16} {path.stat().st_size / 1e6:8.3f} MB")
atom_init.json 0.028 MB is_metal.pt 0.502 MB model.json 0.017 MB
from goldilocks_ml.inference import load_model
model = load_model(work, artifacts={"atom_init": atom_init})
print(model.model_id)
metallicity.is_metal.cgcnn@crystal_graph.v1
Ask it about six crystals¶
Three that conduct and three that do not. No Materials Project account needed — these are textbook structures, built from a lattice constant and a space group.
from pymatgen.core import Lattice, Structure
def cubic(spacegroup, a, species, sites):
return Structure.from_spacegroup(spacegroup, Lattice.cubic(a), species, sites)
ORIGIN = [[0.0, 0.0, 0.0]]
ROCKSALT = [[0.0, 0.0, 0.0], [0.5, 0.5, 0.5]]
crystals = {
"Cu (fcc)": (cubic("Fm-3m", 3.615, ["Cu"], ORIGIN), "metal"),
"Al (fcc)": (cubic("Fm-3m", 4.050, ["Al"], ORIGIN), "metal"),
"Fe (bcc)": (cubic("Im-3m", 2.867, ["Fe"], ORIGIN), "metal"),
"Si (diamond)": (cubic("Fd-3m", 5.431, ["Si"], ORIGIN), "insulator"),
"NaCl (rocksalt)": (cubic("Fm-3m", 5.640, ["Na", "Cl"], ROCKSALT), "insulator"),
"MgO (rocksalt)": (cubic("Fm-3m", 4.212, ["Mg", "O"], ROCKSALT), "insulator"),
}
predictions = model.predict_batch([structure for structure, _ in crystals.values()])
print(f"{'crystal':17} {'says':11} {'score':>7} {'known to be':11}")
for (name, (_, known)), prediction in zip(crystals.items(), predictions):
says = prediction.details["label"]
mark = " " if says == known else " <-- wrong"
print(f"{name:17} {says:11} {prediction.details['score']:>7.3f} {known:11}{mark}")
crystal says score known to be Cu (fcc) metal 0.984 metal Al (fcc) metal 0.978 metal Fe (bcc) metal 0.992 metal Si (diamond) insulator 0.012 insulator NaCl (rocksalt) insulator 0.006 insulator MgO (rocksalt) insulator 0.008 insulator
Six for six, and none of them close.
Every score above sits well clear of both lines: the metals score above 0.97, the insulators below 0.02, and the threshold in use is 0.0478. A textbook crystal is exactly the case a model should find easy, so agreement here is expected rather than impressive — it does not by itself say anything about where the threshold matters.
Where it matters is the structures a real snapshot has that these six do not: ones the network is genuinely unsure about. What the floor costs below measures that on 10 603 real validation structures instead of six hand-picked ones — 137 metals still missed even with the floor in place, against 674 if the threshold were left to maximise MCC instead.
What you get back¶
Not a probability — a decision. The seam hands a consumer one value it can act on, and keeps the number behind it as provenance.
prediction = predictions[0]
print("value ", prediction.value, f"({type(prediction.value).__name__})")
print("parameter ", prediction.parameter)
print("quantity ", prediction.quantity)
print("target_contract ", prediction.target_contract)
print("confidence ", prediction.confidence)
for key, value in prediction.details.items():
print(f"details[{key!r}]".ljust(32), value)
value True (bool) parameter metallicity quantity is_metal target_contract goldilocks.is_metal.dft_band_gap_zero.v1 confidence None details['score'] 0.9843800067901611 details['threshold'] 0.04782969690859318 details['label'] metal details['score_is'] uncalibrated positive-class softmax details['threshold_selected_on'] validation details['threshold_metric'] mcc details['min_recall'] 0.97
Why confidence is empty¶
Because it carries a guarantee, and this score is not one.
The k-distance model puts 0.9 there: a conformal coverage level, which is
provable. A neural network's score is not, even when it behaves well — and this
one does behave well, tracking observed frequency closely on held-out data. But
"measures well" and "is proven" are different claims, and a consumer comparing
the two fields would be comparing different things. Estimates travel in
details.
The line is at 0.0478, not 0.5¶
details['threshold'] is far below the halfway point you might expect. That is
deliberate, and it is the most important thing about this model.
The two mistakes do not cost the same:
| Mistake | Consequence |
|---|---|
| calling a metal an insulator | mesh too coarse, Fermi surface undersampled, answer can be wrong without looking wrong |
| calling an insulator a metal | denser mesh than needed, costs compute |
One is a wrong answer. The other is a bill. Choosing the threshold that maximises accuracy — or MCC, or F1 — treats them as interchangeable, so the protocol constrains the search instead: of the thresholds that miss no more than 3% of metals, take the best one.
None of the six crystals above happen to sit near that line for this model — see the note above. What it costs is concrete anyway: 137 real metals in the validation set still get called insulators even with the floor in place, each one a mesh too coarse for a Fermi surface the calculation will not know it missed. The floor's job is to keep that number down, not to reach zero — the next section is the full trade.
threshold = prediction.details["threshold"]
print(f"threshold in use: {threshold:.4f}\n")
print(f"{'crystal':17} {'score':>7} {'at 0.5':11} {'in use':11} {'truth':11}")
for (name, (_, known)), p in zip(crystals.items(), predictions):
score = p.details["score"]
naive = "metal" if score >= 0.5 else "insulator"
print(f"{name:17} {score:>7.3f} {naive:11} {p.details['label']:11} {known:11}")
threshold in use: 0.0478 crystal score at 0.5 in use truth Cu (fcc) 0.984 metal metal metal Al (fcc) 0.978 metal metal metal Fe (bcc) 0.992 metal metal metal Si (diamond) 0.012 insulator insulator insulator NaCl (rocksalt) 0.006 insulator insulator insulator MgO (rocksalt) 0.008 insulator insulator insulator
What the floor costs¶
Buying recall costs precision, and the price rises steeply. Measured on the validation split, 10 603 structures of which 4 585 are metals:
| Recall floor | Threshold | Precision | Metals missed | False alarms |
|---|---|---|---|---|
| none (best MCC) | 0.486 | 0.901 | 674 | 431 |
| 0.95 | 0.111 | 0.755 | 228 | 1 414 |
| 0.97 | 0.0478 | 0.669 | 137 | 2 204 |
| 0.99 | 0.015 | 0.555 | 45 | 3 639 |
Going from the unconstrained threshold to 0.95 saves 446 metals for 983 extra false alarms. Going from 0.97 to 0.99 saves 92 more for 1 435. The useful range ends before 0.99, where 77% of structures would get a dense mesh anyway — against the 100% of not classifying at all.
0.97 rather than 0.95 buys margin: a floor is met on the validation split, which is a sample. The 0.95 threshold delivers 0.9498 recall on test, below its own floor. The 0.97 one delivers 0.9717, and so keeps 0.95 as well.
Where this goes next¶
Goldilocks Core uses metallicity in two places: how dense a k-point mesh needs to be, and whether to apply smearing. Today it reaches the first only indirectly — the k-distance model's feature vector embeds a metallicity network's learned representation — and answers the second with a rule of thumb: smearing only when every element in the structure is metallic. That rule sends RuO₂, ReO₃, LaNiO₃ and TiN down the insulator path.
What this model is not¶
- It answers "is the DFT band gap zero", using Materials Project labels. That is a computed property under one functional, not a measurement.
- It is not a substitute for an electronic-structure calculation. Treat an unusual chemistry as unverified.
- It is deliberately biased. Roughly a third of what it calls metal is not, and that is the trade being made on purpose.