VERITAS
The instrument that puts clinical-speech AI on trial.
A validity instrument for the training-free severity metrics that score dysarthria from frozen self-supervised embeddings. It does not report one more correlation — it cross-examines any such metric against six confound controls and returns a verdict.
Runs locally · FastAPI :8000 + Vite :5173Six controls. One verdict.
Every training-free metric arrives with one headline correlation. VERITAS runs six independent controls and aggregates them the honest way: one FAIL fails the card — no averaging a confound away.
One number, right for the wrong reason.
Training-free severity metrics are multiplying, and almost every one is “validated” by a single correlation on one dataset. That is not validation; it is one number that can be right for the wrong reason.
Audited properly, the failures are systemic. A popular trajectory-distance metric posts a strong ρ = 0.756 — yet 76% of that signal lives in silence: remove the pauses and it collapses to ρ = 0.181 (n.s.), while silence frames alone reproduce it (ρ = 0.775). Every metric’s frozen representation is almost perfectly speaker-identifiable (AUC 0.996–0.999, chance is 0.5) — a leakage risk no published metric reports checking. And the ground truth needed to check any of it barely exists: usable dysarthric EMA sessions in all of public data number n ≈ 2.
What it actually checks.
Each control is an independent question with a documented threshold. Any metric plugs in through one method (predictive_scores → {speaker: float}), so its validity becomes a re-runnable regression test.
Predictive validity
Does the score actually track clinical severity? The one number every paper reports — and, too often, the only one it checks.
Speaker-identity leakage
Under a leave-one-control-out reference, is the score just re-identifying the speaker? A leakage risk no published metric reports checking.
Silence & pausing
Strip the pauses. If the signal collapses, it was measuring how much the patient stopped talking, not how they articulated.
Speech energy
Control for loudness. A metric that only tracks how loud someone spoke is not measuring dysarthria.
Cross-corpus transfer
Does the metric survive a change of dataset, or was it quietly fit to one corpus’ quirks?
Representation identifiability
At the representation level, how speaker-identifiable is the frozen embedding? AUC 0.996–0.999, where 0.5 is chance.
The metric is a black box. VERITAS wires it to the bench, runs the controls, and signs the slip. Verdict: FAIL — do not discard.
Six components. Five deterministic, one that reasons.
Six-control validity battery — predictive validity, identity leakage, silence, energy, cross-corpus transfer, and representation identifiability, each with a published PASS / WARN / FAIL threshold.
Metric-agnostic plug-in API — implement one method and any severity metric (reference, probe, or intrinsic) receives a machine-readable report card.
A state-of-the-art audit, done fairly — an independent reimplementation of the strongest published metric family (phonological-subspace d′) is certified where sound, and caught on the one confound its authors never checked.
Live audit console — a real FastAPI backend runs the production threshold code on your own numbers; four one-click presets load the real committed metric profiles.
Grounded in real articulography — validated against AG500 electromagnetic tongue-sensor data, exposing that dysarthric EMA ground truth is effectively n ≈ 2.
Fully reproducible research stack — ~50 seeded, numbered pipeline scripts; ~57 tests; two papers; every headline number footnotes a committed JSON.
Client
Server
AI
Data
- TORGO (n=15 + AG500 EMA)
- IPVS (n=61)
- EasyCall (n=55)
- Speaker-disjoint splits
- Frozen WavLM-Large, 1024-d
- SPARC articulatory codes
- MFA phoneme / word alignments
- Cached embeddings
- ATD · PCA-64 + DTW reference
- Pooled WavLM Ridge probe
- Phonological-subspace d′ (SOTA)
- thresholds.py policy
- battery.py · six controls
- stats.py · Spearman / partial / LOCO
- report-card · veritas-bench/1.0
- FastAPI · /audit /report-cards /summary
- Imports the real policy via importlib
- React / Vite console
- Live verdict matrix
Six controls. One verdict.
- 76% of the signal was silence.
- AUC 0.999 — the metric is the speaker.
- Partial ρ = −0.001 — it measured loudness.
I built the machine that catches the machines.
A whole family of clinical-speech metrics reports one correlation and calls it validated. I kept finding that the number was real and the reason was wrong — the score was tracking silence, or loudness, or simply who the speaker was.
So I stopped writing another metric and built the instrument that interrogates them. Six controls, documented thresholds, one FAIL fails the card. The exact policy that grades the papers is the same code the console runs live — delete a result file and the site goes blank. Seed 42, frozen weights, speaker-disjoint splits, every figure traced to a committed file.
It is a released test-suite, a live audit console, and a two-paper contribution — built solo, on real dysarthric speech. Validity, not accuracy. That is the whole point.
