PHI detection: a local baseline
The figures on this page are what our detection pipeline measured on a labeled corpus we wrote, run locally, in August 2026. They are published so that a buyer can see how detection is measured before asking us to promise anything about it. They are a baseline, not a claim of sufficiency.
The boundary
The guarantee is post-detection.
Once content is detected as sensitive, it does not reach a model in untransformed form. Detected values are tokenized before the model is invoked. Detection itself is probabilistic. It misses some identifiers and over-detects others, and the numbers below measure exactly that. Nothing on this page should be read as a guarantee that every identifier is detected.
Precision
0.835
Recall
0.849
F1
0.842
173 casesClinical and operational text.
238 labeled spansEvery class has at least 10.
17 classesScored at span level.
A detection counts as a match when it overlaps the labeled span by at least half, an intersection over union of 0.5 or more. A boundary disagreement counts as a miss, not partial credit.
How the measurement works
The corpus is a set of short texts of the kind a clinical or operational user would actually send: chart fragments, scheduling notes, referral text, billing lines, device references, and account references. Each text is hand-labeled with spans that are identifiers under the HIPAA Safe Harbor list. The pipeline runs over every text, and each detected span is scored against those labels.
Precision is the share of detected spans that were labeled identifiers. Recall is the share of labeled identifiers that were detected. The micro average weights every span equally, so the figure is dominated by the classes with the most spans.
A second harness measures type-level detection on 257 text variants derived from the same cases by perturbing format, spacing, and surrounding text. A variant passes only if every identifier type it contains is detected. 219 of 257 variants pass, a rate of 0.852.
The run is reproducible from the repository's evaluation harnesses. They fail rather than report if the detection service is unreachable, and compute the micro totals instead of leaving them to be summed by hand.
Revision history matters
What changed between runs, and why we publish it
The first internal run understated recall because a configuration defect discarded the output of some detection rules. It was fixed and is now guarded by a test that fails if it recurs.
The second run measured 45 cases, where ten classes rested on a single labeled span and read a perfect 1.000. Deepening the corpus to 238 spans, with every class represented by at least 10, dropped several classes to between 0.400 and 0.800 without a detector change. We publish the deeper figures. A 1.000 on one example means the one example works.
What this baseline supports
On this corpus, six identifier classes reach 1.000 recall at 10 or more spans each: email addresses, IP addresses, medical record numbers, other unique identifying numbers, Social Security numbers, and phone numbers. In a separate adversarial set, the pipeline reaches 1.000 across format evasion, prompt injection, Unicode character substitution, and whitespace injection.
What this baseline does not support
It does not support a claim of overall detection accuracy.
- 01
Several identifier classes are detected in one format each today, and the corpus includes second formats they miss.
- 02
Over-detection has a cost when detection causes transformation. Two classes sit below 0.65 precision, and the run recorded false positives from types absent from the labeled text.
- 03
This is a local harness measurement, not production telemetry. It used the same analyzer model configuration as production, but has not been re-measured against production traffic.
- 04
The 238 spans support comparisons among these classes. They are not a population sample, and the corpus was written by the same team that wrote the detector.
Known evasion classes
The adversarial set contains two documented evasion classes the primary detector does not cover. They are marked expected-fail in the harness and tracked as detector work. Details are available to customers and diligence reviewers under NDA.