Discrimination Backtest — Does Watchlist Rank Predict Designation?
Generated 2026-08-17T02:51:43+00:00
Result. Among entities our data links to a sanctioned party, watchlist rank separates the companies designated in the following year from those never designated — AUC 0.6627 at the start of the invasion wave rising to 0.7863 by 2025, improving every year. A company on the qualified watchlist is designated the next year at 124–238× the rate of a random Russian company. This is the ranking counterpart to the recall backtest: that one asks whether we flag designations at all; this asks whether we rank them near the top.
The recall backtest asks a yes/no question (did any pre-designation link exist) and never uses the score that orders the watchlist. This asks the ranking question directly: are the entities we rank highest the ones actually designated next? Point-in-time landmark case-control — at each Jan-1 date every entity is scored from links dated strictly before it, gated so the counterparty was already sanctioned when the link formed.
Two AUCs, two questions. AUC-population vs a random EGRUL company (symmetric zeros) folds selection and ranking together — being on the list plus where on it. AUC-within-visible compares only entities already linked to sanctioned parties, isolating pure ranking quality. 0.50 = no better than a phone book; 1.0 = perfect. Recall-visible is the share of window designations our data could see at all before T (the recall ceiling). A never-designated control is not a false positive — designation is a throttled sample of the sanctionable set — so this scores ranking, not calibrated probability.
Per-landmark discrimination
The last column is an experiment, not the shipped score: it adds a person-hub channel (a director tenured on ≥2 already-sanctioned boards joins the entity, nominee-capped) to the rank key — the gap to AUC-within-visible is that channel's would-be marginal ranking contribution.
| Landmark T | Cases (visible) | Recall-vis | AUC-population | AUC-within-visible | AUC-visible +person-hub (exp.) |
|---|---|---|---|---|---|
| 2022-01-01 | 1,589 (515) | 32.4% | 0.6612 | 0.6627 | 0.6626 |
| 2023-01-01 | 3,917 (972) | 24.8% | 0.6226 | 0.7322 | 0.7324 |
| 2024-01-01 | 1,578 (419) | 26.6% | 0.6303 | 0.7546 | 0.7545 |
| 2025-01-01 | 1,031 (299) | 29.0% | 0.6414 | 0.7863 | 0.7865 |
Designation enrichment by score bucket (sampling-corrected, next-1yr window)
Lift = how many more times likely a company in the bucket is to be designated in the next year than a random EGRUL company. qualified is the published watchlist definition (≥2 link types or ≥3 sanctioned ties).
| Landmark T | Base rate | flagged lift | qualified lift | multi-channel lift | ≥3-channel lift |
|---|---|---|---|---|---|
| 2022-01-01 | 0.0128% | 161.8× | 238.2× | 983.5× | 834.6× |
| 2023-01-01 | 0.0316% | 75.9× | 136.3× | 443.6× | 3159.7× |
| 2024-01-01 | 0.0128% | 47.7× | 129.9× | 349.0× | 7840.8× |
| 2025-01-01 | 0.0083% | 35.0× | 123.7× | 359.4× | 1655.1× |
Pooled (indicative — controls recur across years)
- Cases 8,115 (2,205 visible pre-window)
- AUC-population 0.6337 · AUC-within-visible 0.766 (experimental +person-hub channel: 0.7661)
- Base designation rate 0.0164%; being on the qualified watchlist raises it 143.1×, ≥3-channel 1547.7×.
Reading it. AUC-within-visible isolates ranking given a link exists (does breadth/tie-count order the linked crowd correctly); AUC-population adds selection (being flagged at all vs a random company). Recall-visible is the orthogonal recall ceiling — the share of designations our data sees pre-designation. The enrichment table is the buyer-facing number: work the qualified list and you hit next-year designations at many times the base rate. A never-designated control is not a false positive — designation is a throttled sample of the sanctionable set, so this is enrichment of real designations, not a precision claim.