⚖️Detection Logic & Accuracy

The settled detection mechanism + provisional accuracy (Wilson 95% CI) + a workbench for accuracy improvement. Real FPR is measured on-device, still ahead.

One place to record the detection logic (the settled mechanism) and its accuracy (provisional, with confidence intervals) — a workbench for the discussion of making accuracy better.

⚠️FPR discipline (how to read the numbers on this page)

Every accuracy figure here is provisional and carries a Wilson 95% confidence interval. No point-estimate claims. The real FPR (how often legitimate calls get falsely blocked) will be measured in the Chris→Hide on-device test (n≥20) — still ahead. Today’s numbers come from a semi-synthetic gold set and are not an operational guarantee.

Part 1: The detection mechanism (settled)

The design is two-stage: first a rule layer (deterministic, fast), then an LLM layer (reads context).

ℹ️Honest status: only the rule layer is live

The champion in production is the rules layer only. The LLM layer exists in code (classify_call / measured probes) but is not on the main line (shelved). Promotion, when it happens, is an internal swap inside the verdict port — callers don’t change.

Rule layer — the fixed checker order (the order itself is the spec)

Blocklist check — match → blocked immediately (before the allowlist)
Whitelist check — match → direct_transfer immediately
Business-hours check — outside hours → after_hours
Auth keyword — match → transferred (score=0.0; only the caller's own utterances are used)
Keyword scoring — weights high=0.3 / medium=0.15 / low=0.05, overridable per client
Opener floor — authority-frame opener detected → OR-merge (max) a 0.6 floor into the score; never terminates alone, only lifts into the grey band
Category detection — SpamCategory (DB) keyword match → allowed_sales / form_guided
Threshold verdict — score ≥ 0.8 → form_guided / ≤ 0.3 → transferred / in between → grey_transferred

Thresholds (the SSoT owns the confirmed values — this is a copy)

Name	Value	Meaning	Source
`block_threshold`	0.8（既定）	at or above → form_guided (terminate side); per-client override via EngineConfig	`backend/engine/scorer.py`
`grey_threshold`	0.3（既定）	at or below → transferred; in between → grey_transferred (to a human)	`backend/engine/scorer.py`
`opener floor`	0.6	floor for opener impersonation; below the 0.8 safety valve, so it never terminates alone by design	`backend/engine/opener_floor.py`
`AEGIS_TERMINATE_MIN_SCORE`	0.8	termination safety valve; score-based terminations (form_guided) derive should_terminate=true only at or above this	`SSoT / backend/config.py`

The confirmed values (0.8 / 0.3 / 0.6 / safety valve 0.8) are consistent with the SSoT. Change the SSoT first.

LLM layer — the classify_call classification (5 values)

existing_contact — existing business partner / known contact
new_business — new business / inquiry
sales_pitch — sales / solicitation
support_request — support request
unknown — none of the above / undeterminable

verdict port — the single entrance to the classification brain (contract highlights)

Every path that asks for a verdict (transcript-tick / transcript-final / the bridge's classify) calls verdict_port.evaluate(). Whether the champion is rules or llm is the port's internal concern, invisible to callers.
The source of truth is always action. should_terminate / should_transfer are projections (derived values), never computed independently outside the port (no duplication).
Terminate-side actions = form_guided / after_hours / blocked. Transfer-side = transferred / direct_transfer / grey_transferred / allowed_sales.
L1 deterministic gates (after_hours / blocked) derive should_terminate=true unconditionally. Score-based (form_guided) is true only when spam_score ≥ 0.8 (the safety valve).
Fail-open: empty transcript → pending, internal exception → error; both derive false booleans (falling toward a human).
The verdict's origin is labeled in call_logs.score_source ('rules' / 'llm' / NULL), so which brain produced a value stays traceable.

Part 2: Accuracy / eval (provisional, with confidence intervals)

What is being measured (the gold set today)

Gold set: v3 (42 items = spam 21 / not_spam 21) — on the feat/eval-gold-v01 branch (unmerged); gold_set.json on main is a 2-item skeleton only
Provenance: public 10 (Chiba Police transcripts etc.) / synthetic-grounded 20 / synthetic 12; real = 0 items
Probes: all 42 items carry llm_observed (measured probes against gpt-realtime GA, commit da55690)

Measurements (the fraud-miss A/B, provisional)

Metric	before	after (plan A)	Caution
Fraud misses (10 special-fraud items)	7/10	3/10 [10.8%, 60.3%]	n=10, so the interval is very wide — trend only
Overall FNR (miss rate on spam)	33.3% [17.2%, 54.6%]	14.3% [5.0%, 34.6%]	n=21 (spam side); the intervals overlap, so nothing stronger than "an improving trend" can be claimed
FPR (false-block rate on legitimate calls)	0.0% [0.0%, 15.5%]	0.0% [0.0%, 15.5%]（維持）	n=21 (not_spam side). Even with 0 events the upper bound is 15.5% — this is not "zero FPR"

Measurement identity: gold v3 × llm_observed (gpt-realtime GA probes) × feat/eval-gold-v01 commit 4be9493 (the A/B comparison). before = baseline (da55690), after = plan A (fraud enum added).

⚠️These are LLM-layer probe values (not the production line's)

The table above turns the LLM layer’s measured probes (llm_observed) into block/pass at a 0.8 threshold. It is not the production rules layer’s operational value — and the source is an unmerged branch (feat/eval-gold-v01). With small n and overlapping intervals, nothing stronger than “an improving trend” can be claimed.

FPR discipline (eval design D4 / D5)

FNR / FPR are always reported with Wilson 95% confidence intervals, and acceptance is judged on the interval bounds (no point-estimate claims).
Every gold item carries a provenance (real / public / synthetic-grounded / synthetic), and metrics are reported both overall and real-only (to detect overfitting to semi-synthetic data). With real = 0 items today, real-only cannot be measured yet.
The real FPR will be measured in the Chris→Hide on-device test (n≥20) — still ahead. Today's numbers are provisional values on a semi-synthetic gold set.

The improvement workbench

The open questions for making accuracy better. When a discussion moves, update this page (src/data/detection.ts).

🧠 When (and on what evidence) to promote the LLM layer to champion

The champion today is the rules layer. Promotion is designed as an internal swap inside the verdict port (no caller changes).
Evidence for promotion = interval comparison of rules vs llm on gold, plus on-device measurements. No promotion while the intervals overlap.

🕵️ Fraud-miss countermeasure: choosing between plan A (fraud enum) and plan B (0.6 unknown band)

Both plans reduce misses 7/10 → 3/10 (provisional). Plan A moves six fraud scores up to 0.95–0.99 (a wide margin); plan B leaves them stuck at 0.6–0.7 (thin ice).
Plan A leads, but is unmerged (feat/eval-gold-v01). The open question is whether to wait for on-device results before merging.

📞 Acquiring real provenance (the biggest hole in the gold set)

Zero of the 42 gold items are real; overfitting to semi-synthetic data is currently undetectable.
The path is ready: calls from the Chris→Hide on-device test (n≥20) get PII-masked and promoted into gold as real provenance via the gold_candidates queue.

📏 Record the rules layer's own gold measurements

eval_scorer.py --source rules can measure the production rules layer on the same 42 gold items, but those numbers are not yet recorded on this page.
Next step: put rules vs llm side by side with intervals — the foundation for the champion-promotion decision.

🏷️ Working through the labeling queue (PII masking required)

A labeling queue of real transcripts (untracked in git, potentially containing PII) exists locally. PII must be masked before promotion (the queue file says so itself).
Who labels, and against which criteria, is undecided. gold_category assignment should follow the taxonomy.

📌How to update this page

Numbers and open questions live in src/data/detection.ts. Every new measurement must be tied to “gold version × target (rules / llm) × commit” and carry a Wilson 95% CI. When on-device results arrive, don’t replace the provisional values — show both (the gap versus the semi-synthetic gold is itself information).