Metrics &
Methodology
How Willow measures continuity, coherence, drift, and context efficiency across AI workflows.
Willow is designed to help AI systems preserve the active thread across model calls, sessions, and workflows. We evaluate that through coherence signals, continuity artifacts, drift checks, compression tests, and fresh-session resume behavior.
Six signals, one continuity model.
Drift is one input. Willow's larger model centers on continuity and coherence — what the system is trying to do, whether it is still doing it, and what to do next.
Thread Integrity
Measures whether the response remains aligned with the active goal, constraints, facts, and intended task.
Fragmentation Risk
Flags whether the conversation or workflow appears stable, drifting, or in need of verification or re-anchoring.
Continuity Resume
Measures whether a system can resume the correct active frame in a fresh session or model call.
Context Efficiency
Measures how much compact continuity artifacts reduce the need to replay full conversation history.
Drift / Stability
Measures movement away from the intended active frame — goal loss, constraint loss, stale context, or unsupported re-anchoring.
Recommended Action
Returns a suggested next step such as answer, verify, re-anchor, or repair.
The thread is fragile by default.
AI systems often lose coherence because they rely on raw context windows, repeated instructions, and provider-specific memory behavior. As workflows get longer, models forget constraints, overweight recent text, lose the original task, or produce a plausible answer that no longer matches the user's intent.
A signal, not a verdict.
Thread Integrity Score estimates whether a response stayed aligned with the active frame. Scores are operational signals — Willow exposes them so applications can decide when to answer, verify, repair, or re-anchor.
{
"thread_integrity_score": 84.186,
"fragmentation_risk": "watch",
"recommended_action": "verify",
"trace_id": "trc_...",
"intent_vector": {
"goal": "preserve active launch plan",
"constraints": [
"do not overclaim",
"keep metrics traceable"
],
"uncertainty_flags": [
"pricing still finalizing"
],
"verification_needed": true
}
}The application stays in control.
Willow does not force a model to be correct. It gives the application a coherence signal before the failure becomes invisible.
| Risk state | Meaning | Typical action |
|---|---|---|
| stable | Response appears aligned with active frame | answer |
| watch | Potential drift, uncertainty, or missing constraint | verify |
| repair | Response likely needs re-anchoring or correction | repair / re-anchor |
What drift actually looks like.
Drift measures how far a response moves away from the active frame. Willow looks for signs such as omitted constraints, goal substitution, stale context, unsupported assumptions, contradiction, and failure to carry forward decisions. We use 'stability' as the more user-friendly label where appropriate.
Lower measured drift generally indicates stronger frame preservation, but drift scores should be interpreted with the task, provider, and benchmark setup. Full scoring formulas and weights are proprietary and not publicly disclosed.
Fresh-session resume, side by side.
In fresh-session resume tests, baseline model calls and Willow-assisted calls are asked to recover the active frame after context changes or session resets.
| Provider | Baseline resume | Willow resume | Drift change |
|---|---|---|---|
| OpenAI GPT-4.1 Mini | 37.5% | 100% | 0.375 → 0.162 |
| Anthropic Claude Haiku 4.5 | 50% | 75% | 0.400 → 0.162 |
| Gemini 2.5 Flash | 12.5% | 50% | 0.125 → 0.000 |
These are controlled private tests, not a universal benchmark claim. Broader validation is in progress.
Preserve the frame, not the transcript.
Willow avoids replaying full raw conversation history when a compact continuity artifact can preserve the active frame.
In this long-context compression test, Willow preserved the required active facts using a compact artifact estimated at 14.6% of the source transcript length.
Compression savings depend on workflow shape, model, prompt strategy, and implementation.
Verified on the live stack.
Careful about what we say.
Continuity is a hard problem. We would rather be precise than loud.
Willow does not currently claim
- zero drift
- guaranteed ROI
- elimination of hallucinations
- universal benchmark leadership
- production validation across every domain
- exact cost savings for every team
- public disclosure of proprietary scoring internals
Instead Willow claims
- controlled private tests show promising continuity and compression improvements
- Willow exposes coherence signals developers can inspect
- Willow helps applications preserve active context without raw transcript replay
- broader validation is ongoing
- proprietary scoring and orchestration logic remain protected
Test it against your real failure modes.
- 01Choose a workflow where your AI system loses context.
- 02Run the same task through your baseline.
- 03Run it with Willow Core.
- 04Compare resume accuracy, drift/stability, token usage, repair time, and output quality.
- 05Inspect thread_integrity_score, fragmentation_risk, recommended_action, and continuity artifacts.
- 06Repeat across multiple providers and session resets.
Quick answers.
A short companion to the full FAQ.
Are these public benchmarks?
Not yet. These are controlled private tests and early validation results. Public methodology packaging and broader third-party validation are in progress.
Can developers inspect the outputs and evaluation approach?
Yes. Willow exposes coherence fields, continuity artifacts, and benchmark methodology at the integration level. Proprietary scoring internals, weighting logic, repair algorithms, and orchestration prompts are not publicly disclosed.
Does Willow replace memory?
No. Willow is not raw chat memory or conversation replay. It is a runtime continuity/coherence layer designed to preserve the active frame and steer context more efficiently.
What happens when risk is high?
Willow can recommend verification, repair, or re-anchoring. The application decides how to handle the recommendation based on the workflow.
Can I bring my own evals?
Yes. Willow is best evaluated against real workflows where your current AI system loses context, drifts, or requires manual repair.
Evaluate Willow against your own continuity failures.
Developer Preview access is request-only while we expand validation and integration support.