Downwind
← The Bump Log

Claim Gating: Why I Forbid AI From Saying "You Worked Harder"

By Matt Sullivan · July 23, 2026

Every fitness app celebrates your workout and tells you that you worked harder. It is the most natural sentence in the genre: you did the workout, so you must have pushed. My AI coaching layer is mechanically prevented from generating it. Not discouraged by a prompt. Prevented, by a validation function, with a test suite.

This is the story of why, including the part where a key metric failed an audit against ground truth. I am writing it up because the pattern generalizes far beyond my paddling app: it is the same problem anyone deploying an AI agent that speaks to their customers has to solve, and many teams are currently solving it with prompt instructions and hope.

The physics problem underneath the language problem

I build analysis software for downwind ocean paddling. A paddler (me) rides wind swell in an outrigger canoe or surfski; the core skill is linking rides, and the core question after a session is whether a fast run was skill, conditions or luck. The app ingests GPS speed traces and deterministically calculates: time and distance on bumps, dead water, ride chains, session efficiency and more.

Here is the learning that shaped everything: GPS speed tells you what happened. It cannot tell you why. A speed increase can come from the paddler's effort or from the ocean carrying the boat, and a GPS trace alone cannot distinguish those two causes. So any sentence of the form "you worked harder," "you wasted energy," "you were pushing through dead water" is, on speed-only data, speculation dressed as coaching.

The funny thing is that we speak these sentences to ourselves when we review the data. This is our brain doing exactly what it's built to do: construct the most coherent causal story it can conjure only from the data in front of it, actively ignoring any signal it cannot see. "I went faster" is instantly internalized as "I worked harder."

Most products are designed to generate those sentences because users love positive feedback, and language models produce it fluently. I decided the Downwind coach would not make false claims, and then, more importantly, I decided that "would not" had to be a property of the system rather than a hope about the model.

The claim taxonomy

Every statement the coaching layer generates is internally classified into one of four claim classes:

Before any text is generated, the model is told which evidence exists, as boolean flags: has_speed_data, has_heart_rate_data, has_stroke_evidence, stroke_evidence_quality. The rule set is short:

The audit that made this real

The framework was forced by an embarrassing finding.

An early version of the pipeline relied only on GPS data and labeled certain GPS signatures "effort bumps": short speed wiggles interpreted as the paddler working. Once I started detecting strokes from watch IMU data, I stumbled into checking the interpretation against actual effort: strokes. The GPS-based "effort bump" detection was accurate for only 28% of events.

The label was the misnomer. The system had been detecting sub-threshold speed noise and calling it human effort because I labeled the signature "effort bump." The coding agent locked that name into code, and carried the semantic association forward into the LLM coaching prompt. Users saw "effort bumps" in reports and were coached on how they expended their effort to catch and link these bumps. It was all error, no signal.

I retired the term. The deprecation is documented, with the 28% figure and the reason, in the shipped vocabulary rules, not in a private postmortem. Coding agents often refer to this lesson in principle as part of planning and verifying work. Users see effort attribution only when the sensor evidence for it exists.

I want to be precise about what that finding means and what it does not. It means the app's inference failed validation, and I did not know until I had audited that against an independent signal. It does not mean GPS-based inference is worthless, just misused. The general lesson I take from it: any AI product making causal claims from a single signal source should assume its causal labels are wrong until checked against an independent one.

Enforcement lives in code, not in the prompt

The part of this system I would defend most strongly: enforcement is not prompt instructions. The prompt does state the rules, but prompts are requests, not governance. A governance layer built on requests fails silently under distribution shift, model updates, or an unlucky sample; regardless of what evaluations are in place.

So the pipeline enforces the rules mechanically:

One more example of the pattern, because it shows the gate is about evidence quality generally, not just causation. Many Garmin watches default to "smart recording," which writes a GPS datapoint every four to seven seconds. Bump detection still runs on that data, but it undercounts, so the rules forbid the model from making bump-count, dead-water, or efficiency claims on any sparse recordings. The report instead states that the session was recorded in a low-rate GPS mode and tells the user how to enable one-second recording. The system is built to admit its input is inadequate rather than produce a falsely confident analysis of it.

The business value

There is an honest business reason this architecture survived contact with a solo founder's time constraints: the claim gate is also the paywall. GPS-only users get outcomes and patterns free. The wrist sensor unlocks more valuable causal coaching, because the wrist sensor is what makes causal coaching true. The upgrade is: you are not paying to remove an arbitrary feature flag, you are paying for the evidence that permits a stronger class of claim. Epistemics and monetization point the same direction, which enables a defensible pricing strategy.

Why this matters beyond paddling

Replace "paddler" with "customer" and "effort" with any fact your AI agent might assert, and this is the agentic deployment problem I spend my consulting time on. When an AI system speaks for your company, the questions are identical:

My experience across a small consumer product and much larger enterprise agentic systems is consistent on one point: the teams that treat "what is the AI allowed to say" as an architecture decision ship systems they can defend, and the teams that treat it as a prompt-writing task inadvertently find out what their system really says from their users.

The single sentence I would put at the top of any such system's rules is the one at the top of ours: describe outcomes freely, make causal claims only when evidence is present, and prefer silence to speculation.

I build Downwind, an analysis and coaching tool for downwind ocean paddling, and consult on agentic AI deployment for mid-market companies through Outside Product. The claim-rules document, claim taxonomy, and the 28% audit described here are from the shipped system.