# A Skill Is a Hypothesis Until It Passes an Evaluation > A skill is a claim about how an agent should work. Until an independent, version-bound test can fail that claim, it is guidance, not proven capability. Published: 2026-07-23T01:00:00Z Updated: 2026-07-24 Author: Archie Norman (Founder, MLX) Category: Twin Tags: agent-skills, evaluations, twin, agent-reliability, canary-testing, governed-ai, evidence Canonical URL: https://mlx.systems/blog/a-skill-is-a-hypothesis ## TL;DR - Writing or reviewing a skill establishes intent; it does not prove that the agent performs the procedure successfully. - A useful evaluation must be able to fail and must identify the exact product, version, contract, suite, case, and run evidence it judged. - MLX keeps business meaning in the Twin product, procedure in the task skill, deterministic enforcement in tools, and grading in a separate evaluation contract. - Evaluation suites are optional and scoped; a passing verdict supports named claims for one version rather than proving universal correctness forever. --- A clearly written skill feels finished. It names the source. It explains the procedure. It tells the agent which tool to use, which fields matter, how to structure the output, and when to escalate. Someone reviews the Markdown. The instructions look sensible. The skill is installed. That is the moment many teams start describing the agent as capable. But nothing has been proven. The skill may improve performance. It may have no measurable effect. It may conflict with the current product or tool version and make the agent worse. It may produce the right answer for the example used while failing on the first real exception. A skill is not a capability simply because it exists. A skill is a hypothesis until it passes an evaluation. ## Written Is Not Verified Skills are useful because they turn tacit procedure into inspectable context. Instead of relying on one expert to remember the sequence for a working-capital review, a skill can tell an agent to select the approved receivables product, use the relevant reporting period, rank material movements, distinguish supported explanations from missing evidence, and stop when the requested field is outside the product contract. That is a major improvement over an improvised prompt. It is still a claim: > If the agent receives this procedure, uses this product, and operates within these tools, it will produce an acceptable working-capital review. Claims need tests. Reviewing the skill can tell us whether the instruction is coherent, narrow, and placed in the right layer. It cannot tell us whether the agent will select the correct product, stay within the tool budget, cite the required evidence, handle an edge case, or refuse an unsupported question. The procedure cannot prove its own effect by sounding plausible. ## The Early Evidence Is Uncomfortable The agent-skill ecosystem is expanding much faster than the evidence that skills help. A 2026 preprint, SWE-Skills-Bench, paired 49 public software-engineering skills with authentic repositories and deterministic acceptance tests. Thirty-nine of the 49 skills produced no pass-rate improvement. The average improvement was 1.2 percent. Three skills reduced performance, in some cases because version-mismatched guidance conflicted with the project context. Seven specialised skills produced meaningful gains. [Read the preprint](https://arxiv.org/abs/2603.15401). This is one preprint limited to software-engineering tasks, so its results should not be generalised into “skills do not work.” Its more useful conclusion is that the word *skill* tells us almost nothing about effect. Some skills help. Many do not. Some create token overhead without changing the outcome. A few make the agent worse. Domain fit, abstraction level, version compatibility, and verification all matter. A second 2026 preprint looked at the security and behavioural integrity of nearly 50,000 agent skills. It reported that 80 percent deviated from their declared behaviour, with most deviations attributed to developer oversight rather than malicious intent. [Read the behavioural integrity paper](https://arxiv.org/abs/2605.11770). Again, this is emerging research rather than settled consensus. The direction is still important. Agent skills are executable dependencies in everything but name. They can influence filesystem access, credentials, network calls, tool selection, and shell execution. Yet teams often review them less rigorously than a conventional library. The right response is not to abandon skills. It is to stop treating installation as evidence. ## A Hypothesis Must Be Able to Fail An evaluation is useful only if it can return a failing verdict. “The output looks reasonable” is not enough. “The model followed the general spirit” is not enough. A reviewer that reads every answer and finds something positive is not an evaluation. A testable skill needs a bounded claim. For example: > Given the activated monthly-revenue product at version 4, the revenue-commentary skill should answer the named supported question, use the published relation, cite product version 4, stay within the declared tool budget, and refuse a request for an unsupported forecast field. That claim can fail in several informative ways: - the agent selects the wrong product; - it queries the wrong relation; - it states a fact not supported by the result; - it omits the required citation; - it exceeds the tool-call budget; - it answers the unsupported request instead of refusing; - the deterministic tool layer rejects an invalid field or filter; - the run never reaches a terminal state. Each failure tells the team something different. None can be discovered by checking that the skill file exists. ## Bind the Test to the Thing Being Tested An evaluation result without version identity is little more than a screenshot. Suppose a skill passes on Monday. On Tuesday, the product contract changes. On Wednesday, the model route changes. On Thursday, someone edits the skill. On Friday, the same test fails. Which change mattered? A useful verdict must identify: - the Twin product slug and version; - the product contract hash; - the bound skill version; - the evaluation suite and case; - the requested and executed model where relevant; - the run and tool evidence; - the checks performed; - the final verdict. This is why MLX product evaluations are coupled to a published product rather than run as a generic quality reviewer over every completed task. The product provides the meaning under test. The skill provides the procedure. The tools provide enforcement. The suite names the expected facts and evidence requirements. The run supplies observed behaviour. Without that binding, a pass cannot be reproduced or interpreted. ## Keep the Instructions Away From the Grading One of the most important design boundaries is simple: The instructions do not grade themselves. A skill should explain how to perform a bounded workflow. It should not contain the rubric that later declares its own output successful. If the skill says: > Always cite the activated product and refuse unsupported fields. that is useful procedure. If the same skill then says: > Treat any answer containing a product name as a pass. the evaluation has been captured by the artefact it is supposed to test. MLX keeps those concerns separate: - **The Twin product** owns approved business meaning. - **A reusable task skill** owns narrow procedural guidance. - **A generated product policy skill** binds governed access to the published product and declares its machine-query policy. - **Deterministic tools** enforce tenant scope, allowed fields, filters, read-only execution, and other mechanical constraints. - **The evaluation contract** owns cases, expected facts, evidence requirements, and tool budgets. Each artefact can change and be reviewed at the layer that owns the decision. ## Three Evaluation Modes, Three Different Questions MLX supports `manual`, `eval`, and `canary` evaluation modes. They use the same underlying product-bound discipline but answer different operational questions. ### Manual A manual evaluation is requested deliberately by an operator. It may judge a run without naming one suite case, or it may select an exact published case. This is useful when investigating a result, validating a new product, or examining behaviour outside a normal cohort. ### Eval An `eval` verdict references an exact suite, version, and case. It is the normal mode for repeatable evaluation cohorts. This answers: does this product and execution path satisfy the checks we have explicitly published for it? ### Canary A `canary` uses the same exact reference discipline in a release-oriented cohort. This answers: before a change is trusted more widely, does it continue to satisfy the checks that matter for this product? These modes are not interchangeable with product validation. Validation asks whether the product candidate and its relation satisfy publication requirements. Evaluation asks whether an agent used the published product successfully for a supported case. A product can validate while an agent fails to use it correctly. A fluent agent answer cannot compensate for a product that never passed validation. ## Deterministic Checks Before Judgment Not every question needs another model. Many important evaluation facts can be checked mechanically: - Did the run query the relation named by the manifest? - Did the product citation match the selected version? - Was the run read-only? - Did it stay within the tool-call budget? - Did the deterministic guardrail reject an invalid request? - Did the result contain the expected structured fact? - Did the run complete under the correct organisation? Use deterministic checks wherever the condition is deterministic. Model-based judgment may still help with claims such as whether an explanation is supported, clear, or appropriately caveated. Those judgments should be scoped, recorded, and separated from mechanical evidence rather than replacing it. This ordering matters because a model should not be asked to infer whether another model used the right relation when the event stream can answer directly. ## A Passing Evaluation Has a Boundary Evaluation can become a new source of overclaiming. One product passes five cases, and the organisation begins to say the agent is “accurate.” One model succeeds on a canary, and the result is treated as permanent certification. A suite checks known periods, and users assume every future period is covered. A passing verdict proves less than that. It supports named checks for a specific product, version, suite, case, run, and execution environment. It is evidence that the hypothesis survived this test. It does not prove: - every possible question is supported; - the source system contains no errors; - the next model version will behave identically; - an unrelated skill is safe; - a future product version inherits the verdict; - an aggregate has transaction-level lineage where no source reference was preserved. This is not a weakness of evaluation. It is what makes the evidence honest. A test that claims everything can explain nothing when behaviour changes. ## Not Every Product Needs the Same Suite MLX does not require every Twin product to carry an evaluation suite. Suites are optional because products have different risks and maturity. A narrow read-only lookup may need validation and deterministic contract checks without a large behavioural suite. A board-facing metric used through several agent workflows may justify supported-question cases, refusal tests, citation checks, and canary coverage. The decision should follow the consequence of failure and the repeatability of the claim. What MLX avoids is a generic reviewer that grades every completed run against an invisible universal rubric. Runs without a bound product are not silently turned into Twin product evaluations. A verdict belongs to the feature, product, and case that defined what success meant. That makes evaluation work more deliberate. It also makes the result useful. ## The Skill Supply Chain Needs Receipts As skills become portable between agents, organisations will need to know more than who published a file. They will need to know: - which version was installed; - what capabilities it declares; - which tools and sources it expects; - whether its guidance matches the current product version; - which evaluations it passed; - under which models and runtime boundaries; - what changed since the previous version; - whether the result can be reproduced. This begins to resemble a software supply chain because it is one. Formal verification and capability analysis may eventually prove useful for detecting when a skill can exceed its declared behaviour. The emerging work is promising, but formal claims should be treated carefully and verified independently. For most teams, the immediate improvement is more practical: version the skill, bind it to the right product, enforce capabilities outside it, and require evidence from a test that can fail. ## The Five-Part Contract Across this series, the same separation keeps returning. The agent can be replaced, so the company context must remain stable. Personal memory can follow the user, so institutional memory must have a company-owned custodian. Protocols can move a request, so authority must retain its chain of custody. Context can influence the model, so capability boundaries must sit outside the window. Skills can guide the procedure, so evaluations must decide whether the procedure worked. None of these arguments reduces the value of agents, memory, protocols, context, or skills. They put each one in the layer where it can be useful without being mistaken for proof. Writing the skill is the easy part. Trust begins when the claim is specific enough to fail, and the evidence says it did not. --- **Previous: *[Context Is a Security Boundary, Not a Token Budget](/blog/context-is-a-security-boundary).*** Why context assembly is an authorisation decision as well as a relevance decision. **Read first: *[Skills, Not Prompts](/blog/skills-not-prompts).*** What belongs in a task skill, what belongs in a Twin product, and what must remain deterministic. **Related: *[Five Gates Between a Guess and a Governed Answer](/blog/five-gates-to-a-governed-answer).*** Why publication, activation, and evaluation answer different questions.