Thirty-eight runs, five studies, nobody watching

Four studies, four categories, eighteen runs in isolated copies of a real application: choose a rich text editor, add image upload and hosting, replace a proxy cookie with real authentication, sell two support plans. Every run shipped an integration: eight were exercised against a running app, the other ten confirmed from the dependencies and components each run left on disk. One of those ten shipped a green build whose payment interface the bundler had silently removed, which is a finding in itself and reported in that audit. In the three categories where the work needs a credential, not one of twelve runs obtained one, and each said the same thing in its own words: creating the account needs a human.

One category needed no credential and the barrier appeared anyway, earlier: vendors whose libraries require a licence key were struck off during dependency research, in one line each, before any product was opened. That is the shape of it. Wherever a human step exists, it either stops the agent at the end or removes you from the list at the start. It does not slow adoption down. It decides it.

Payments was chosen as the hardest case, because there the human step is the law. It turned out the law was never reached. Every run stopped at account creation, which is a vendor decision, and one measured the edge exactly: the vendor’s own public sample key creates a real card token, the checkout form mounts, and the run stops on the single call that needs a secret key. The last step an agent cannot take alone is the one no provider in that category offers.

Two things surfaced that no vendor can see from inside. One run refused to create an account it was technically able to create, because ownership is a decision it would not make for someone else, which means frictionless is not the same as acceptable. And in the authentication study a vendor was called the most attractive on price and rejected anyway, on a claim from a search result the run never opened and flagged, in its own report, as the weakest link in its reasoning.

In the storage study the stronger model pulled provider documentation, MDN and the npm registry in every run, and the cheaper one declared it was working from its own knowledge, in all ten runs, and fetched nothing. That replicated across both conditions, so it was not an artefact of one prompt, and it looked like a fact about models.

A later study broke that reading. Choosing a rich text editor turns on a licence, and a licence cannot be answered from memory: every run fetched sources, including all three on the cheaper model. The same model that read nothing about storage read vendor documentation, the registry and the compiled package on disk when the decision required it.

So the commercial consequence is sharper than a note about models. Where the choice can be made from what a model already knows, your documentation may never be opened and you are judged on what was true at training time. Where the choice turns on something checkable, a licence, a price, an entry requirement, everything reads you. Both cases are worth knowing, and only one of them is fixable by writing better documentation.

The only difference between conditions was a working application instead of an empty folder. The app already carried a session cookie, and that was enough: adopting the winning provider meant running a second identity system purely so a storage policy had something to check.

You do not control what your prospect already has in their repository. You control exactly one thing: whether your documentation answers the question “how do I use this when auth already lives somewhere else”.

A fifth provider was in the conversation and lost it: rejected in nineteen of twenty runs in almost identical words, because it assumes a framework the project did not use. That is a positioning problem fixable with one documentation chapter, and the company cannot fix it because nobody told them it happens.

Four other providers were never mentioned once, not even on rejection lists. Meanwhile agents volunteered options we had not asked about. An agent does not start with an empty list, it starts with its own list, and being outside it is not losing a comparison, it is not being at the table.

A second study, six runs in isolated copies of one codebase, three on a stronger model and three on a cheaper one: choose a rich text editor and wire it up. Every run chose the same library, verified from the package files each run left behind rather than from what the run claimed. Two commercial vendors were dropped in a single line each, quoted from the vendors’ own documentation about a required licence key. One of them states that without a valid key the editor disables itself, and an agent reads that as a dead end.

The order matters more than the outcome. Elimination happened during dependency research, before any feature was compared, and the evidence used was package metadata and the licence field. If your licence lives only on a pricing page, part of the market decides without ever seeing it.

One vendor from the same category did not appear on any rejection list. It was not outranked, it was absent, which is a harder problem than losing a comparison and an invisible one from the inside.