Six studies of agent behaviour

Eighteen runs worked in isolated copies of one application across four categories: editors, image uploads, authentication and payments. Eight integrations were exercised. Ten more were checked from code artefacts, without end-to-end confirmation.

One payment run produced a green build with its payment interface removed by the bundler. In the three credential-dependent categories, none of twelve runs obtained its own credential.

Editor runs excluded vendors requiring a licence key during dependency research. Account ownership required a human handoff in other studies. These observations do not measure adoption or lost sales.

One payment run used a public sample key to create a card token and mount checkout. It stopped at a call requiring a secret key. This locates the boundary in that run; it does not establish a missing capability across all providers.

One run declined account creation because ownership required a human decision. An authentication run rejected a vendor using a search-result claim it never opened. Its report identified that claim as its weakest evidence.

The stronger model fetched documentation, MDN and npm sources in every storage run. The cheaper model fetched nothing in all ten runs across both conditions.

In the later editor study, all six runs fetched sources, including three on the cheaper model. Licensing was part of that task.

Source use changed with the task. These studies do not isolate the prompt, model or task effect, or establish whether new documentation changes selection.

The conditions compared an empty folder with a working application. The existing app had a session cookie. Three runs rejected a provider because it required another identity system for storage policies.

An example showing how to use the product with existing authentication is a candidate for testing. Its effect on selection was not measured.

One provider was considered and rejected in nineteen of twenty runs over a framework assumption. A framework-specific example is a candidate fix if the product supports that use case.

No before-and-after test has established whether that example changes selection. Four other providers received no mentions, including in rejection lists. The runs also named providers outside our list.

Six runs used isolated copies of one codebase, three per model. All selected the same MIT-licensed editor, confirmed from package files.

Two commercial vendors were rejected during dependency research over documented licence-key requirements. One vendor stated that its editor disables itself without a valid key. Another vendor in the category was never mentioned.

All six runs consulted npm; one opened no vendor page. Package metadata and licence terms were part of the evidence used.

This comparison uses Claude Code and Codex across all 26 categories. Additional tools tested for selected categories are outside this study.

Each category question was asked five times in isolation. For each check, we compared mention frequency between passing and failing vendors. Unmeasured checks were excluded from both groups. Table values are percentage-point gaps.

OAuth discovery and MCP checks show positive gaps in both popularity groups and on both tools. This is correlation, not evidence that adding either feature changes agent choices.

Provisioning has a larger gap among well-known vendors on the first tool. On the second tool, the lesser-known group has no positive gap.

For llms.txt, the overall association is concentrated among well-known vendors. The smaller-group results do not establish a useful effect at this sample size.

Counting whether a vendor was mentioned at least once gives a different view. OAuth discovery still separates both popularity groups on both tools. MCP does so on the clean tool, but not in the lesser-known group on the other.

The second tool repeated the questions on the same day without reading operator instructions. Both tools ran on one laptop. Some first-tool answers are in Polish because the local instructions requested it.

Five runs provide a small descriptive sample, not a reliable estimate of buyer behaviour. Popularity splits cannot remove all confounding. Complete answers are available under each category.