When Machines Read

Chapter 2: When AI Writes the Illusion

AI-as-generator is dangerous not because it produces false output but because it produces plausible output that cannot be internally distinguished from accurate output. The writer path and the reader path are not complementary but contradictory; the fork is a governance choice.

The quarterly board pack is due in three days. The strategy team has been preparing it for two weeks, but the technology section is still weak. The CTO's team has submitted their update: accurate, detailed, and unreadable. Twelve pages of service dependencies, incident summaries, and migration status. The board will not engage with it.

So the chief of staff feeds the CTO's update into an AI assistant, along with the company's strategy document and last quarter's board narrative. The prompt is simple: produce a two-page technology summary that connects the current state of the technology estate to the company's strategic priorities, suitable for a board audience.

The AI-generated summary is impressive. The summary connects the migration of three services to a strategic priority around customer self-service. It cites a 23% reduction in incident volume as evidence that the architecture is stabilising. It links the data platform investment to the board's interest in AI readiness. The language is precise, the structure clear, the tone authoritative.

The CTO reads it carefully.

The 23% incident reduction came from a team quietly disabling alerting on a flaky integration, not from any architectural improvement. The three migrated services are technically live but carry no production traffic; they were proofs of concept that were never promoted. The data platform “investment” is essentially a single engineer maintaining an extract pipeline that breaks monthly. The connection between platform modernisation and customer self-service is plausible but fabricated, because the self-service capability still depends on a legacy system that nobody wants to discuss.

While every fact in the summary is individually defensible and the synthesis remains internally coherent, it is entirely unverified against operational reality.

...

See also: Full contents · Preview chapters · Illusions of Work