When Machines Read

Chapter 2: When AI Writes the Illusion

AI-as-generator is dangerous not because it produces false output but because it produces plausible output that cannot be internally distinguished from accurate output. The writer path and the reader path are not complementary but contradictory; the fork is a governance choice.

The quarterly board pack is due in three days. The strategy team has been preparing it for two weeks, but the technology section is still weak. The CTO's team has submitted their update: accurate, detailed, and unreadable. Twelve pages of service dependencies, incident summaries, and migration status. The board will not engage with it.

So the chief of staff feeds the CTO's update into an AI assistant, along with the company's strategy document and last quarter's board narrative. The prompt is simple: produce a two-page technology summary that connects the current state of the technology estate to the company's strategic priorities, suitable for a board audience.

The result is impressive. The summary connects the migration of three services to a strategic priority around customer self-service. It cites a 23% reduction in incident volume as evidence that the architecture is stabilising. It links the data platform investment to the board's interest in AI readiness. The language is precise. The structure is clear. The tone is authoritative.

The CTO reads it carefully.

While the 23% incident reduction was merely caused by a team quietly disabling alerting on a flaky integration rather than any actual architectural improvement, the three migrated services are technically live but carry no production traffic, having been proofs of concept that were never promoted. Furthermore, the data platform “investment” is essentially a single engineer maintaining an extract pipeline that breaks monthly. Ultimately, the connection between platform modernisation and customer self-service is plausible but fabricated, as the self-service capability actually depends on a legacy system that was omitted because nobody wants to discuss it.

While every fact in the summary is individually defensible and the synthesis remains internally coherent, it is entirely unverified against external reality.

...

See also: Full contents · Preview chapters · Illusions of Work