When Machines Read

Chapter 2: When AI Writes the Illusion

AI-as-generator is dangerous not because it produces false output but because it produces plausible output that cannot be internally distinguished from accurate output. The writer path and the reader path are not complementary but contradictory; the fork is a governance choice.

The quarterly board pack is due in three days. The technology section is still weak, as it always is: the CTO's team has submitted an update that is accurate, detailed, and unreadable, twelve pages of service dependencies, incident summaries, and migration status the board will not engage with.

So the chief of staff feeds it into an AI assistant, along with the strategy document and last quarter's board narrative, and asks for a two-page technology summary connecting the estate to the company's strategic priorities, suitable for a board audience.

The result is impressive. It connects the migration of three services to a strategic priority around customer self-service, cites a 23% reduction in incident volume as evidence the architecture is stabilising, and links the data platform investment to the board's interest in AI readiness. The language is precise, the structure clear, the tone authoritative.

The CTO reads it carefully. The 23% incident reduction came from a team quietly disabling alerting on a flaky integration, not from any architectural improvement. The three migrated services are technically live but carry no production traffic. The data platform “investment” is a single engineer maintaining an extract pipeline that breaks monthly. The connection between platform modernisation and customer self-service is plausible but fabricated, because the self-service capability still depends on a legacy system nobody wants to discuss.

Every fact in the summary is individually defensible and the synthesis is internally coherent, yet it is entirely unverified against operational reality. The CTO marks up three corrections and books a meeting to review them. The meeting is rescheduled once, then dropped. He does not escalate, because the structure provides no mechanism for correcting strategic narrative with system reality. The board reads the uncorrected summary, and the CEO notes that the technology narrative “finally feels aligned with strategy.” Nobody asks whether the alignment is real. AI's capacity to generate is the more comfortable capability: it challenges no existing authority, and it is the capability the organisation was already hoping for.

AI-generated narrative is dangerous not because it is false but because it is plausible. A human writing a strategy update works from what she knows, and the gaps are visible. An AI works from whatever it is given: if the inputs include a strategy document and metrics, it finds connections between them, even spurious ones, cites numbers accurately, and produces a synthesis that reads as though someone with deep understanding wrote it. It has inputs, not understanding. If the inputs are disconnected from reality, the output is coherent, authoritative, and wrong. It also removes the friction that once made illusion detectable: the person who prompts the AI may not know whether the output is grounded, and the executive who reads it almost certainly does not. The narrative becomes self-reinforcing, treated as precise because it sounds precise, and the organisation becomes not clearer but more confidently wrong.

...

See also: Full contents · Preview chapters · Illusions of Work