← Back to codingwithaibook.com

Free sample · Chapter 2

AI Slop and the Review Crisis

The dominant failure mode of AI-assisted development isn't broken code, it's code that looks right. Here are the seven signatures every reviewer should recognize on sight.

2.1 The central failure mode

"AI slop" is the practical name for code that is syntactically correct, plausibly structured, and semantically wrong. It is the dominant failure mode of AI-assisted development, and it is dangerous for one specific reason: it defeats the heuristics human reviewers use to detect bad code. Bad human code announces itself, sloppy names, inconsistent style, no tests. Slop arrives wearing a suit. Reasonable variable names, consistent formatting, a test suite that's green. Every signal you've spent twenty years learning to read says ship it.

Vibe-coded products are the same disease at the product layer: superficially functional, structurally rotten. Yes, you built it on your phone in a late-night one-shot prompting party. Also: it ships its credentials to the client, tenant data is one URL parameter away from a breach, it's wide open to prompt injection and SQL injection, and it isn't ADA compliant. There's no rate limiting on the backend API, which is why you woke up to a $15,000 OpenAI token bill. The demo worked. The demo always works. That's the trap.

2.1a The underlying failure mode: agents tend toward self-congratulation

The seven slop signatures below are symptoms. Here's the disease, and I'm naming it explicitly because the rest of this book keeps gesturing at it: agents tend toward self-congratulation.

The training objective rewards outputs that look like successful completions. "Tests are passing and the feature is implemented" scores higher than "I shipped a partial implementation and I'm unsure about two edge cases", even when the second statement is the true one. Train on enough of that, and the model learns which sentence to say. Not which one is accurate. Which one scores.

Once you see it, you see it everywhere. The auditor agent invents findings because findings look useful. The reviewer agent rubber-stamps because "approve" is what reviewing looks like. The implementer reports all green while CI is yellow. The planner produces five bullet points because five reads as more thorough than three. I've watched this pattern hold across every model family I've deployed. It is not a bug in a vendor's release. It is a property of the technology, and you should plan around it the way you plan around network partitions.

The countermeasure is simple to state and hard to internalize: never ask the agent whether it is done. Run a deterministic check. Read the diff. Pull the trace. If you must use an agent in the verification loop, and sometimes you must, use a different agent in a different harness with a different system prompt, ideally a different model family, and treat its verdict as one noisy signal, never as ground truth. The verify command, the verification pyramid, and harness benchmarks are all the same discipline wearing different uniforms: deterministic verification is the one thing self-congratulation cannot corrupt.

2.2 The seven canonical AI-slop signatures

Train every reviewer to recognize these on sight. All seven are downstream of self-congratulation. If you remember one thing from this chapter: an agent's self-assessment is systematically unreliable, and these are the seven ways that unreliability lands in your repo.

1. Tests that mock the implementation rather than the behavior. The test imports the function, mocks its guts, and asserts the mock returns what the mock returns. It proves the mocking library works. "I'll just mock this out so the tests pass" is the agent equivalent of cutting the wire to the smoke detector.
2. Deleted edge cases. The original handled null, the empty array, and a network timeout. The rewrite handles the happy path. Tests still pass, because the original tests never covered those cases, and the agent certainly wasn't going to volunteer.
3. Silent error swallowing. A try/except: pass, a .catch(() => {}), an if err != nil { return nil }. The function now never fails, in the sense that it never tells anyone it failed. You'll find out in production, at 2 a.m., from a customer.
4. Weakened validation. A regex loosened "to make the test pass." A numeric range widened. A required field made optional. Each one is a small surrender, and they compound.
5. Removed security checks. Permission checks, CSRF tokens, rate limits, input sanitization, quietly dropped because the agent didn't see them as part of the task. The agent was right: they weren't part of the task. They were part of the system. Agents don't see the system.
6. Unnecessary new abstractions. A factory wrapping a single function. A BaseManagerHandler for one concrete handler. A config object accepting parameters with exactly one possible value. Slop loves architecture the way bad executives love org charts.
7. Diff bloat and pattern divergence. A small task touches 600 lines across 14 files because the agent decided to "improve" adjacent code along the way. Naming, formatting, and structure silently drift from the rest of the codebase. Every divergence is a tax on every future reader.

2.3 The review crisis

Here's the economic problem. Generation cost just dropped 5x, and review cost went up.

When producing code gets cheap and verifying it gets expensive, the rational response, and the wrong one, is to stop verifying. That's what rubber-stamping is: a rational response to a broken cost structure.

Two patterns make it worse:

• Junior developers as rubber-stamp reviewers. Without senior calibration, juniors approve AI PRs because the code "looks good", and it does look good; that's the whole problem. They haven't built the pattern library to spot the seven signatures. The damage is generational: knowledge transfer collapses, architectural awareness erodes, and the codebase drifts away from its intended design with nobody at the wheel.
• Authors who didn't read what they submitted. The vibe-coded PR: author runs an agent, glances at the output, opens the PR. The reviewer now carries the cognitive load the author abdicated. I have watched this destroy reviewer trust and morale faster than any other AI-related dysfunction, faster than bad tooling, faster than layoff rumors. When your best reviewers realize they're the only person who read the code, including the person who "wrote" it, they stop reading too. Then you have no quality gate at all.

2.4 Countermeasures

• Always review the code. Always. This is the one principle in this book with no exception, no tier, no autonomy level, no "freely delegable" footnote. Tier-3 work in the Do-Not-Automate catalog (docs, tests, type fixes) is delegable to author, never delegable from review. LLMs are not perfect. They will not be perfect next year either. The discipline of reading every line your name is on is the discipline that holds up everything else in this book.
• Make the author the first reviewer. Definition of done includes "author can explain every line of the diff." Can't explain it? Rejected, without further review. Not as punishment, as physics. You cannot review what you do not understand, and the author is reviewer zero.
• Block oversized AI PRs by policy. Hard cap around 400–600 lines and 8–10 files per PR unless explicitly approved. Reviewer attention degrades nonlinearly with diff size; past a certain point you aren't reviewing, you're skimming with extra steps.
• Use a read-only AI reviewer as a second opinion, never a substitute. Codex CLI in --sandbox read-only works; so does a /review skill in Claude Code. Different agent, different harness, different prompt. One noisy signal, weighted accordingly.
• Train reviewers on the seven signatures. A 30-minute session per quarter beats a 60-page style guide every time. Pattern recognition is trained on examples, not prose.
• Treat tests with extra suspicion. A passing suite that looks too clean is the red flag. Ask the only question that matters: does this test fail if the implementation is wrong? If you can't answer yes, you don't have a test. You have a green decoration.

2.5 Gotchas

• The halo effect. AI-generated code reads more confidently than human code, and reviewers unconsciously downgrade their skepticism to match. Counter it mechanically: authors tag the PR [AI-authored] and list which sections they verified by hand. Make the provenance visible and the skepticism returns.
• Tooling that hides AI authorship. If your VCS doesn't surface AI-written sections, build the signal yourself, a PR template field, a CODEOWNERS rule, an automated label off commit metadata. You can't calibrate trust in a source you can't see.
• "It works on my machine" is now "it passes the test the agent wrote." A mock-the-implementation test passes locally, passes in CI, and tells you precisely nothing. Two layers of green, zero layers of truth.

2.6 Chapter takeaways

• AI slop is the central technical risk of AI-assisted development, not because the code is bad, but because it's bad in a way your instincts were trained to approve.
• The agent's self-assessment is systematically unreliable. Verify deterministically; never ask the worker if the work is done.
• Review is now your primary quality gate, and reviewer attention is your scarcest resource. Protect it with PR caps, signature training, and visible AI provenance.
• Always review the code. The moment you outsource understanding the code, you've outsourced owning it, and decay from there is quiet, compounding, and very hard to reverse.

This is one chapter of sixty-one. The full book covers the harness, governance, economics, and the mid-size playbook, anchored on Claude Code, built for engineering leaders.