The incident first.
We were editing a content file on one of our sites — a PHP array, a few hundred lines. Edited, uploaded straight to the server. One bracket count was wrong: an array block that needed four closing brackets had three. PHP is interpreted, so a syntax error surfaces only when the file loads. The whole site went white. Five hundred error, every page, not one page.
It was back two minutes later. Add the bracket, upload again, done.
Two minutes is not long, but every visitor in those two minutes saw a completely broken site. Anyone who had been about to get in touch was not coming back.
AI made the mistake; that is not where the fault lies
The bracket was AI’s, and there is no point pretending otherwise. But concluding “AI is unreliable” is the wrong reading and leads somewhere useless.
People drop brackets too, more often. The real difference sits elsewhere: AI produces changes far faster, so the volume reaching production per hour is far larger. At the same error rate, the absolute number of errors goes up.
So the question is not how to stop AI making mistakes — that target is unreachable, for people too. The question is: how many things stand between a mistake and the damage it does.
That day the answer was none. The file went from an editor to production with no check in between. That, not the bracket, is what caused the outage.
Guard one: validate before it lands
A deployment script went in the same day, and the logic is unremarkable:
Every file syncs to a temporary directory on the server first. Each PHP file is syntax-checked there. Only if all of them pass does anything overwrite the live plugin directory. If one fails, the temporary directory is deleted and production has not been touched by a single byte.
The tooling is not the point — every language ships a syntax check. The order is the point: validate, then land. The original order was land, then roll back if something breaks, and rolling back assumes you already know something broke.
It costs a few seconds per deploy. In return that class of outage became impossible rather than unlikely — a file with a syntax error cannot reach the live directory at all.
Guard two: go and look afterwards
Syntactically valid code can be entirely wrong.
The same week: a script that generated code had a string operation written as “replace a trailing u together with the quote mark”, intended to strip a stray quote. What it actually did was truncate every name ending in the letter u.
That error is syntactically perfect. Nothing raised an exception. The generated file looked fine — unless you happened to inspect the few entries ending in u.
So the second half of the deployment script requests a dozen key pages after installing and checks the response codes. Not “does the site load” but each page individually. Anything other than a 200 fails the deploy and prints the rollback command.
Later another check went in: after the static pages are generated, confirm the files for key pages actually exist. A missing page raises no error — it just quietly becomes slow or empty, which is the kind of thing that hides for months.
Guard three: make mistakes one step from undone
The first two intercept. The third accepts that something will get through.
Every change is in version control — templates, styles, scripts, content data. Any change can be traced to what, why and when. Reverting is one command.
Configuration files on the server get a timestamped copy before anything touches them. That rule looks old-fashioned until the file you are editing is shared by fifty websites, at which point it is the only reason you are willing to edit it at all.
What experience actually contributes
“Let AI write it and have a human review it” is correct and useless. Reading AI’s code line by line gives back the speed advantage, and human attention decays fast — by line three hundred, reading and not reading are close to the same thing.
What helps is concentrating on particular places:
Anything that happens in bulk gets spot-checked. Loops, batch replacements, regular expressions — this is where AI most often produces code that is mostly right. The truncated-names bug was exactly this. You do not catch it by reading; you catch it by running three or four real values through and looking at the output.
Anything that fails silently gets verified deliberately. A wrong SEO tag raises no error. An uncleared cache raises no error. A missing image, a missing page — none of them raise anything. These have to be checked on purpose, because they will never come to you.
Anything shared gets backed up first. Break a file belonging to one site and you affect one site. Break a shared configuration and you affect all of them. The care taken should differ by an order of magnitude.
Anything irreversible gets a pause. Deletions, migrations, domain changes, database operations. AI performs these as readily as it changes a colour, and the consequences are not comparable.
Why any of this is worth paying for
Clients are not buying “we use AI”. Using AI is not a barrier to anything; you could start this afternoon.
What they are buying is this: changes are validated before they go live, something confirms afterwards that they actually work, problems are found before you notice them, and if something does go wrong, undoing it is one command rather than an incident.
None of it is visible on a normal day. You find out whether anyone built it for you on the day something breaks.
Our own site runs this way, and every guard exists because something went wrong first — including those two minutes. Every project we deliver runs on the same process.