The incident first.
We were editing a content file on one of our sites: a PHP array, a few hundred lines. Edited, uploaded straight to the server. One bracket count was wrong: an array block that needed four closing brackets had three. PHP is interpreted, so a syntax error surfaces only when the file loads. The whole site went white. Five hundred error, every page, not one page.
It was back two minutes later. Add the bracket, upload again, done.
Two minutes is not long, but every visitor in those two minutes saw a completely broken site. Anyone who had been about to get in touch was not coming back.
AI made the mistake; that is not where the fault lies
The bracket was AI’s, and there is no point pretending otherwise. But concluding “AI is unreliable” is the wrong reading and leads somewhere useless.
People drop brackets too, more often. The difference sits elsewhere: AI produces changes far faster, so the volume reaching production per hour is far larger. At the same error rate, the absolute number of errors goes up.
Stopping AI from making mistakes is an unreachable target, for people too. The question to ask is: how many things stand between a mistake and the damage it does.
That day the answer was none. The file went from an editor to production with no check in between. That is what caused the outage.
Guard one: validate before it lands
A deployment script went in the same day, and the logic is unremarkable:
Every file syncs to a temporary directory on the server first. Each PHP file is syntax-checked there. Only if all of them pass does anything overwrite the live plugin directory. If one fails, the temporary directory is deleted and production has not been touched by a single byte.
The tooling does not matter; every language ships a syntax check. The order matters: validate, then land. The original order was land, then roll back if something breaks, and rolling back assumes you already know something broke.
It costs a few seconds per deploy. In return that class of outage became impossible rather than unlikely. A file with a syntax error cannot reach the live directory at all.
Guard two: go and look afterwards
Syntactically valid code can be entirely wrong.
The same week: a script that generated code had a string operation written as “replace a trailing u together with the quote mark”, intended to strip a stray quote. What it did was truncate every name ending in the letter u.
That error is syntactically perfect. Nothing raised an exception. The generated file looked fine, unless you happened to inspect the few entries ending in u.
So the second half of the deployment script requests a dozen key pages after installing and checks the response codes, each page individually. Anything other than a 200 fails the deploy and prints the rollback command.
Later another check went in: after the static pages are generated, confirm the files for key pages exist. A missing page raises no error. It just quietly becomes slow or empty, which is the kind of thing that hides for months.
Guard three: make mistakes one step from undone
The first two intercept. The third accepts that something will get through.
Every change is in version control: templates, styles, scripts, content data. Any change can be traced to what, why and when. Reverting is one command.
Configuration files on the server get a timestamped copy before anything touches them. That rule looks old-fashioned until the file you are editing is shared by fifty websites, at which point it is the only reason you are willing to edit it at all.
What experience contributes
“Let AI write it and have a human review it” is correct and useless. Reading AI’s code line by line gives back the speed advantage, and human attention decays fast. By line three hundred, reading and not reading are close to the same thing.
What helps is concentrating on particular places:
Anything that happens in bulk gets spot-checked. Loops, batch replacements, regular expressions: this is where AI most often produces code that is mostly right. The truncated-names bug was exactly this. Reading will not catch it. Running three or four real values through and looking at the output will.
Anything that fails silently gets verified deliberately. A wrong SEO tag raises no error. An uncleared cache raises no error. A missing image or a missing page raises nothing. These have to be checked on purpose, because they will never come to you.
Anything shared gets backed up first. Break a file belonging to one site and you affect one site. Break a shared configuration and you affect all of them. The care taken should differ by an order of magnitude.
Anything irreversible gets a pause. Deletions, migrations, domain changes, database operations. AI performs these as readily as it changes a colour, and the consequences are not comparable.
What you are paying for
“We use AI” sells nothing on its own. Using AI is no barrier to anything; you could start this afternoon.
What clients are buying is this: changes are validated before they go live, something confirms afterwards that they work, problems are found before you notice them, and if something does go wrong, undoing it is one command rather than an incident.
None of it is visible on a normal day. You find out whether anyone built it for you on the day something breaks.
Our own site runs this way, and every guard exists because something went wrong first, including those two minutes. Every project we deliver runs on the same process.