The Punk PM #49

Hey there, punk!

There have been some agent horror stories in the tech press lately. Teams watching their systems go sideways because an AI decided to do exactly what it was told, just not quite in the way anyone intended.

My take: that's not an AI problem. It's a product problem.

We're treating agents as something tech owns and deploys. But if you build and ship an agent that interacts with real systems and real data, you've built a product. And products need someone thinking about risk, scope, and what happens when things go wrong.

Let's get into it.

Quote of the Week 🙊

Insight 🦉

There's a story blowing up at the moment about PocketOS. An AI agent deleted their production database (and almost killed the company) during what was supposed to be a routine task on a staging server.

The founder blamed Anthropic. He blamed Railway.

He didn't seem to spend much time asking how an agent on a staging environment had the permissions to reach production in the first place.

That's not just a technical question. It's a product question.

Someone decided what that agent could do. Someone defined its scope (or didn't). Someone made a call—consciously or not—about how much autonomy was appropriate, and what the blast radius would be if it went wrong. Those are product decisions. And most teams aren't treating them that way.

Anyone who's been in tech long enough has a human error story that rhymes with this. The junior dev who dropped the wrong table. The analyst who fat-fingered a billing system and sent someone an electricity bill for £20m (that one was me—it made the UK tabloids). These things have always happened. But they used to happen at human speed, with human cognitive limits, in environments where the damage was usually containable.

Agents are different. They don't get tired. They keep executing until the job is done, or until there's nothing left. The potential impact of a mistake is several orders of magnitude bigger. The product thinking that goes into deploying them needs to match that.

Here's what gets missed in most of the discourse: agents don't have judgement. It just feels like they do. The fluency is convincing. But there's no moment where an agent pauses and thinks, this seems riskier than the brief implied. It executes. Making sure it doesn't execute the wrong thing—that's still someone's job.

The best mental model for this is the new junior hire. You wouldn't hand them production access on day one. You'd start small, watch their work, expand their remit as they earned it. You'd also make sure the environment was set up so that an inevitable mistake couldn't take down the whole system.

Agents deserve the same amount of management. Probably more.

PocketOS isn't a story about AI going rogue. It's about a team that hadn't done the product work of deploying an agent responsibly. Product management, at its core, is about risk mitigation. At PocketOS, it looks like no one was thinking about risk at all.

Action 🚀

Next week, pick one agent or AI-powered workflow your team is currently running and ask: who owns this? Who defined its scope? What happens if it goes wrong, and do we have a plan for that?

If you can't answer those questions quickly, that's your starting point.

Inspiration 💡

I Managed a Swarm of 20 AI Agents for a Week and Built a Product – Zach Wills ran 20 agents in parallel and came out the other side with eight rules worth knowing. The biggest lesson: this is management, not automation. Clear instructions, active oversight, and constant course-correction. Sound familiar? Read more

10-Minute AI System Readiness Check – Judy Ossello makes the case that building AI is the easy part — defining its scope and responsibilities is where most teams fall short. A quick diagnostic for finding where accountability gets blurry before it becomes a problem. Read more

I Didn't Expect an AI Agent to Feel This Unnerving – Craig Hepburn gave an agent access to his daily work and communications, then stopped using it. Not because it failed — because it acted with judgement he couldn't see or verify. A useful gut-check on where trust, responsibility, and autonomy need clearer lines. Read more

Signing Off ✍️

If this resonates with you, hit reply and let me know. And if you think a friend or colleague would enjoy The Punk PM, feel free to share it with them!

Play it your way,

Toby