What stuck after two years of AI on my team
A live video product, a team of engineers, two years of folding AI into how we work, and the practices that survived contact with production.
We had forty feature flags to move off one provider and onto another. Tedious work, the kind that eats a sprint and teaches nobody anything. I did one by hand, carefully, then handed an agent the diff and the remaining thirty-nine.
It finished in an afternoon. Every change was reviewable, most were correct, and the four that were wrong were wrong in the same way, which made them easy to spot. That migration is the clearest thing I can point to when someone asks whether this works.
I lead the team behind a live online classroom. Students and tutors meet on video at a scheduled time, thousands of times a day, and a failure means a lesson stops rather than a page reloads. That constraint shapes everything below. We cannot ship an experiment and see what happens.
The shape of work that agents handle
Bounded, mechanical, verifiable. The flag migration fit all three. So did porting a test suite off a tool that was being retired, and an SDK upgrade across a codebase.
Scoping is the whole skill. “Migrate these forty flags, here is one done correctly” produces a good afternoon. “Clean up our feature flags” produces a mess that takes longer to review than to have written. The difference is whether a human has already made every judgement call and left only the typing.
Review, not generation
Review is where the payoff was.
We run generated and human code through an agent with an adversarial prompt before a person looks at it. Find the input that breaks this. Name the case the author did not consider. It catches a specific class of boundary bug for a few cents, and by the time a human reads the diff, the obvious problems are gone and their attention goes to design instead.
The second thing was internal knowledge. Decisions on my team used to live in Slack threads, which is where decisions go to die. We route that exhaust into a wiki our tools can search, so when a new engineer asks why the session recorder works the way it does, the answer arrives with links instead of archaeology. Nobody trusted it until the citations were there. Then everybody used it.
What did not survive
Autocomplete as the main event. Line-level suggestion is fine and it optimises the part of the job that was never slow. Typing was not the bottleneck.
Unscoped delegation, every time. Our failures share a shape: ambiguous task, no verification step, and output confident enough to read as competent. An agent that is wrong does not announce it. It produces something plausible, and plausible costs more to unpick than broken.
Buying licences and announcing them. Nothing changed on my team until the process changed around the tools. Adoption happened when “ready for review” started including the adversarial pass, and when migration tickets started being written in the shape an agent could take.
What I tell other teams
Match the model to the task. Frontier models for architecture and debugging, cheaper ones for bulk transformation. This reads as cost control and it is. It also forces you to say what a task involves before you hand it over, and half the time that is when you notice the task was underspecified.
The teams getting compounding returns are not the ones with the best prompts. They restructured the boring half of their process so a machine can hold it, and spent the freed attention on deciding what to build. That reorganisation is slower than buying seats, and it is the only version I have seen last.