great article again. The documentation gap the piece names is real. But there's a layer beneath it that doesn't get named often enough: organisational memory.
Every established organisation runs on a mixture of systems — some modern, some old, some that predate the current leadership by a generation. We talk about legacy systems, shadow IT, technical debt. What we talk about far less is the reason things are the way they are. The incident that prompted a rule. The regulatory penalty that caused a constraint to be hardcoded. The customer complaint that rewrote a workflow. The catastrophic failure that introduced a flag nobody touches, in a system nobody fully remembers, with a comment in the code that says "do not change this" and nothing else.
The person who knew why is gone. The ticket is closed. The meeting notes were never written. The knowledge lived in one engineer's head for fifteen years, and when they left, it left with them.
AI doesn't see organisational memory. It sees a pattern that looks inefficient. It sees a field that's always set to the same value and reads that as an optimisation opportunity. It sees a rule that appears redundant against current data and removes it. The development team reviewing the change doesn't know why it was there either — because it was lost to history long before the AI arrived.
That's the risk this piece is circling but not quite landing on. It isn't just that AI touches systems we don't understand. It's that AI can't distinguish between a legacy inefficiency and a legacy safeguard — and nor, increasingly, can we. The consequences of getting that wrong aren't always visible until they're serious.
This was a really interesting read. As a freelance writer and editor it's natural for me to be very cautious of AI but it is undoubtedly going to be a part of our lives for the foreseeable future. I want to learn more about AI and how it works so that we can work with or alongside it in harmony rather than have to compete against it. Thank you for sharing!
This is really thought-provoking because it’s not just about AI, but about how little humans sometimes understand the systems we already built ourselves. And the line about “automation without understanding” feels especially important. Technology becomes dangerous when speed starts mattering more than awareness. Sometimes it feels like we’re so focused on making systems more efficient that we forget to ask whether we still truly understand the world we’re handing over to them.
This is such an important distinction. The scary part isn’t just AI becoming more capable, it’s humans becoming less connected to the systems making decisions around them. “Automation without understanding” feels like the real warning sign because efficiency can quietly replace awareness before anyone notices.
great article again. The documentation gap the piece names is real. But there's a layer beneath it that doesn't get named often enough: organisational memory.
Every established organisation runs on a mixture of systems — some modern, some old, some that predate the current leadership by a generation. We talk about legacy systems, shadow IT, technical debt. What we talk about far less is the reason things are the way they are. The incident that prompted a rule. The regulatory penalty that caused a constraint to be hardcoded. The customer complaint that rewrote a workflow. The catastrophic failure that introduced a flag nobody touches, in a system nobody fully remembers, with a comment in the code that says "do not change this" and nothing else.
The person who knew why is gone. The ticket is closed. The meeting notes were never written. The knowledge lived in one engineer's head for fifteen years, and when they left, it left with them.
AI doesn't see organisational memory. It sees a pattern that looks inefficient. It sees a field that's always set to the same value and reads that as an optimisation opportunity. It sees a rule that appears redundant against current data and removes it. The development team reviewing the change doesn't know why it was there either — because it was lost to history long before the AI arrived.
That's the risk this piece is circling but not quite landing on. It isn't just that AI touches systems we don't understand. It's that AI can't distinguish between a legacy inefficiency and a legacy safeguard — and nor, increasingly, can we. The consequences of getting that wrong aren't always visible until they're serious.
This was a really interesting read. As a freelance writer and editor it's natural for me to be very cautious of AI but it is undoubtedly going to be a part of our lives for the foreseeable future. I want to learn more about AI and how it works so that we can work with or alongside it in harmony rather than have to compete against it. Thank you for sharing!
Thank you so much! I'm glad you found it useful!
This is really thought-provoking because it’s not just about AI, but about how little humans sometimes understand the systems we already built ourselves. And the line about “automation without understanding” feels especially important. Technology becomes dangerous when speed starts mattering more than awareness. Sometimes it feels like we’re so focused on making systems more efficient that we forget to ask whether we still truly understand the world we’re handing over to them.
This is such an important distinction. The scary part isn’t just AI becoming more capable, it’s humans becoming less connected to the systems making decisions around them. “Automation without understanding” feels like the real warning sign because efficiency can quietly replace awareness before anyone notices.