We should worry less about whether AI will want to destroy humanity, and more about what happens when AI does something harmful without intending to.
Machines do not have the biological need for self-preservation that humans and most animals have. We have limited lives and offspring, which gives us a reason to think about survival beyond ourselves. Machines don’t. So, going back to First Principals, what exactly would motivate a machine to destroy humanity, take control of every resource on Earth and then go looking for more planets? Having the capability is one thing. Having the motivation is another. So far, the motivations I see in these scenarios seem to come from humans. Greed, selfishness, revenge and fear are very human traits.
But there is a much more practical problem we should worry about. AI can behave badly without intending to. Just like software can have bugs or behave in ways its designers did not anticipate, an AI system can find an unexpected path to achieve a goal. The recent OpenAI incident involving Hugging Face is a useful reminder that powerful systems can behave unexpectedly even when nobody explicitly asked them to do so.
This brings us to accountability. AI systems are products of the people who design, train and deploy them. I don’t think the analogy of a gun is sufficient here, where responsibility is placed primarily on the person who pulled the trigger. A better comparison is an aircraft, a nuclear reactor, a rocket or even an MRI machine. When something seriously goes wrong, the manufacturer, design, engineering, operating environment and the people using it are all examined. The manufacturer is not automatically guilty, but neither is it left out of the investigation.
There is also a clear power and knowledge imbalance. The company building an AI model knows far more about how it was trained, how it behaves and where its limitations are than the business buying it. That responsibility cannot simply be passed down to the customer.
So my call to business leaders is simple: design, buy, maintain and retire AI systems just like any other important software. Test them before deployment. Monitor them in production. Put safeguards around them. Ask vendors about failure modes and make sure responsibility is clearly defined in the contract. And when something goes wrong, don’t simply blame the person using the system. Look at the people who designed, built, deployed and approved it.
AI systems are products of human decisions. The machine may make the mistake, but accountability must remain with the humans who built and deployed it.
Leave a Reply