What do Zillow, Air Canada, Samsung, and New York City have in common?
They've all experienced highly public and expensive AI failures but not because the technology itself failed.
According to Ryan Spelman, Managing Director of Cyber Risk Consulting at K logix, these incidents demonstrate a recurring pattern: organizations often focus on what AI can do without putting the governance in place to ensure it's used responsibly.
By looking beyond the headlines, we can uncover valuable lessons about accountability, oversight, and governance that apply to organizations at every stage of their AI journey.
Let's look at five examples.
1. Zillow's $200 Million Mistake
In 2021, Zillow expanded its Zillow Offers program by using AI to automate home purchasing decisions. The goal was straightforward: use machine learning to predict home values, purchase properties, and resell them for a profit.
Instead, the company announced it would shut down the program after reporting losses exceeding $200 million.
The challenge wasn't simply that the model was inaccurate. During an unusually volatile housing market, the AI struggled to account for rapidly changing conditions that required human judgment.
The Governance Lesson
Ryan Spelman points out, "the biggest question here is: should AI be used for this decision at all?"
Some business processes benefit from automation. Others still require human expertise, particularly when decisions involve significant financial risk, unpredictable variables, or changing market conditions.
Good AI governance begins by evaluating whether AI is the appropriate solution and not assuming it always is.
2. Air Canada's Chatbot Became Legally Binding
A customer contacted Air Canada's AI-powered chatbot to ask whether they could purchase a flight and later apply for a bereavement fare refund.
The chatbot incorrectly said yes.
After following the chatbot's instructions, the customer was denied reimbursement by Air Canada. The airline argued that the chatbot was a separate system and that it shouldn't be held responsible for the incorrect information.
Air Canada was held responsible for the chatbot's response.
The Governance Lesson
Spelman returns to a simple principle when discussing AI governance, "A computer can never be held accountable. Therefore, a computer must never make a management decision."
Whether an AI system is developed internally or provided by a third-party vendor, accountability always remains with the organization using it. AI can generate recommendations and responses, but it cannot assume responsibility for their consequences.
Every AI-enabled process should have clear ownership, defined review procedures, and someone accountable for the output. Governance ensures that while AI can support decisions, humans remain responsible for making them.
3. Samsung Accidentally Shared Proprietary Source Code
Shortly after ChatGPT became widely available, Samsung engineers reportedly pasted confidential semiconductor source code into the public version of the tool while asking for help debugging software.
Although the employees were simply trying to work more efficiently, the incident raised concerns that proprietary information had been shared with a public AI platform.
Samsung ultimately restricted the use of generative AI tools internally while evaluating new policies.
The Governance Lesson
Spelman notes "Good governance recognizes people will want to use these tools."
Shadow AI has become one of the most common AI governance challenges organizations face today.
Employees want to use AI because it improves productivity. If organizations simply prohibit AI altogether, employees often turn to unapproved tools instead.
A better approach is to provide approved AI platforms, establish acceptable use policies, educate employees on what data can and cannot be shared, and make secure AI usage easier than unauthorized alternatives.
4. Clearview AI's Privacy Problems
Clearview AI built a facial recognition platform by scraping billions of publicly available images from the internet without obtaining user consent.
While the technology itself proved highly capable, regulators argued that the company's data collection practices violated privacy laws in multiple jurisdictions.
The company has faced significant legal challenges, regulatory actions, and financial penalties as a result.
The Governance Lesson
Data governance is just as important as model governance.
Organizations need to understand:
-
- Whether they have permission to use it
- Where training data comes from
- Which regulations apply across different jurisdictions
- How customer and employee data is being processed
AI compliance cannot be treated as an afterthought. It needs to be part of the planning process from the beginning.
5. New York City's Chatbot Started Giving Illegal Advice
New York City launched an AI chatbot to help residents navigate city services.
Initially, the chatbot performed well. Over time, however, it began providing inaccurate and potentially illegal guidance, including advice related to housing discrimination and labor practices.
The issue wasn't that the chatbot suddenly became malicious, it simply wasn't being monitored closely enough as its behavior evolved.
The Governance Lesson
Deploying an AI application is only the beginning.
Organizations should continuously:
-
- Review AI outputs
- Monitor for model drift
- Audit responses
- Log interactions
- Update controls as risks evolve
AI systems are not "set it and forget it" technologies. They require ongoing governance throughout their lifecycle.
The Common Thread: Governance Makes the Difference
These five incidents span different industries, technologies, and use cases, but they all point to the same conclusion: the biggest AI failures rarely stem from the model itself. Instead, they result from missing governance.
Organizations that successfully adopt AI establish clear ownership, involve legal and security teams early, define acceptable use policies, monitor AI outputs, and continuously evaluate whether AI is making the right decisions in the right situations.
As AI capabilities continue to evolve from generative AI to increasingly autonomous AI agents the need for governance will only grow. Organizations don't need to eliminate every risk to benefit from AI, but they do need the right processes and safeguards to identify, manage, and reduce risk while enabling innovation.
The organizations that realize the greatest value from AI won't necessarily be the ones adopting it the fastest. They'll be the ones building it responsibly, with governance serving as the foundation for every AI initiative.
Subscribe
Stay up to date with cyber security trends and more