Whose Fault Is It When AI Makes Mistakes?

Cassie Kozyrkov Cassie Kozyrkov
October 28, 2020 AI & Machine Learning

Don’t get me wrong, I love machine learning and AI. But I don’t trust them blindly and neither should you, because the way you build effective and reliable ML/AI solutions is to force each solution to earn your trust.

(Much of the advice in this article also applies to data insights and models made without ML/AI, especially the very last paragraph.)

Blind trust is a terrible thing.

Before you start thinking that this has anything to do with robots or sci-fi, stop! ML/AI systems aren’t humanlike, they’re just useful thing labelers with poetic names. Instead, their untrustworthiness comes from somewhere else entirely. I think it’s best to show not tell, so let’s see a familiar example…

Whose Fault Is It When AI Makes Mistakes?
Never trust ML/AI blindly, always force it to earn your trust with appropriate testing. Image: source.

Setting the scene for wishful thinking

Welcome back to our cats, whom you might have already met when we built a classifier to split my original dataset of six photos into two groups.

Whose Fault Is It When AI Makes Mistakes?
The original training dataset from Unsupervised Learning Demystified. Do you see the two groups?

Would you look at that? My amazing magical machine learning system succeeded in returning the exact results I was hoping for!

Whose Fault Is It When AI Makes Mistakes?
This example works for supervised ML and for unsupervised ML, so don’t worry about that. The important thing is that the model perfectly identifies my two cats with 100% training accuracy.

Bingo! These were two individual cats and the model returns their labels perfectly, making me misty-eyed with parental pride at my clever classifybot. This means I’ve just built a Tesla and Huxley Classifier!!! …Right?

Our minds play tricks on us

Not so fast! That’s our human wishful thinking playing tricks on us again. I got so emotionally caught up in my kitties that I might not have noticed that all the Tesla photos have a radiator in the background and all the Huxley photos don’t. Did you notice it? Look again.

Whose Fault Is It When AI Makes Mistakes?
This system was never a cat detector. I jumped to that conclusion thanks to confirmation bias, selectively noticing the evidence that confirmed the story I liked and failing to notice evidence that suggests the system is actually a radiator detector.

Cat detector or radiator detector?

My solution— unbeknownst to me — is actually a radiator detector, not a Hux/Tes detector. Testing with new examples has a reputation for being helpful, so let’s see if we catch the problem that way…

Whose Fault Is It When AI Makes Mistakes?
Turns out that using more examples from the same dataset doesn’t do much to, ahem, iron out the issue.

Even if I tested it out on new data by showing it these new photos, there wouldn’t be so much as a peep of an alert. Oh dear. Is that a problem?

Not if Tes is always photographed with a radiator and Hux always isn’t. If that’s the case, who cares how it works? It always works. It’ll give the right cat every time. There’s no problem.

That was a big if. What if the cats move to another apartment (true story)? What if you inherit my classifier without my notes and use it on your photos?

In those cases, the label returned will be “Huxley” and any mission-critical systems relying on my Hux/Tes Detector will crash and burn horribly.

Whose Fault Is It When AI Makes Mistakes?

Whose fault is it?

While the fires rage, let’s do a quick autopsy:

  • As is common with AI, the recipe converting pixels into labels is too complicated to wrap my head around.
  • All I observe is the inputs (pixels) and the outputs (Hux / Tes label).
  • Because I’m human, I don’t notice things that are right under my nose (radiator and Tes always coincide).
  • The story I tell myself about how the inputs relate to the outputs is not only oversimplified, it’s also skewed away from the radiator explanation by wishful thinking.
  • That’s okay. I don’t have to understand how it works as long as I can be sure it does work.
  • The way to check whether it works is to evaluate how it does on a battery of relevant examples it hasn’t seen before.

So far, so good. There’s actually no problem yet. You trust plenty of things without knowing how they work, for example the paracetamol many of us take for our headaches. It does the job, yet science can’t tell you how. The important bit is that you can verify that paracetamol does work (unlike that whole thing with strapping a dead mole to your head).

Image for post
Don’t know how this popular 100+ year old drug works at a molecular level? Neither does science.

Think about complex AI systems the same way as you think about headache cures. Make sure they work and you’ll be fine. Okay, where’s the unraveling? Drumroll, please!

  • Unfortunately, I check performance on examples that aren’t like the examples I want my system to operate in.

There it is. This last one is where it actually went horribly wrong. The rest are okay as long as we test the system appropriately with appropriate examples. So, the answer is: it’s my human fault.

If you test the system for one job and then apply it to another… what were you expecting?

If I teach and test a student using a set of examples that don’t cover the task I want that student learn, then why would I be surprised if there’s a mess later? If all my examples are from the world where Tes and the radiator always go together, then I can only expect my classifier to work in that world. When I move it to a different world, I’m putting it where it doesn’t belong. The application had better be low-stakes because there’s no excuse of “Oops, but I didn’t know it wouldn’t work outside the setting it was made for.” You know. And if you didn’t know before, you do now.

This is why it’s so important to have your goals and users in mind from the beginning. Specify the specs and setting before you begin. Please put a responsible adult adult in charge or stick to toy applications.

Where bigger things are at stake, don’t just throw cool buzzwords at irrelevant datasets.

Without skilled and responsible leadership, well, I hope your application never has anyone’s health, safety, dignity, or future hanging on it…

Common sense, not magic

I’ve been using the word “examples” instead of “data” — they’re the same thing — to remind you that this isn’t magic. The point of ML/AI is that you’re expressing your wishes using examples instead of instructions. For it to work, the examples have to be relevant. The more complicated the task, the more examples you’ll need. You communicate using examples every day, so you knew this stuff already. Maybe you were wondering whether the math says something different; it doesn’t. Rest assured that common sense was your best algorithm all along.

If you want to teach with examples, the examples have to be good. If you want to trust your student, the test has to be good.

Blind trust is a terrible thing

You don’t know anything about the safety of your system outside the conditions you checked it in, so keep these reminders handy:

  • If you didn’t test it, don’t trust it.
  • If you didn’t test it in [environment], don’t trust it in [environment].
  • If you didn’t test it with [user population], don’t trust it with [user population].
  • If you didn’t test it with [data population], don’t trust it with [data population].
  • If an input is unusual, don’t trust your system to output something sensible. Consider using outlier detection and safety nets (e.g. flagging an unusual instance for human review).

If you use a tool where it hasn’t been verified safe, any mess you make is your fault. AI is a tool like any other.

  • Experfy Insights

    Top articles, research, podcasts, webinars and more delivered to you monthly.

  • Cassie Kozyrkov

    Tags
    AIAI & Machine LearningML
    Leave a Comment
    Next Post
    The Next Frontier In Machine Learning Is Something Anyone Can Master

    The Next Frontier In Machine Learning Is Something Anyone Can Master

    Leave a Reply Cancel reply

    Your email address will not be published. Required fields are marked *

    More in AI & Machine Learning
    AI & Machine Learning,Future of Work
    AI’s Role in the Future of Work

    Artificial intelligence is shaping the future of work around the world in virtually every field. The role AI will play in employment in the years ahead is dynamic and collaborative. Rather than eliminating jobs altogether, AI will augment the capabilities and resources of employees and businesses, allowing them to do more with less. In more

    5 MINUTES READ Continue Reading »
    AI & Machine Learning
    How Can AI Help Improve Legal Services Delivery?

    Everybody is discussing Artificial Intelligence (AI) and machine learning, and some legal professionals are already leveraging these technological capabilities.  AI is not the future expectation; it is the present reality.  Aside from law, AI is widely used in various fields such as transportation and manufacturing, education, employment, defense, health care, business intelligence, robotics, and so

    5 MINUTES READ Continue Reading »
    AI & Machine Learning
    5 AI Applications Changing the Energy Industry

    The energy industry faces some significant challenges, but AI applications could help. Increasing demand, population expansion, and climate change necessitate creative solutions that could fundamentally alter how businesses generate and utilize electricity. Industry researchers looking for ways to solve these problems have turned to data and new data-processing technology. Artificial intelligence, in particular — and

    3 MINUTES READ Continue Reading »

    About Us

    Incubated in Harvard Innovation Lab, Experfy specializes in pipelining and deploying the world's best AI and engineering talent at breakneck speed, with exceptional focus on quality and compliance. Enterprises and governments also leverage our award-winning SaaS platform to build their own customized future of work solutions such as talent clouds.

    Join Us At

    Contact Us

    1700 West Park Drive, Suite 190
    Westborough, MA 01581

    Email: support@experfy.com

    Toll Free: (844) EXPERFY or
    (844) 397-3739

    © 2023, Experfy Inc. All rights reserved.