Failure Modes and Models
How does your Machine Collapse?
This is a second version of a previous blogpost of mine: https://blog.tiruvadi.net/posts/failure/ — take a look over there to see the side/footnote commentary.
Not all machines that achieve the same function are equal to each other — how a machine achieves a goal determines whether and how it fails.
In turn, how and when a machine fails determines how robust it is. How robust do you think AI/LLM is?
Can even you be intelligent if you can’t perform your function unless everything is perfect?
Failure Modes
This post was inspired by this provocative tweet:
This is a sentiment I’m seeing a lot today (in Sept 2023).
My position is simple: failure modes — or how something fails — tells you an incredible amount about how something works inside.
Maybe there are 5 different designs that can all achieve the same function, but it’s really hard to make 2 different designs that fail in all the same ways.
Models Achieve a Goal
Models are machines that are meant to capture relationships. Science cares about relationships that we find in the real world. So it makes sense that good models matter in science.
The idea that some models are useful is important because utility is how we chisel models into something that moves from predictive to insightful. At the end of the day, we want models that are both useful and insightful, but when it comes to applications we really only care about useful models… right?
Models Make Assumptions
Humans are great with adapting to “out of sample” situations — at least better than they have any right to be!
The problem is that all machines have to make assumptions — namely in how they treat the “outside world”.
In the real world, you can’t just abstract away the outside world or assume it will always behave in the way you need it to for your model to hold. That means in the real world your model’s assumptions will be put to the test, maybe even broken. How the resulting broken model “works” depends heavily on its internal structure.
So the answer to the provocative tweet is simply: it matters a great deal.
Example: Two Models
I need a machine that takes an input and multiplies it by a fixed value. So let’s try to implement two different models that achieve the same goal.
Two models that achieve the same goal
We have two models that accomplish the same goal: they take input 1 I1 and they multiply it by a value a. They also take a second input I2 but, when the two models are working, that input doesn’t actually have any impact on the output.
So both models “work”, why should I care how they actually arrive at the output that I want?
Two models that fail in very different ways
Suppose we have a chance that one piece in both model “breaks” and the downstream value of the break turns into a 1. How many different ways can each model break? What would the output be for each of those breaks?
Looks like, very clearly, the two models “work” identically when things are fine, but when things start breaking they start looking very different. The fact that they don’t even have the same number of ways they can break is enough to say: yup, they’re not identical. But on top of that we see that the outputs that they give us are varying levels of useful and workable — in the real world some of these outputs aren’t mission failures. That matters a lot.
Summary
Understanding how a machine or model accomplishes its goal is very, very important in the real world because the real world often breaks foundational assumptions.
The AI we see today may mimic benchmark performances, but they appear to be quite fragile — small deviations break the desired functionality- and brittle — failing spectacularly when they fail.
Understanding failure modes, robustness, and the ability to do a job inside the storm will be critical to move us from chatbots to anything even remotely AGI.
