Be careful with health metrics
Part 1: Signal, Noise, and Metrics
We’re in the era of metrics — measure it, do it a lot of times, ???, answer everything! Including how to live forever!
Obviously that’s too good to be true. But it’s worth chasing at least, right?
In this post I’ll talk about three common misunderstandings we’re seeing today related to metrics — particularly the driving philosophy of wearables and anti-aging efforts.
In subsequent posts I’ll talk about why these misunderstandings are so damaging, and how chasing metrics may be chasing unicorns.
What is a metric?
First, let’s draw an initial system diagram.
This is the smallest system diagram I could think of — in even the simplest metric there are at least 3 moving parts (boxes), and two major processes (arrows).
Let’s talk about the moving parts…
Signal
Misunderstanding 1: The metric = the thing we care about
The thing we care about, or system, sits at the core of our diagram. This could be our heart, our muscles, our brain, or even the health of the entire world.
The system is often something we can’t touch or directly see. So we need to find a way to know what it is from afar.
Measurement of our system is the first step towards getting a workable signal, or reflection of the system we care about in our measurement.
We wish it stopped there, but the gap between your metric and the thing you actually care about can have some unwanted guests…
Noise
Misunderstanding 2: the metric changes = the thing we care about changes
Measuring our system gives us a sequence of numbers, or a recording. This recording, hopefully, contains signal.
Unfortunately, like all measurements, the recording has some unwanted guests: all the other things besides the thing we care about. We call this noise.
Noise can come from all sorts of places, can be correlated with system we care about, and sets a floor for what our metric can see and do.
The key point about noise I want to make here: just because the metric changes doesn’t mean your signal changed — it could have been your noise that changed.
Metrics
Misunderstanding 3: Changing the metric -> changes the thing we care about
A metric comes from some calculation we do on the recordings, typically to “aggregate” across multiple recordings into a single go/no-go signal.
The metric is ideally an accurate reflection of the system we care about., and how its behaving. In other words, changes to the system will change the metric.
Naively, we also want to think that our metric and our system are 1:1 coupled. In other words, changes to the system will change the metric, and changes to the metric will change the system.
That last bit should give you pause.
Summary
A metric is the result of measuring something we care about. The goal is for it to capture information that we can act on.
Like all measurements, metrics are messy. When things get messy, the relationship between the metric and the system becomes very complicated.
This can be particularly problematic, even deadly, when the metric is a health metric. I’ll talk about this more in the next post.
Takeaways
- Just because you measure something doesn’t mean your measurement reflects that something
- Just because your metric improves does not mean the thing you care about improved