Be careful with health metrics

Part 1: Signal, Noise, and Metrics

We’re in the era of metrics — measure it, do it a lot of times, ???, answer everything! Including how to live forever!

Abstract representation of measuring (dots) something as complex as health (colorful manifold). No matter how well we measure, we’re missing important pieces and maybe even catching unimportant ones.

Obviously that’s too good to be true. But it’s worth chasing at least, right?

In this post I’ll talk about three common misunderstandings we’re seeing today related to metrics — particularly the driving philosophy of wearables and anti-aging efforts.

Wearables embody the naive hope that measurement is all we need to achieve health. https://mobisoftinfotech.com/resources/blog/wearable-technology-in-healthcare/

In subsequent posts I’ll talk about why these misunderstandings are so damaging, and how chasing metrics may be chasing unicorns.

What is a metric?

First, let’s draw an initial system diagram.

We have a system (blue) we’re trying to assess in a number (metric). Measurement gives us recordings (green) that we can then do calculations on to give us metrics of the system.

This is the smallest system diagram I could think of — in even the simplest metric there are at least 3 moving parts (boxes), and two major processes (arrows).

Let’s talk about the moving parts…

Signal

Misunderstanding 1: The metric = the thing we care about

The thing we care about, or system, sits at the core of our diagram. This could be our heart, our muscles, our brain, or even the health of the entire world.

The system is often something we can’t touch or directly see. So we need to find a way to know what it is from afar.

Conflating the metric with the thing we’re trying to measure is easy to avoid when talking, but a lot of our actions betray a misunderstanding. There’s always a gap between the metric and what we care about — mind that gap.

Measurement of our system is the first step towards getting a workable signal, or reflection of the system we care about in our measurement.

We wish it stopped there, but the gap between your metric and the thing you actually care about can have some unwanted guests…

Noise

Misunderstanding 2: the metric changes = the thing we care about changes

Measuring our system gives us a sequence of numbers, or a recording. This recording, hopefully, contains signal.

Noise can cover up our system, making it looks like our system is changing just because our metric is changing. But this can lead us astray.

Unfortunately, like all measurements, the recording has some unwanted guests: all the other things besides the thing we care about. We call this noise.

Noise can come from all sorts of places, can be correlated with system we care about, and sets a floor for what our metric can see and do.

Noise seems simple, but dealing with noise is very nuanced. More nuanced than we have time to address here. Just know — it’s there and it can cause changes in our recordings that are not changes in our signal.

The key point about noise I want to make here: just because the metric changes doesn’t mean your signal changed — it could have been your noise that changed.

Metrics

Misunderstanding 3: Changing the metric -> changes the thing we care about

A metric comes from some calculation we do on the recordings, typically to “aggregate” across multiple recordings into a single go/no-go signal.

Metrics come from calculations on recordings. If we aren’t careful to separate signal (green) from noise (red) then we’re letting the noise get into our metric. This may or may not break the whole point of the metric.

The metric is ideally an accurate reflection of the system we care about., and how its behaving. In other words, changes to the system will change the metric.

Naively, we also want to think that our metric and our system are 1:1 coupled. In other words, changes to the system will change the metric, and changes to the metric will change the system.

If we do an action to change the metric, then obviously we’re changing the system that we actually care about… right?

That last bit should give you pause.

“But the metric looks healthier, so the engine is healing itself!”

Summary

A metric is the result of measuring something we care about. The goal is for it to capture information that we can act on.

Summary of how we should think about any/all metrics. Changes in health metrics, in particular, need to be carefully analysed to determine if they’re reflecting signal or noise.

Like all measurements, metrics are messy. When things get messy, the relationship between the metric and the system becomes very complicated.

This can be particularly problematic, even deadly, when the metric is a health metric. I’ll talk about this more in the next post.

Takeaways

• Just because you measure something doesn’t mean your measurement reflects that something
• Just because your metric improves does not mean the thing you care about improved

BECOME a WRITER at MLearning.ai

Mlearning.ai Submission Suggestions