MedML: Patient and Data — Part 2

Training and Testing your Understanding

Every physician asks questions to their patients and tries to understand what’s behind the patient’s response.

Turns out, these questions provide some of the first bits of data physicians work with. Our goal is to then understand what’s causing the symptoms from the data we have.

Follow along with our MedML Workshop (video) and Python Notebook (colab).

In this post we’ll talk about two ways we can do inference on our data

1. Standard Approach — ‘Evidence-based Medicine’ style focusing on experiments
2. Training/Testing Splits — ‘Machine Learning’ style focusing on analysis

Standard Approach

We think heart rate and cardiac output are related to each other. So we could design and run a careful experiment to focus on those two variables, see if they change with each other.

The standard way to think about data — run a good experiment, collect data, analyse the data, see if it ‘looks like something’. This puts a lot of pressure on the experiment.

This involves plugging all our data into a statistic and seeing whether that statistic is significantly different than ‘zero’ — you may recognize that as a p-value!

If we find that our p-value is < 0.05 we would publish it and wait for the next round of experiments to replicate our results. If we don’t find that our data is significant then we’re supposed to move onto the next experiment…

Training/Testing Split

But we’re not interested in the data — we’re interested in the system that the data is measured from, the heart beating behind the numbers. And that system generated every datapoint we have, whether it’s in the first half or the second half.

We can use this fact to do something clever: split our data, randomly, into two pieces. If there’s a relationship between HR and CO, it should be there in both sets.

We’ll call these pieces the training set and testing set.

Splitting your data into a training (top) and testing (bottom) lets you learn a model in the training half then see how well it predicts in the separate testing set.

The goal with the training set data is to learn a pattern, or model. The goal with the testing set data is to see how good our model fits data it has never seen. We calculate how accurate it is and where it makes mistakes. Almost always, these models help us get a better idea of patterns that may be promising.

Interactive Example

The best way to learn is to play with the concept hands on. Check out our Python notebook for an interactive example — take the data, split it up, see how well different splits work!

This example is relatively simple— it turns out training/testing becomes way more useful when we’re looking at hundreds or thousands of variables all at once.

How does this help us treat patients?

Physiology is a bunch of complex patterns, and understanding how your patient’s physiology, and pathophysiology, works is important to improve care. It’s also really hard to do.

ML is already helping us do this by building complex models from data. One way we can double-check these models is by doing Training/Testing splits. We show this in a toy model here, but this general idea can scale up to big datasets with many variables, making it an invaluable tool in pretty much every ML algorithm around.