Missing the dance for the dancer
Statistics is more than isolating variables and testing nulls…
Updated: 2022-01-17
Statistics is something I’ve gone from hating to absolutely loving. The quickest way to convey why I love it: it’s inference. It’s basically “what do I need to do to my data to be able to infer the distribution that generated it”. That distribution? That’s what everyone in science, engineering, and/or medicine is interested in. It’s the truth. Or at least, as close as we’ll ever get to it.
So I’ve gained some statistical pet peeves. The one I’ll briefly introduce today is the notion of testing significance of a multivariate model.
Suppose we’ve learned a linear model $M \in \mathbf{R}^N$ that maps from $M: X \rightarrow Y$ where $X \in \mathbf{R}^N$ and $Y \in \mathbf{R}$. That model has $N$ coefficients.
How can you test to see if the model you learned in statistically significant? There are several ways, and I prefer to test significance on the “outside”, or in the space of Y. I prefer this because, in general, I’m relatively ok with false positives and very averse to false negatives. For an engineer, a false positive is something that will be tested again, because you’ll make design decisions or build off of it in a way that depends on it being true. If it isn’t, your device won’t work and you should catch the false positive error in a troubleshoot. A false negative, on the other hand, may have you ignoring a possibly amazing, useful technique. In many cases, the same argument can be translated to the clinical world, but there is a bit more nuance over there since lives are on the line.
Ok, back to statistics.
An approach that I’ve found people suggest to test the significance of that model is to go into each coefficient, remove it, and see how much it affects the ultimate performance of the model.
This irks me. I’ll use a car as an analogy for why.
Imagine I give you a car and say “this works, it takes you from $A$ to $B$”. And you respond with “well, I want to know if the car is statistically significantly going to take me from $A$ to $B$”. So you remove a part from the engine, and see if it starts. It does. Great, so that part was insignificant. You remove another part. It doesn’t start. I guess that part is immensely significant. You then go to the wheels. You remove this weird pad-like thing. The car still starts. I guess the pad thing on the disc thing in my car isn’t statisticall significant. Ok. Next, let’s remove the gas in the gas tank. Car still starts. Great, so gas in the gas tank is not statistically significant for me to get from $A$ to $B$…
I think you get the idea. Breaking apart a model to test its significance seems absurd as a way to test its significance. Instead, we should take the car for a drive, not from $A$ to $B$, but a trajectory much shorter. Then we can reason through to see if we can reasonably say we can get from $A$ to $B$.
Great. With that off my chest, I guess I’ll go code the coefficient-removal approach to testing significance of my model…
Parting words
I’ll expand on and explore this space in future posts, but just wanted to put down my current thoughts in digital writing. Partly because I want to start talking through what I think is a major limitation in how science is integrating into medicine, partly to figure out if I’m mistaken about anything and learn.
-V