@hrefna Not going to disagree with you on Nate Silver now; I long ago decided to ignore him.
To take a different direction: I saw him as a reaction to a different problem: Experts predicting & reading into data what should be.
People trying to fix polls to give Hillary the lead she should have. Twitter should go hard down if you can the SREs (as opposed to lots of minor outages & velocity fail).
I'm intrigued how to combine expertise with dispassionate examination of the numbers.
@hrefna Maybe there's a distinction between expertise with stats and expertise in the domain and how they are combined? I dunno.
I know that doing this well is highly valued - lots of money working for hedge funds estimating political/market probabilities in a way that's objective as possible.
There's probably something about taking a multi-model approach: If it makes sense under a range of viewpoints and assumptions, it's probably not bad.
@sgf Heh, so I have an entire Rant™ about this specifically, actually, and as it pertains to Nate Silver that came up during COVID when he got on his "lab leak hypothesis" kick.
When he got his real start that was responding to the fact that scouts in baseball went on "gut instinct" when you had this wealth of data that would contradict them.
So he was applying stats knowledge + some domain expertise (he was a big baseball geek) to an area where there just wasn't a lot of this analysis.
@sgf Something similar happened when he entered into politics—entering into a domain where there was a lot of data but no synthesis of that data that could be easily synthesized by a law consumer—but his major innovation there was in the "for a lay consumer" piece.
PEC and Votamatic (among many others) were running around, I even had my own model that I played with during that era as a hobby project.
There were also a lot of pundits who thought they knew better than the polls.
@sgf So here we have some stats and modeling knowledge intersecting with a domain where the public perception was "we had these polls and then we had these pundits" and the discussion was all around individual polls.
So it isn't bad that he entered into this area, but his models weren't being evaluated correctly and in many ways he promoted evaluating them incorrectly.
He talked about all of the Factors™ that went into it, but it was never clear how many of those factors were needed.
@sgf Some of those factors from a modeling perspective should be looked at dubiously, things that would have a real potential to "double count" certain effects.
When you are a modeler you have a very real power to put your thumb on the scale without realizing it, and it is something you have to be very careful of.
Still, models are evaluated first against results, and his base results were pretty good! But looked potentially quite underconfident and with tail effects that weren't examined.
@sgf That could be forgiven if he hadn't tried to expand into other domains.
While this was all happening something else was going on as well. He was engaging in just Traditional Punditry™ but with a data gloss and not differentiating, or entering into domains where he didn't know what he didn't know, or even domains where both he didn't know what he didn't know AND there were a large number of ground-level mathematical modelers already (things like epidemiology).
@sgf Domains like epidemiology have very strong modelers running around and a massive amount of contextual knowledge. There's even differences between Infectious Disease Epidemiologists and Nutritional Epidemiologists are outside of their lane to talk about it.
Then there were domains like sociological modeling.
These are where you still need statistical modeling specialists (e.g., biostatisticians), but they work in conjunction with domain experts.
@sgf He did none of that and, oblivious to his own blind spots and his own limited expertise, he continued to promulgate this "data journalism" that built models around data that they didn't understand, sharing them with a population who didn't know any better, and reaching conclusions that were in opposition not to pundits but to people who actually did this kind of work already.
That's where some of my early objections here come into play.
@hrefna Thanks for the rant, very interesting! I hope it didn't take too much out of you - sometimes writing things like that energise me, something they wear me down. Great to read, though.
@sgf thanks for listening ^^ This is one of my Energizing Topics™ (before going into industry I wanted to go into Epidemiology or biostatistics and did a lot with ecological modeling) so really the energy is in stopping myself short of writing a dissertation on it >>
@hrefna
From outside the US, I don't understand people's issues with Nate Silver.
He spent a lot of time explaining his model. It's all pretty transparent.
I think people are angry because something he said had 30% chance of happening happened. Which, you know, is what 30% chance things do about 1/3 of the time.
This is the problem with the #SafeHomeDotOrg (malpractice) report with flawed methodology.
It was just "we took some numbers and had our data person opaquely added some math to it! All good now!"
"Wait, why is it a problem that our model shows Alaska and Arkansas above Colorado? That's just what the data says!"
But that isn't what the data says, that's maybe what can be derived if you know nothing about the data, but part of the job of a data professional is to look past the superficial.
Well, you took a bunch of data that has flaws and added some vague handwaving to account for it.
You know, sometimes mathematical modeling is that way. Probably more often than we'd like to admit. But you need to report what you are doing with some validation.
Which is all data modeling not data analysis.
Except your model isn't even predicting anything testable. You're just generating reports of a model's output that you haven't validated for no reason but marketing your own brand.