Lockdown Effectiveness: Much More Than You Wanted To Know is the most ambitious post I’ve tried to write since starting the new blog.

I posted an early draft for subscribers only and tried crowdsourcing opinions. Most of the comments I got on Substack weren’t too helpful, but several people sent me private emails that were very helpful.

I had expected that anti-lockdown academics would want to remain anonymous so nobody gave them grief over their unpopular position. I actually found the opposite - the anti-lockdown people didn’t care that much, but the pro-lockdown academics I talked to insisted on keeping their privacy. Apparently pro-lockdown academics who get too close to the public spotlight have been getting harassed by lockdown opponents, and this is a known problem that pro-lockdown academics are well aware of. I was depressed to hear that, though in retrospect it makes sense.

Writing the post made me think a lot of Robin Hanson’s idea of “pulling policy ropes sideways”. The idea is, the Democrats and Republicans (or whoever) are in a giant tug-of-war over some issue, like looser or stricter lockdowns. There are so many people pulling, on both sides, that you adding your efforts to one side or the other will barely matter. Meanwhile, if you pull the ropes sideways - try to make a difference in some previously unexplored direction that nobody is fighting - you can often have much more effect, plus there’s no reason to think that the direction everyone is fighting over is the most interesting direction anyway.

Over the past ~year, I’ve seen endless terrible arguments over whether we should have more or less lockdown. People asked me to write a post on it. It’s something I personally was wondering about and wanted to write a post on. And the dynamics of media - where I get more clicks if I write about things more people are interested in - incentivize me to write a post about it.

But the smartest people I talked to kept - is “derailing” the right word? - derailing onto more interesting and important pull-the-rope-sideways plans. If we had just gotten test-and-trace right at the beginning of the pandemic, we wouldn’t have had to worry about lockdowns as much. Accelerating vaccine production, which we could have done in dozens of little ways, would have made lockdowns less necessary. Having better-targeted or better-choreographed lockdowns is more important than adjusting some slider of lockdown strength from MORE to LESS or vice versa. I felt that some of the experts I talked to were trying really hard to get this across, and I was asking “Yes, that’s all nice and well, but blue state good red state bad? Or red state good blue state bad?”

My first excuse was that everyone else was already doing this, but poorly. I was annoyed at this and wanted to figure out what was going on, and I hoped other people did too.

My second excuse is that it was educational. It was educational for me to figure out who was right vs. wrong about what and how, and to see exactly how big the gap between media perception and reality was. Part of rationality is learning to fight in hostile terrain, and sometimes that means wading into the place where the people are pulling the rope. In a war, it’s helpful to attack places where your enemy doesn’t have a lot of forces mustered - but if you focus on this so hard that you never enter a pitched battle, then you limit your options compared to people who are more comfortable doing that.

This question was too multi-dimensional. As in, you could calculate everything right according to some model, and then someone else could say “but actually none of that matters, the real issue is X”, and you would have a hard time proving it wasn’t.

A long time ago, I remember being asked whether banning marijuana was good or bad. I spent a long time figuring out the side effects of marijuana, how addictive it was, how many people got pain relief from it, how many people were harmed by the War on Drugs, etc - and it turned out all of this was completely overwhelmed by the effects of deaths from intoxicated driving. If even a few people drove on marijuana and crashed and killed people, that erased all its gains; if even a few people used marijuana instead of alcohol and then drove while less intoxicated than they would have been otherwise, that erased all its losses. This was - “annoying” is exactly the right word - because what I (and everyone else) wanted was a story about how dangerous and addictive marijuana was vs. how many people were helped by its medical benefits, and none of that turned out to matter at all compared to some question about stoned driving vs. substituting-for-drunk-driving, which nobody started out caring about.

It might actually be even worse than that, because there was some hard-to-quantify chance that marijuana decreased IQ, and you could make an argument that if there was a 5% chance it decreased IQ by let’s say 2 points across the 50% of the population who smoked pot the most, and you took studies about IQ vs. job success, criminality, etc, really seriously, then lowering the national IQ 1 point might have been more important than anything else. But this would be super-annoying, because the studies showing that it decreased IQ were weak (and you would have to rely on a sort of Pascal-type reasoning) and people reading something on the costs/benefits of marijuana definitely don’t want to read something mildly politically incorrect trying to convince them that IQ is super important. And if there are twenty things like this, then all the actually interesting stuff people care about is less important than figuring out which of the twenty 5%-chance-it-matters things actually matters, and it’s really tempting to just write it off or put it in a “Future Research Needed” section, but that could be the difference between your analysis being right vs. completely wrong and harmful.

The same was true here. How do we quantify the effect of Long COVID? Who knows? Given the giant pile of bodies, maybe we just round COVID off the the number of deaths it causes, and ignore this mysterious syndrome where we’ve only barely begun the work of proving it exists? But under certain assumptions, the total suffering caused by Long COVID is worse than the suffering caused by the acute disease, including all the deaths!

Or: lockdowns cost X amount of money, which is more/less than some other government programs. But the costs of lockdown mostly come out of the pockets of real people, whereas the costs of eg building a new highway or Space Shuttle come out of taxes, which can distribute the burden across everyone (and disproportionately on the rich). Does that mean we can’t really compare these two kinds of costs? I solved this by including an asterisk in the post, saying “maybe we can’t compare these two kinds of costs, who knows?” But plausibly, grappling with this fairly would affect the costs by a full order of magnitude and make lockdowns seem super-terrible. And there are lots of things like this!

This is an especially bad fit with writing a “red state good, blue state bad” (or vice versa) article, because how do you distribute glory/praise/credit/Rationality Points if one side was right about everything except for some complicated weird effect that nobody thought about and which you can’t quantify intelligently? And then when you add this in, maybe the other side was “right” for reasons they never thought of and that they shouldn’t get any credit for at all?

Maybe a more honest version of me would have rewritten the post to focus more on the emotional costs (the part which I made Conclusion 2). It really is a striking result that it’s hard to justify the emotional costs of lockdown even given very optimistic assumptions about the number of lives saved / Long COVID cases prevented / etc. This argument is pretty unrelated to most of what people have talked about in the news, which is mostly (completely false) claims that lockdowns cause more suicides, lockdowns devastate businesses, etc. And it’s so stupid - emotional damages! People being annoyed that they can’t go to the bar (I realize for some people the emotional damages were deeper than that, but not everyone missed a family member’s funeral - I think the part that really adds up is multiplying the inconvenience of not being able to go to the bar by 300 million people). Maybe a more courageous post would have looked more like “Hey, when you add this really simple thing in to the analysis, lockdowns are really obviously bad, right?” But it just felt too weird and transgressive to focus on something authorities weren’t even talking about.

And the most plausible counterarguments tried to muster emotional damages on the other side - maybe lockdowns now funge against lockdowns later which also cause emotional damages. Or maybe not-having-lockdowns makes lots of people terrified of COVID which causes emotional damage. So maybe both sides’ strongest argument is emotional damage, and we should ignore the business closures and hospital overcrowding and so on and just argue about whose emotional damage is worse. But this would be awful, because emotional damage is much harder to quantify than cases or deaths and we’d never get anywhere. But if we don’t do this, maybe we’re streetlight-effecting ourselves into a false conclusion.

I ended up writing the post in a way that focused on maybe twenty different things, some of which resolved in a pro-lockdown direction, and others of which resolved in an anti-lockdown direction - and didn’t focus on the fact that thing number 19 overwhelmed all the others and I don’t know what to think about that.

A friend asked me why I was spending so long on this post, and I responded something like “If I make it long enough, maybe nobody will read it, and then I will get credit for writing an important and well-researched post, but nobody will know what’s in it and so nobody will get angry at me”. This is obviously not the most virtuous way of thinking about things. But I imagine everyone faces similar constraints, and making it so long that nobody will read it is at least better than modulating the conclusions to be more palatable.

This post landed me in the annoying position of having to try to critique mathematical models while being much less mathematically knowledgeable than the people who made them.

I would have preferred to avoid this position, but there were a bunch of contradictory mathematical models producing order-of-magnitude different results, all of which were by credentialled people and published in peer-reviewed journals, and which ones were right vs. wrong turned out to be pretty important.

Challenging mathematically gifted people about math is scary and unpleasant. “We have HARD MATH on our side,” they say. “Yes,” you ask, “but how come this other person also did a mathematical model and got completely different results?” “Oh,” they reply condescendingly, “obviously they failed to crenulate their zeugma. Obviously.” And then you’re confronted with the prospect of either trying to learn enough math to figure out what a zeugma is and why you would want to crenulate one - not just to a basic level, but to a level where you can understand disputes between statisticians about when to do it or not - or just hating everyone in the world and retreating to total Cartesian skepticism.

I’m not a total idiot about statistics. I’ve taken graduate level classes in it, I’ve even occasionally been tapped to teach statistics to doctors - a task which, like my previous job of teaching English to Japanese elementary schoolers, reflects more on the innocence of my students than on my own intellect. But I still frequently found myself thinking back to my old post on Euler, who would win religious debates against atheists by telling them “(a+b^n)/n = x, therefore, God exists! What is your response to that?”, and the poor atheists would be like, “well, you’re a mathematician and I’m not, guess I can’t really argue here.”

One thing I found helpful was to ask Person A what they thought of Person B’s work, relay it to Person B, Person B would say something like “he’s neglecting to consider that you need to fulminate the synecdoche”, then I would relay that back to Person A, and after enough of this I would get a meaningful sense of where they disagreed (or occasionally one of them would just admit they had made a mistake). This felt kind of like an adversarial-collaboration-by-proxy - neither of these people was going to hash out their disagreements on their own, but with me as a middleman they could kind of get it done. I’m actually kind of excited about this idea, since some of the people who I most want to have do adversarial collaborations might not contact each other spontaneously, or might not be very good at writing up their results.

Other times I just went for maximally stupid models and saw what happened - eg correlating the stringency indices for all US states against their death rates. This is much worse than a careful model of the sort that a smart statistician could dream up, but it has very few free parameters: it’s so dumb that it couldn’t lie even if it wanted to. Then I checked whether any of the things I was excluding were important enough to change my conclusion, drew giant error bars, and called it a day. Sometimes this was helpful in bounding things - when you see the plot of state stringency indices vs. death rates, you can see that lockdown states do better than non-lockdown states, but also that there’s no way lockdowns are producing order-of-magnitude improvements. Other times it was helpful in figuring out where the differences in other people’s different models were coming from.

I feel bad about this - it’s a limitation of me rather than a fundamental limitation in how well we can understand this issue - but I couldn’t think of a better way around it.

One thing some people fairly called me out on was asymmetric political bias adjustment.

Some of the more interesting studies finding lockdowns didn’t work were done by an economist whose other work leans right (or at least libertarian). He’d written a lot of articles showing that communism is bad for growth, pushing back against wokeness, debating various things about Hayek, etc. Also some newspaper columns criticizing lockdown in pretty extreme terms, comparing it to “house arrest”, etc. This all seemed, if not suspicious, at least relevant. While we still have to judge his studies fairly, we can understand he has an axe to grind - and I mentioned this in an early draft of the article.

Some people called me out on this. A lot of scientists who did studies saying lockdowns are good have probably written newspaper articles saying lockdowns are good; some of them have probably also used “pretty extreme terms” (one article called relaxing restrictions “human sacrifice”). Probably many of these scientists have published vaguely liberal-sounding things, like showing that unrestrained capitalism has negative effects; probably most of them are pro-wokeness. I didn’t go over lockdown supporters’ previous work with a fine-tuned comb and say “these people have been liberal in the past, therefore they have an axe to grind!”

I don’t really know what to do here. Academia is disproportionately liberal, which makes it stand out when a conservative professor gets a result that supports conservative positions - but just seem like business as usual when liberal professors get a result that supports liberal positions. I hate to be responsible for compounding this unfairness. On the other hand, I can’t shake my feeling that it’s suspicious when this very conservative professor is the only guy finding that lockdowns don’t work.

If a professor comes up with some study that shows guns decrease crime, and that professor is a gun owner, goes hunting every day, and donates to conservative organizations, is that suspicious? What if another professor shows that guns increase crime, and this professor has never owned a gun in her life, hates hunting, and donates to liberal organizations? It feels like “enthusiastic gun owner” is more of a “marked” group than “non-gun-owner”, but that’s just a coincidence - if we had been in a world where academia leaned conservative and most professors owned guns, it would be the opposite. But should we really discount the fact that the pro-gun study professor has an NRA bumper sticker on his car? I’m still not sure how to think about this.

Different people had very different pandemic experiences.

I dutifully discussed Sweden like everyone else, but I was most interested in the American data - whether there was any difference between blue states and red states. It was hard for me to worry too much about the first phase of the pandemic - it seemed like everyone had locked down then, that was the right choice, things had gotten under control, and then the more interesting question was what happened afterwards.

Some of the people I talked to for this post were Western European. They’d had very different times. In particular, Western European lockdowns were much stricter than ours, and Europeans tended to be either angrier about them, or more interested in defending them. Western Europe also seemed more scarred by the early phase of the pandemic and more interested in re-litigating it, to a degree that I hadn’t seen in Americans.

Whenever I talked to Aussies or New Zealanders, they just really wanted to stress that they had ascended beyond such primitive mortal concerns, defeated the virus their own way, and were somewhat annoyed that the rest of us were squabbling about the relative merits of our inferior plans rather than focusing on how great they were. Sorry, guys.

I think I approached this from a very American perspective, which was unavoidable but probably made other people feel like my focus was weirdly out of tune with what they cared about. Sorry again.

Several people brought up that my living in the Bay Area made me too credulous of data showing strong effects for voluntary behavior change. The Bay Area was probably the most voluntary-behavior-changing place in America - maybe because lots of people are in tech and have no problem working remotely.

This interacts badly with a conclusion that everything depends on emotional distress, because people in different countries are going to have different emotions. For example, one person brought up that I (as a younger person) might be underestimating the emotional distress older people feel about COVID, because they (unlike me) are at serious risk of death. This was a fair point - except that some US polls suggest older people report less emotional distress over COVID than younger people. But European polls suggest the opposite is true there - so the US picture might just be because older Americans are more likely to be conservative, and in our political climate that means they’re ideologically committed to being less worried about COVID.

A surprising number of good studies on lockdowns included rationalists and effective altruists among their authors.

I mean, maybe I’m biased when I call them “good studies”. But some of them I identified as good studies first, and then later learned that the people involved were in the same subculture I was. I feel kind of like a member of a conspiracy, armed with the secret hand signals that let me recognize my co-conspirators - “oh, you have a machine learning degree but you’re doing epidemiology, suspicious!” Or “you use the word ‘metaforecasting’ infinity percent more often than any non-rationalist I have met”. Don’t worry, I’m not going to out you people without your consent.

I exploited this shamelessly. When I felt like I had no idea who to trust, I trusted the people who I had a chain of social connections with - friends of friends, things like that. People laughed at my conflict-of-interest statement, but the conflict was serious, and I’m sure that some version of this is responsible for a lot of terrible science and echo chambers. Still, I had to ground my chain of who to trust somewhere, and deep down I still feel like this was a good choice.

But also, this is interesting! Most of these people didn’t start out as epidemiologists. They started out as smart statistics/CS/AI people willing to lend their expertise to any sufficiently important project that needed it. It turned out that the most important projects in 2020 all had to do with modeling coronavirus. And if you were sufficiently motivated and had something to contribute, you could get in it!

This makes me really optimistic about the ability of people to do good cross-disciplinary research - and about the ability of social movements to cross-fertilize it and make it happen.