The scare quotes are well placed. The problem with Bostrom et al. is not that they are spending money. It's not even the fact that they are wasting their time on imaginary risks. It's that, by trying to bend policy around those imaginary risks, they are increasing our vulnerability to real-life risks.
For example, there has been a lot of talk for the last few decades about bioterrorism. In 2001, it finally happened; a bioterrorist attack occurred. It killed five people.
In the meantime, natural infectious diseases kill several million people every year, and far worse is possible. In 1918-1919, a flu epidemic killed two or three times as many people in a single year as the First World War had killed in four years. In 2002-2003, a SARS outbreak killed a thousand people before it was stopped by quarantine measures. We got off _very_ lightly; SARS is at least as infectious as the 1918 flu, is no more curable, and has nominally roughly twice the lethality rate. The real lethality difference is higher, for the 2002-2003 outbreak was small enough that most of the victims could receive oxygen treatment, which significantly improves survival; in a pandemic, of course, there would not be enough such treatment to go around. And there have been historical plagues deadlier by far than SARS. Terrorists have political goals relative to which the weapon that killed their own communities and families would be counterproductive. The forces of mutation and natural selection working on disease organisms have no such reason for restraint.
Now what exactly do the bioterrorism criers think is going to happen if they start being taken seriously enough to influence policy? It's only too obvious what's going to happen, because it's the same thing that always does: more regulation on biotech research, more red tape, more of the sort of field day for the paranoid and bureaucracy gone mad that we already see in the security theater at airports. It's hard enough to fly to Disneyland in those conditions, let alone do cutting-edge research.
And so when - not if - something deadlier and more insidious than SARS does come along, whatever chance we might have had of being prepared for it may end up being thrown away.
It's not difficult to understand why our instinctive assessment of threats is so wildly irrational. In the ancestral environment there was plenty of disease, to be sure, but almost no medicine, so little selection pressure to be sensitive to that threat. Violence was the main cause of death _that you could do something about_. The human brain is wired to assume our fellow man is the primary threat, regardless of the facts.
But we can sometimes override even hardwired assumptions, once we understand that we live in a world in which they are no longer true. We had better learn to override this one, and quickly.
You might as well say there was a lot of talk about flight in the centuries before it was actually invented. Historical data can only take you so far.
>Now what exactly do the bioterrorism criers think is going to happen if they start being taken seriously enough to influence policy? It's only too obvious what's going to happen, because it's the same thing that always does: more regulation on biotech research, more red tape, more of the sort of field day for the paranoid and bureaucracy gone mad that we already see in the security theater at airports. It's hard enough to fly to Disneyland in those conditions, let alone do cutting-edge research.
Are you in favor of government regulation for people researching new designs for nuclear bombs?
It's all about cost benefit analysis. The TSA is a waste of time not only because hijacking is rare, but because the average hijacking kills fewer than 1000 people. If there was a strong theoretical argument for how a hijacked plane could permanently end the human race, it would not be a waste of time.
I'm in favor of unregulated biotech research if the expected benefit from additional disease cures exceeds the expected risk from potential engineered viruses. Frankly, I'm more concerned about the engineered viruses because it seems like an engineered virus has a better chance of killing off everyone (as opposed to not everyone, which is an extremely important distinction in my view; I'm playing for team humanity) just because engineered things tend to work better than things that assemble by chance.
> Are you in favor of government regulation for people researching new designs for nuclear bombs?
The cases are not particularly similar. It doesn't matter that we have crippled nuclear weapons R&D with regulation. It will matter a great deal if we similarly cripple biotech R&D.
> it seems like an engineered virus has a better chance of killing off everyone (as opposed to not everyone, which is an extremely important distinction in my view; I'm playing for team humanity)
I am not at all as optimistic as you are about that. It is the way of extinction that what kills the last individual may have nothing to do with the underlying factors that doomed the species. The last passenger pigeon died of old age. In my view, far more likely than a single super-disaster that kills everyone at the same time is a sequence of events where one factor delays progress enough to make us vulnerable to another that kills enough people or causes enough disruption to crash civilization, at which point we can't reboot because all the easily accessible fossil fuel deposits are long gone and there's no way to jump directly from wood-burning stoves to solar panels, leaving the survivors to struggle along until ordinary geological processes finish us off. That's the kind of scenario I'm concerned about.
It's definitely plausible that you are right and that deregulating biotech research is the way to minimize existential risk.
I'm not sure I've seen any statement from the Future of Humanity Institute on this issue, except for this blog post by research associate Robin Hanson which would appear to support your point of view:
