2000, Nick Bostrom
essay explores some of the social, political, economic and technological issues
that the world may have to face in the mid-21st century. A central
theme is the need to regulate molecular nanotechnology because of its immense
abuse potential. Advanced nanotechnology can be used to build small self-replicating
machines that can feed on organic matter - a bit like bacteria but much more versatile,
and potentially more destructive than the H-bomb. The necessity to prevent irresponsible
groups and individuals from getting access to nanotechnological manufacturing
capability is a prime concern in 2050. The essay shows how this quest for containment
shapes many aspect of society, most notably via the institution of a global surveillance
network. A dialogue format is used for two reasons. First, in order to enable
several conflicting views to be discussed. And second, to illustrate how public
debates may be conducted at that time.
This essay was written under various external constraints, including that it should deal with the in 2050, so not too much weight should be put on the time line and some other details.]
Much has happened over the past fifty years, but what are the changes that
really matter? Have they been for better or for worse? And where are we heading
in the next fifty years? With us tonight to discuss these questions are three
very distinguished polymaths: Emily Brown, CEO of Cisco Systems; Dr. Chun Ten,
technology editor of The Economist; and Neil Brigge, professor of philosophy at
New York University and a director of the Foresight Institute. Neil, do you think
that the human condition has changed over the past half century?
We live longer and healthier lives. We are much richer. We have the ability to
choose our sensory experiences to a much greater extent than ever before. Right
now, you are watching a smooth-skinned, well-groomed Neil, but the real physical
Neil sitting here in my VR-box may be a much less appealing sight - I may not
have shaved or washed for a week!
People are still concerned with how they present themselves, both in physical
space and in cyberspace. The fraction of our income that we spend on personal
presentation has been increasing steadily throughout the century. As it becomes
easier to satisfy our material needs, we seem to focus more on what we are in
relation to others - there is great demand for what the economists call "positional
We are the vainest people ever?
Or the most "people-centered", depending on how you want to see it.
I now see a backlash against conspicuous consumption in some communities. High-status
people try to signal their superiority by shunning cosmetic filters and other
conveniences that the less well endowed cannot afford to do without. But this
"nudism", as they call it, is very partial. It is only a few selected
enhancement-options that are being stripped off.
At the turn of the century, many were concerned about global warming and other
ways in which humans were damaging the environment. Global warming turned out
not to be a significant problem, but did this early environmentalist movement
have any lasting effects on how things developed?
Environmentalist thinking opened a lot of minds to the idea that we are living
on this finite planet and that what is done in one part of the sphere can affect
what happens on the other side. "Global conscience", was perhaps the
most beneficial lasting effect. Remember that it is only in the last twenty years
that widespread undernourishment has finally been eliminated. Of course, bioengineering
and nanotechnology were essential tools in achieving that, and this helped reshape
environmentalist thinking. At the turn of the century, many environmentalists
were very suspicious of new technology. Now most people accept that it is thanks
to technological progress that we are able to sustain a world population of eleven
billions and still preserve significant parts of the natural environment.
What also happened was that the early focus on the environment gradually widened
to include all aspects of human life. Today we have the debate between transhumanists,
who are enthusiastic about transforming the human condition, and naturalist conservatives
who are happy about technology as long as it doesnít change human nature too much.
While most naturalists will take life-extension supplements, they donít approve
of the use of mood-drugs or cosmetic psychopharmacology.
The emergence of law and order in the international plane - "global conscience"
as Emily calls it - is probably the reason why we are still here today. There
is still no effective global immune system that could defend against malicious
self-replicating nanomachines. Without the steadily improving global surveillance
network, some terrorist group or scientifically-literate madman might long ago
have made their own nanotech assembler and manufactured a "virus" that
could have destroyed the biosphere and eradicated intelligent life from this planet.
Indeed, and this could still happen. It just takes an ill will and a single security
breach somewhere. We must just hope the surveillance network improves rapidly
enough that this will never happen.
How was it possible for the surveillance network to be put in place so quickly?
Well, a lot of the infrastructure was already there. Companies monitored customers,
the police monitored public places, the military monitored foreign countries.
Once gnat-cameras became cheap and you could link everything up to data bases
and face-recognition software, surveillance networks began covering larger and
larger territories in more and more detail. The other part of the explanation
is that policy-makers could become persuaded by the necessity to regulate nanotechnology.
And here I think better collective decision-making institutions played a key role.
Idea futures is a good example of this. As you know, idea futures is the market
where speculators place bets on hypotheses about future scientific or technological
breakthroughs, political events and so forth. Trading prices - the odds - in idea
futures markets gradually became recognized as authoritative estimates of the
probabilities of possible future events. In the twentieth century, policymakers
would sometimes commission the opinion of bodies of experts that they appointed,
but it turns out that the market is much better at making these predictions than
politically selected committees. Such markets were once banned under anti-gambling
laws in most places. Only gradually were exceptions granted, first for a stock
market, then for various commodities and derivatives markets, but only in this
century did we see the rise of wide-ranging free markets with low transaction
costs, where speculators could trade on most any claim. Itís hard to overestimate
how important this was in making societyís decision-making a bit more rational.
I agree that this was important. The tragedies of the tens and twenties, where
genetically engineered biological viruses were used to kill millions, also helped
prepare the world. People have seen the damage that a malicious guy can do with
self-replicating organisms. And nanotech is much, much more powerful. I think
we have been incredibly lucky so far. Maybe we are witnessing the result of what
philosophers call an observational selection effect. Maybe most civilizations
in our infinite universe destroy themselves when they develop nanotechnology,
but only the lucky ones remain to wonder about their luck. So our success so far
should not make us complacent.
Itís amazing how quickly people have got used to the idea that everything they
do can now be known by anybody who is interested in finding out. When you are
going on a date with someone, you can check out their previous relationships,
and so on. If you had suggested this to somebody fifty years ago, theyíd have
been horrified! They would probably have referred to it as Brave New World,
or Orwellís 1984, with Big Brother watching you all the time. But itís
like a nudist colony: when everybody is naked, the embarrassment quickly wears
off. So we had all these little secrets that we thought were so important, little
vices. But when we can see that everybody has similar little vices, our standards
adapt and we become more tolerant. As long as youíre not doing anything really
bad or break the law, you donít have to worry. And the streets are much safer
key to preventing global surveillance from turning into global suppression is
to always insist on basic liberties and to keep government and law-enforcement
agencies under constant scrutiny. It is crucial that we make sure that the system
is transparent in both directions, so that we can watch whoís watching us.
Yet there are still rumblings about it being a plot by the technologically advanced
countries to monopolize the power coming from the nanotech revolution.
The logic is crystal clear to every honest thinker. Since there is as yet no general
defense against nanotech attacks that could potentially destroy the biosphere,
such attacks must be prevented at any cost. The only way to do that is to limit
the number of powers who have nanotechnology. If a single evil madman gets it,
then itís "game over".
I agree that is a good argument, and that we have to limit access to nanotechnology
until we have developed reliable defenses. But that doesnít mean that the leading
powers donít derive an unfair advantage from this technological monopoly. The
people who are excluded from building nano-assemblers have a right to be compensated.
Current transfer payments are nowhere near what they are entitled to.
It is also a problem of intellectual property laws, which has been a burning issue
for quite some time now. - [Emily Brown disappears]
It appears that we have lost transmission from Emily - she was on her yacht. Hope
she hasnít run into bad weather - Ok, letís turn our gaze towards the future.
What lies ahead? Neil, I know that you have expressed pessimism about the future
not a pessimist, but Iím concerned that we might have underestimated the risk
that nanotech could proliferate before we can build defenses. There has been a
series of incidents already. As technological know-how spreads, it may be harder
and harder to make sure that no rogue group develops an assembler. Building an
effective global immune system for self-replicating nanomachines is a very, very
difficult challenge. It has to be present everywhere, on land, sea, and in the
atmosphere. It must avoid attacking biological life forms. And it has to work
100% of the time. We are very far from being able to do that at the present time.
Surveillance technology is improving rapidly, however.
Yes, but what if someone develops a good counter-surveillance technology? Intelligence
agencies are now discussing proposals for monitoring and preventing research in
that field as well. But the problem is that the task is so ill-defined. Who can
say what research might potentially lead to some way of tricking the global surveillance
Are there any idea futures claims that measure the probability of these things?
Yes, but they donít really tell us much. You see, who would want to bet on the
hypothesis that civilization will be destroyed? If you are right, you would not
get paid! It is disconcerting to note that what economists call the time discount
factor seems to have been going up. (The time discount factor is a measure
of how much more value people place on a present good compared to having the same
good at some point later in the future.) That could be interpreted as people being
worried about whether they will be around to benefit from their savings. [Emily
Welcome back, Emily!
Iím terribly sorry! Iím on a yacht here, and my son was playing with the antenna
- Iím so sorry!
Donít worry. My attention-meter shows that the intermezzo increased our eyeball-count.
Thatís why we are doing this live. Our viewers like it when things go wrong! You
should give your kid a bonus!
I donít think so!
We were talking about the future. Neil was just trying to convince us that
he is not a pessimist, and he went on to explain all the excellent reasons we
have for thinking that there is a great risk that we will all be turned into gray
goo by nanoreplicators running amok. Is that a fair summary, Neil?
I suppose so. But on the other hand, if we do manage to avoid doomsday, things
could turn out really nice. Thatís why I donít call myself a pessimist, because
if we are careful we might achieve a great outcome.
My grandfather is 102 years old and he is quite active. This is all thanks to
life-extension. We can only dream of all the wonderful things that will become
possible when we learn to redesign and expand our minds. There is more to lose
than ever before. Having got this far, we really must try to make sure that we
donít destroy ourselves with rogue nanotechnology, even if it means holding some
types of developments back temporarily.
There are many people who wonít make it. Over a hundred thousand people deanimate
everyday, many due to aging-related ailments. More than two thirds of the worldís
population still donít have a cryonics contract! When they deanimate, instead
of having the information in their brains preserved by freezing or vitrification,
they are just cremated or buried. For worms, there certainly are such things as
free meals! And this at a time when in the idea futures market there is more than
a 50% chance of reanimation of a human brain within three years! This preventable
loss of human life, day after day, year after year, almost makes the two world
wars or the Baghdad flu insignificant by comparison.
The difference is that this is largely self-selected death. Itís a price you pay
for freedom of religion.
For some people that is true. But there are millions who simply cannot afford
a cryonics contract, or donít have enough education to understand what the options
are. Now, Iím not suggesting that we should force anybody to have themselves
frozen when they deanimate, but I think we have a duty to provide enough information
and financial help so that each individual can at least make an informed decision.
Wonít overpopulation be a big problem if all the people who are currently in
cryonic suspension are suddenly reanimated?
Actually, there arenít that many of them. About three billion people have made
arrangements to have themselves suspended when they deanimate, but only about
150 million people have been suspended to date.
Idea futures indicate that if these people are reanimated, it will be through
being uploaded into computers. It looks much more feasible to disassemble these
vitrified brains cell by cell, molecule by molecule, scanning off the neural network,
and then run an emulation of that neural network on a computer. In vitro
repair is harder than in silico repair - that is, doing it in a computer
simulation. And I, for one, would much rather be uploaded than having my biological
brain repaired. I already spend most of my time in virtual reality, and Iíd like
the security of being able to make a back-up copy of my mind every hour or so.
If for some reason I want to manipulate physical objects, I would rent a robot
body that was suitable for what I wanted to do.
I can understand those who are scared of uploading. Iím scared of it myself. Who
can guarantee that a brain emulation would truly be conscious? And even though
our virtual reality is pretty good at vision and sound, I still think it canít
compete with the meatspace in the other sensory modalities. Virtual sex is great,
but I prefer to touch my husbandís body directly.
The other sensory modalities of virtual reality will work much better for uploads.
It is easier to do it for uploads, because you can directly activate neurons in
their sensory cortices. You donít have to build these complicated mechanical devices
we have now to stimulate our body surfaces.
I still see it as a last resort. I would rather be an upload than be dead, but
I think that as long as my biological body is functioning well, Iíll stick with
could be huge advantages to being among the first uploads. I was attending a neurocomputation
conference last week, and I was talking to several groups there who are studying
hypothetical ways of enhancing the intellectual abilities of an upload, by adding
new neurons and new connections and so forth. If uploads could become more intelligent
than humans, I would prefer to be an upload. Also, computing power increases.
There is an idea futures claim that within two years of the first human upload,
there will be an upload running at a clock speed one hundred times greater than
a biological brain. That means the upload will think a hundred times faster than
we do! Last time I checked, this claim was trading at better than even odds.
Since uploads are software, they can easily make copies of themselves. We could
see a population explosion with a vengeance, and we might have to brush up on
Malthus again. Malthus was a late eighteenth century political economist who argued
that unless population control is instituted, then population growth will eventually
annihilate any improvement in the standard of living for the masses that economic
growth has brought about. Even rapid space colonization, which hasnít been economical
so far, but which idea futures indicate is likely to begin within a decade, will
not be able to keep up with a free-breeding population of uploads.
Uploading raises huge ethical and political issues. Not to speak of the legal
challenges. If you are an upload and you make a copy of yourself, so there are
now two almost identical implementations of you, who is married to your husband?
We have only scratched the surface of this problematique. I would like to see a
moratorium on uploading experiments and some form of artificial intelligence research
until we have a better understanding of where we are going. We are playing with
afraid weíre out of time. Itís been fascinating. I suppose weíll see over the
next few years how this plays out. Thank you all for participating. Neil, I leave
the final word with you.
We are in the process of crossing a road. The one thing we must not do is stop
in the middle. We need greater-than-human intelligence to build defenses against
nano-attacks. We would not reduce the danger by slowing down; on the contrary,
that would make the risks even bigger. The best we can do is to press onward with
all possible speed, using as much foresight as we can muster, and hope that there
is an other side that we can get to.