March 13, 2000 / By JOHN MARKOFF
An unlikely prophet is voicing a plea for reason and restraint in the
increasingly chaotic stampede toward the technological future.
Breaking ranks with the customarily optimistic and self-congratulatory
high-technology industry, Bill Joy, the chief scientist of Sun
Microsystems, has issued an impassioned critique of uncontrolled
progress in digital, biological and material sciences. He has challenged
scientists and engineers to rethink their ethical standards and step back
from advances that might ultimately threaten the human species.
In a 20,000-word essay in the April issue of Wired magazine,
which goes on sale Monday, Joy, a computer industry
pioneer and one of the nation's leading technical authorities,
writes that although the world has survived any number of
potentially devastating technologies developed in the
20th century, several new branches of research pose
threats of technological devastation at the hands of a
small group or even an individual.
"The 21st century technologies -- genetics, nanotechnology and robotics
-- are so powerful they can spawn whole new classes of accidents and
abuses," he writes in the article, titled, "Why the Future Doesn't Need
Us."
"Most dangerously, for the first time, these accidents and abuses are
widely within the reach of individuals or small groups," he writes. "They
will not require large facilities or rare raw materials. Knowledge alone will
enable the use of them."
Such warnings about unbridled progress are not new, but until now they
have typically come from social critics or scientists outside the
mainstream.
In contrast, Joy is a leading computer researcher who developed an early
version of the Unix operating system while a graduate student at the
University of California at Berkeley. He helped found Sun Microsystems
and more recently has been involved in the development of powerful
software technologies like the Java programming language and Jini, a
system for linking hundreds of thousands of appliances and other devices
by way of the Internet. He also served as co-chairman of a presidential
commission on the future of information technology.
Indeed, Joy also wrote a more generally optimistic cover article titled
"Digital Wonders" for the March 6 issue of Fortune magazine. In a
telephone interview, he said the Fortune piece was focused on short-term
issues in the computer industry, while the Wired article addresses
long-term forces that will be more difficult to control.
He lives in Aspen, Colo., where he moved in 1989 from Silicon Valley to
set up a small laboratory that explores new technologies for Sun, which is
based in Palo Alto, Calif.
Joy's critique is striking because computer industry technologists, almost
without exception, take a more sanguine view of the social consequences
of advances. They typically argue that any negative effects will be far
outweighed by the positive effect of new technologies.
Nathan Myhrvold, a physicist who is on leave from his job as the chief
technology officer at Microsoft Corp., said in an e-mail interview,
"People have made apocalyptic predictions about technology constantly
for as long as there has been technology. I think it is because change
frightens them. What is more, the most common form these dire
predictions take is 'this next generation of stuff -- wow! that is really
different and really scary.'"
But the new technologies will be more difficult to control in the future, Joy
argues, because most are being driven by the commercial sector, not by
the military, which in the past has controlled many potentially dangerous
technologies.
Danny Hillis, a computer scientist who founded Thinking Machines Inc.,
an early supercomputer company, said of Joy: "Bill is pretty unusual.
There are very few technologists who step back and try to look at the
whole picture. Most who do tend toward optimism."
Joy argues that advances are now on the horizon in each of
the three areas he addresses. In the field of robotics, he
warns of a generation of superintelligent machines that
could compete with their human creators for resources. Such possibilities
have recently been addressed by other technologist-authors, including
Ray Kurzweil ("The Age of Spiritual Machines: When Computers Exceed Human
Intelligence"), Hans Moravec ("Robot: Mere Machine to Transcendent Mind")
and George Dyson ("Darwin Among the Machines: The Evolution of Global
Intelligence").
In genetics, Joy worries that the widening availability of knowledge about
powerful genetic engineering will lead to a "white plague" -- a
human-designed disease that can kill selectively.
In nanotechnology, he describes the increasing possibility of an accidental
or intentional release of a submicroscopic self-replicating mechanism that
could cause widespread destruction.
Such a calamity was first suggested by Kurt Vonnegut. In his novel "Cat's
Cradle," a substance called ice-nine sets off a thermal chain reaction that
leads to the freezing of the oceans.
"A lot of people have looked at all of this as science fiction, and as a
result they haven't taken it seriously," Joy said.
At the heart of his warning, he writes, is an attribute that has not been
true of destructive technologies until now but is a part of all of the most
compelling 21st-century technologies: the ability to self-replicate.
"While replication in a computer or a computer network can be a
nuisance," he writes, "uncontrolled self-replication in these newer
technologies runs a much greater risk: a risk of substantial damage in the
physical world."
Joy insists he is not a Luddite, and yet he also argues that the issues
raised by the convicted bomber Theodore Kaczynski have not been
adequately addressed by the nation's technologists.
"Kaczynski's actions were murderous and, in my view, criminally insane,"
he writes. "But simply saying this does not dismiss his argument; as
difficult as it is for me to acknowledge, I saw some merit in the
reasoning."
Joy suggests parallels between the ethical dilemma faced by nuclear
physicists in the invention and the use of the atomic and hydrogen bombs
and similar challenges facing technologists today.
"We should have learned a lesson from the making of the first atomic
bomb and the resulting arms race," he writes. "We didn't do well then,
and the parallels to our current situation are troubling."
He argues that today's scientists must find a path that was not taken by
physicists during World War II: "I feel, too, a deepened sense of
personal responsibility -- not for the work I have already done, but for
the work that I might yet do, at the confluence of the sciences."
"The only realistic alternative I see," he writes in Wired, "is
relinquishment: to limit development of the technologies that are too
dangerous, by limiting our pursuit of certain kinds of knowledge."
He cited the precedent of the decision by the U.S. government to
abandon the development of biological weapons.
Verification in the case of genetics, robotics and nanotechnology,
however, will require that scientists adopt strict codes of ethics, he
writes.
In the telephone interview last week, Joy said he doubted that the
development of advances could be reined in in the commercial world,
and he criticized scientists as being largely silent on the inherently
destructive potential of rapidly evolving technologies.
Asked if he thought a technological species could expect to survive the
ever-accelerating evolution of its market-driven technologies, Joy said:
"The answer is 'yes, but not without additional care.' I think it's possible
-- but it's not a given. Survival won't come for free.
(posted without permission)
