Will Complex Code Cause Our Collapse?

Our complex society increasingly depends on computer software code, and that code is growing ever more complex and unmanageable. It is already very common in large companies for there to exist large software systems that no single person understands very well. When you have a large software system with more than 100,000 lines of code, it will often be that one person knows certain aspects of the software system, and someone else knows other parts of it; but there is no person who understands the full system very well.

As demand for software functionality grows, software engineers sometimes resort to using code generators. These are software tools that can quickly generate many lines of code. But such code is often very hard to understand. By using a code generator, a software developer may quickly add 10,000 lines of new code to a software system. But he may not understand such code. A rough rule of thumb among programmers is: if I didn't write the code, I don't understand it.

Many advanced computer programs use what are called neural networks or deep learning. When such code is used, the software ends up being pretty incomprehensible to humans. Software decisions end up being driven by extremely complex data, often data that is distributed across many different layers. In complex cases of such implementations, the computer itself doesn't understand how the data determines the decision, and neither does a human. It's what programmers call a “black box.”

There is a strong possibility of a future complexity crisis in which humans find they have created software systems of unfathomable intricacy that they can no longer understand. We can imagine a certain level of complexity – call it Complexity Level 10 -- that is the most complex level that any human can understand. It is all too possible that humans might build their way up to Complexity Level 11 or Complexity Level 12 or some higher level. There would then be a possibility of an “overshoot and collapse” situation, in which computer systems around the world start to break down because they have become too complex for anyone to understand, maintain or fix.

You don't have to have lots of bugs for a complex system to fail. A space probe to Mars failed because of a single line of errant software code. In a case like that, it wasn't good enough that 99.999% of the code worked right.

On May 6, 2010 there occurred an event called the Flash Crash, in which the stock market underwent a trillion-dollar dip, dropping by 900 points at 2:32 PM. By the end of the day, the market had largely recovered. The dramatic dip of the Flash Crash was apparently caused by program trading, in which investment portfolios are controlled by extremely complicated computer programs. No one is exactly sure why the Flash Crash occurred. It seems to be an example of complex computer programs acting in an unpredictable manner. We can only wonder whether some future version of the Flash Crash may bring down the financial system, or perhaps the electrical grid.

Some people are not worried about such a possibility, because they think that super-intelligent computers will fill in the gap. The idea can be stated like this:

Sure, software code will become too complex for humans to understand; but that's no problem because our ever-more-brilliant computers will be able to understand that code. Our computers will probably take over the job of writing and maintaining their own software, freeing us humans from such burdens.

But I believe we should reject the idea that computers will become smart enough to understand their own software code. Computers process information, crunch numbers, and process information. But they do not currently understand a single thing. A computer may be able to tell us instantly when Abraham Lincoln was born, but no computer has any real understanding of what a birth is, what a day is, what a human is, or who Abraham Lincoln was. There is no reason to think that any future advances will somehow give computers the understanding they now lack. A computer that does not understand anything will not suddenly be able to understand a little bit if we add some more lines of software code or some more chips or processors. Thinking that a computer will one day have understanding once you add faster processors or more lines of code to its software seems to be like thinking that one day when you get a much better TV, you'll be able to have a child fathered by your favorite TV character.

It seems, then, that we will not be saved from a software complexity crisis by computers that understand software code that has progressed beyond human understanding. A software complexity crisis will be worsened by short-sighted programming managers who demand more and more features be added to software, regardless of how this makes the code more and more difficult to maintain. We can compare such figures to real estate developers who keep yelling, “Higher, higher, higher!” to their architects, without worrying about buildings in danger of collapse because they are built too high.

The risks from such a software complexity crisis are great. Imagine it is the year 2030, and you are a typical computer programmer. Computer systems around the world may be undergoing more and more breakdowns, and your job is to fix one of them. You take a look at the software code, and see before you an ocean of unfathomable intricacy, perhaps a million lines of hard-to-read code. You ask yourself: how on earth did something like this ever come into existence? It's like the tangled jungle of complexity that is the US Tax Code, but much worse. After looking at just a little of the software, you feel like some ordinary person reading a 50-page scientific paper on quantum mechanics. You know your choice: either admit to your boss that you are hopelessly over your head, or cross your fingers and try to make some “blind fix,” rather like a layman walking into a nuclear power plant, and trying to fix rising core temperatures by fiddling with some of the dials. 

The agony of code too complex for you to understand

Then imagine such a situation happening in 10,000 different offices, to 50,000 “over their head” programmers, and you have a taste of the software complexity crisis that may lie ahead. I mentioned the possibility of the financial system or the power grid failing because of such a crisis. Another possibility is that we may upgrade nuclear weapon systems so that they are centered around computer systems that become way too complex to maintain or understand. A single fault in such a system might cause a nuclear war. The movie Fail Safe depicted such a thing when a small electronic unit failed, but the same thing might happen because of a single errant line of software code. Will some nuclear holocaust one day occur because of some computer code that grew too complex to be manageable?

The 5 Likeliest Roads to Ruin

Anders Sandberg recently wrote an essay entitled The 5 Biggest Threats to Human Existence. Since Sandberg is a Research Fellow at the Future of Humanity Institute at the University of Oxford, we might expect him to get things right. But I think only some of the five items he mentions are substantial risks to human existence. Below is Sanberg's list, along with comments on each item in the list.

#1. Nuclear war

There is no arguing with this item on the list. Some people seem to think that the threat of nuclear war ended back when the Cold War ended, but that is not true. The United States and Russia still each have about 8000 nuclear weapons. With recent tensions related to Ukraine, it sometimes seems as if the Cold War is still here. You also have to consider the threat posed by other nations with large nuclear stockpiles, such as India and Pakistan.

#2. Bioengineered Pandemic

There is also no arguing with this item on Sanberg's list. The danger of a killer super-virus run amok seems as real as ever, particularly in light of the recent discovery of some smallpox virus vials in a Maryland lab, along with the discovery that some workers at the Center for Disease Control were exposed to live anthrax virus. Both news items suggest that conditions at biological research labs are more lax than we imagine. One can only imagine how conditions are at some foreign labs that might be brewing up the next super-virus in their labs. The terrifying thing about genetically engineered viruses is that they can in theory be created in small labs that could be hidden anywhere.

#3. Superintelligence

Here is the first misfire in Sanberg's list. This is the idea that intelligent machines may take over the planet and get rid of us. But there is almost no chance that this will happen in the foreseeable future. For superintelligence to develop, Moore's Law must stay in effect for several more decades, meaning that computer hardware power and speed doubles every 18 months. But a computer chip expert recently predicted that Moore's law will not stay in effect past the year 2020, as engineers find it harder and harder to pack more processing power into a tiny space.

There is also the fact that superintelligent machines would requires software billions of times more powerful than existing computer software. Computer software development does not grow at any exponential rate comparable to Moore's law. Computer software development progresses at a much slower rate no greater than about 10% per year. Technologists have imagined that we will be able to take some gigantic shortcut to machine intelligence, by scanning the human brain, and transferring “the software of the brain” to the computer. This is fanciful wishful thinking, and there is no particularly good reason to think this will be possible any time soon.

#4. Nanotechnology

I think Sanberg also errs in putting this item on his list. If you believe Eric Drexler about nanotechnology, then we will be able to use it to achieve precise atomic manufacturing, something which would have earth-shaking results for manufacturing (and might create gigantic risks along the lines of “gray goo” involving nanotechnology run amok). But I suspect that such hopes and fears are overblown. Nobel Prize winner Richard Smalley thought that Drexler is way off the mark, that it won't be possible to ever use nanotechnology for precise atomic manufacturing, and that there is no risk of nanotechnology running amok along the lines of the “gray goo” scenario.

#5. Unknown unknowns

I guess you cannot argue with this vague item on Sanberg's list, except by saying that such an item doesn't belong because the purpose of such a list is to warn of dangers, so there's not much point in including this vague catchphrase.

What items should we list as the five greatest risks to human civilization? I would suggest the list below.

#1 Nuclear war

This items is included for the reason listed above, that there are still many thousands of these weapons in existence.

#2. Bioengineered Pandemic

This item is included because it seems all too possible that some future lab might brew up a virus far more deadly than Ebola or smallpox, and all too possible that such a virus might escape such a lab (by design or accident).

#3. Environmental ruin

Why did Sanberg omit this item from his list? Far greater than the risk of superintelligence or nanotechnology is the risk that we will make the planet uninhabitable for ourselves (or barely livable for ourselves) by our polluting activities. There is the possibility that global warming might cause various events (such as the melting of methane hydrates or excessive ocean acidification) that act as feedback loops which then create even more global warming. The result might be human extinction. For a post estimating the odds of such an extinction, see here.

#4. An attack from beyond our planet, natural or purposeful

This item (ignored by Sanberg) includes all threats from outside our planet. One such threat is the threat of a large solar flare that ruins all of our electronics, in an event similar to an electromagnetic pulse attack. Another such threat is the risk of an asteroid or comet striking our planet. If an asteroid only 20 kilometers wide hit our planet, it could be enough to kill everyone (most dying from starvation caused by a nuclear winter). Still another threat from the skies is the threat of an extraterrestrial invasion. Invading extraterrestrials might decide to wipe us out entirely and take the planet for themselves. While each of these threats is rather remote, together they add up to a significant risk. 

#5. Resource depletion

Many are worried that Peak Oil will soon occur, causing a downward spiral of civilization. If we run out of easy-to-obtain oil, the whole forward momentum of our current civilization may start to reverse, leading to a downward spiral of collapse. It may seem unthinkable that this might cause a collapse of civilization, but no one in the Roman Empire around 350 AD imagined how great a collapse would occur in the centuries ahead.

What can we do to reduce these risks? We can accelerate programs for dismantling nuclear weapons. We can tighten up international treaties on biological warfare, and protocols for inspecting labs that might create biological weapons. We can increase the modest funding of astronomical programs to monitor near-earth asteroids. What can the average person do? He or she can help reduce risks #3 and #5 by doing the same thing – conserve and reduce consumption that uses up our resources and worsens global warming. Remember, you can't really consume your way to happiness, but we just might consume our way to extinction or collapse.

Strange Social Practices of the Future

As the future carves out a new and different world for us to live in, we will no doubt change our social practices and attitudes. But in what ways will our customs and attitudes change? Below are a few possibilities.

Work Rationing

Work rationing will be what happens if society discourages people from working more than a certain number of hours per week. We could easily see work rationing if advances in automation cause a big increase in unemployment. Imagine if more and more people are losing their jobs to computers and robots. There could then be new laws that strongly discourage people from working for more than a particular number of hours per week – perhaps 40, perhaps 35, or perhaps 30.

The simplest way to enact a work rationing program would be to enact wage laws that would require not just time and a half for overtime, but double pay or triple pay for overtime. Overtime might be defined as anything more than 40 hours per week, 35 hours, or 30 hours.

Energy Rationing

The last thing anyone wants is to go back to a system like that followed during World War II, in which everyone was issued coupons that had to be produced in order to buy gas. But a practice like this may come back, if dire predictions about Peak Oil come true. The practice could be updated by giving everyone a gas card with a magnetic strip, one that would have to be produced whenever you buy gas. A more general energy rationing program would be one that also rationed air travel. Each citizen might be issued an air travel card entitling him to no more than a certain number of miles of air travel per year.

Drug Legalization

We currently have the huge problem that many millions of Baby Boomers are nearing retirement without much money saved for retirement. Many millions won't be able to afford travel or golf during their golden years. Perhaps the government may deal with the problem by encouraging drug legalization, and encouraging drug use by the elderly.

One can imagine how a cold government bureaucrat might think such a policy was good. From the government's standpoint, it is a tragedy when a young person dies from a drug overdose, because that means a loss of tax revenue. But from the government's standpoint, it is no tragedy when a very old person dies, as that saves the government costs in social security and medicare payments. We can therefore imagine a future government (with grave financial problems) encouraging not just drug use by elderly citizens, but also drug use that involved a high risk of accidental overdose. Will future state-sponsored television commercials ask: had your heroin today, Grandpa?

Robot Wives and Robot Children

One major problem that may plague the future is overpopulation. We don't see its effects all that dramatically in the United States, but in nations such as China one can see the grim effects of overpopulation in cities that are typically covered in very thick smog.

If overpopulation worsens, we may see the encouragement of novel social practices designed to minimize reproduction. We may see a social acceptance of men living with robot wives rather than real wives. We may see a social acceptance of married couples living with robot children designed as substitutes for real children. Perhaps the government might even give a free robot child to any couple who promised not to have a real child.

Vegetarianism as the Norm

Currently vegetarians are in the minority in countries such as the United States. But as global warming worsens, and people realize what a large percent of greenhouse gases are produced in order to support meat eating, then we could see a reverse of present attitudes. Meat eaters might then become an ostracized minority, somewhat like cigarette smokers are today. We can imagine a future in which meat advertisements are forbidden on television, and meat eaters have to wait to be seated in the relatively few restaurants that still serve meat.

Assisted Suicide

Currently only three US states have laws allowing physician-assisted suicide: Washington, Oregon, and Montana. But what if overpopulation problems worsen, and what if the nation starts to be bankrupted by the economic costs of supplying retirements benefits and health care for the very aged? We might then see something like vending machines that dispense suicide pills. In order to use the machine, you would have to swipe a credit card. The software in the machine would check that you are over a particular age, such as 80 or 85. The software might also be linked with a central medical computer, which might allow anyone to use the machine if that person had a diagnosis of a fatal disease.

Lawsuits About Whether Someone is Dead

In our society death is something with very significant legal ramifications. Whether you are dead may determine who is the owner of your house and other investments, and it may determine whether it is or is not the time for an insurance company to make a payout. But what if the status of your death is blurred by technology? What if you have uploaded your mind into a computer or a robot? Are you then dead, or not dead?

We can imagine relatives battling out such issues in court. One relative may argue that dear old Dad is dead, because his body has been buried; therefore, his will should now be executed. But maybe his surviving wife argues that Dad is not really dead, because he has had his mind uploaded into a computer. 

 A news story of the future

The 7 Likeliest Disasters of This Century

Let us now look at which calamities and misfortunes we are most likely to see in this century. I will create a list that steers away from fascinating apocalyptic disasters which are very unlikely to occur, and focuses on disasters that have the highest chance of occurring before the year 2100. I do not mean to argue that any of the things on my list are necessarily likely to occur in this century. I simply will look at which gigantic calamities are the most likely to occur in our century, regardless of whether the chance of their occurrence is greater than 50%.

First, I may mention a few fascinating possibilities which make great topics for “disaster porn” (ala Sci-Fi Channel),  but which are not at all likely to occur in our century:

1. An asteroid or comet collision could one day wipe out mankind, but the chance of a very bad asteroid or comet collision occurring in this century is very low. One need merely consider that we haven't had such a collision during the roughly 5000 years of recorded history, so the chance of it occurring in the remainder of this century is much less than 2%.
   
2. There is a gigantic volcanic time-bomb sitting underneath Yellowstone National Park, and scientists think the last time it erupted it pretty much buried North America in ash and dust. It could do the same thing again, as mentioned here, but thankfully the previous eruption was more than 600,000 years ago. So the chance of such a disaster occurring in our century is much less than 1%.
   
3. The widely discussed possibility of a “gray goo” disaster caused by an uncontrollable reproduction of nanotechnology is probably something that has a low chance of happening in this century. Nobel Prize winner Richard Smalley argued that there are technical reasons why he don't have to worry about this possibility.
   
4. There is very little chance that before the year 2100 robots will take over the world or make us slaves or get rid of us. While there will be fantastic leaps in robot hardware, you need equally great leaps in software before robots are a huge risk to mankind; and software is only increasing gradually rather than exponentially.
   

But what calamities have a relatively high chance of occurring in this century? Below is my list.

1. A City-Busting Hurricane

Global warming is slowly increasing the temperature of seawater, and it is warm seawater that leads to hurricanes. The National Oceanic and Atmospheric Administration says, “It is likely that greenhouse warming will cause hurricanes in the coming century to be more intense globally and have higher rainfall rates than present-day hurricanes.” 

Unfortunately, many of the world's largest cities are in danger from this possibility. Hurricanes cause storm surges that temporarily and locally raise sea levels. When you combine such surges with sea levels that may already be elevated because of global warming, the chance of devastation is very high. The devastation in New York caused by Hurricane Sandy and the similar devastation in New Orleans may be just a foretaste of far worst disasters in our century. Before 2100 we will probably see at least one major city more or less wiped out by a hurricane.

2. A City-Busting Nuclear Explosion

Despite the progress that has been made in reducing Cold-War worries about nuclear war, there is a still a very large chance that at least one city will be destroyed by a nuclear bomb before the year 2100. Even though the United States and Russia have reduced their nuclear arsenals, they each still have about 2000 active nuclear weapons. India and Pakistan each have about 100 nuclear weapons. The continued risk of nuclear war was highlighted by a recent announcement by the Bulletin of Atomic Scientists that it was keeping its Doomsday Clock (a warning of global risk) set at five minutes before midnight. Many of the US and Russian nuclear weapon systems are very old, built using technology that does not meet modern standards of reliability. The risk of an accidental launch or detonation is significant.

Besides the danger of a conventional nuclear war, there is the ongoing risk of nuclear terrorism. Outside of the world's nuclear bombs is enough highly enriched uranium and plutonium to make 40,000 additional nuclear weapons. Much of this exists in places in less than ideal security. If terrorists were to steal only a softball-sized amount of this highly enriched uranium, it would then be fairly easy for them to construct a homemade nuclear bomb. We are very lucky that no city has yet been lost in a mushroom cloud.

3. A Mega-drought

Currently California is facing the worst drought in its 163-year history. A few years ago, Texas faced a similar drought. These are probably just like tremors warning us of the earthquakes to come. This study projects future droughts using a Palmer Drought Severity Index, in which bigger droughts have a larger negative number. The worst drought ever recorded had a Palmer Drought Severity Index of -3 or -4. But the study projects that by the year 2100 some parts of the United States will see droughts with a Palmer Drought Severity Index of -8 or -10, more than twice as bad as the worst drought ever recorded. The same study forecasts that by the year 2100 some Mediterranean areas will face droughts with a Palmer Drought Severity Index in the -15 or -20 range, four or five times worse than the worst ever recorded. 

4. An Energy Crunch

By an energy crunch I mean a gap between the amount of energy the world has and the amount of energy the world needs for its current economic activities. There is a huge risk of a gigantic energy crunch in this century, because we have limited supplies of fossil fuels. It is predicted that global oil production will peak within a few decades. It is also predicted that global coal production will peak in this century, probably a few decades later than oil production will peak. The result may be a catastrophe even worse than global warming, one that leaves millions shivering or sweating in a world where electrical power is unreliable. In this study a Cal Tech scientist cites a forecast that 90% of global fossil fuels will be exhausted by 2067.

5. Mass Famine

There are four reasons why a mass famine is a large possibility before the year 2100:

1. The large possibility of terrible droughts, discussed above.
   
2. The global stressing of available water supplies, which will make it harder to irrigate crops.
   
3. The large possibility that global oil supplies will plunge long before the year 2100, because of the depletion of this limited resource. Oil is a major factor in agricultural production, being used to transport food and make fertilizers.
   
4. The fact that global population will have greatly increased, rising to as high as ten billion.
   

Add up all these things and you have a very high risk of a mass famine that may kill millions.

6. Global Depression or Economic Collapse

The global economy is something that has grown erratically, in fits and starts, with very little previous planning or design. No one ever designed the global economy to last a thousand years, and there is no reason to be very confident that it can survive something such as a gigantic energy crunch, particularly if such a disaster is combined with some of the other disasters discussed here. In particular, the global economy has a strong tendency to rely on debt, and once things start going wrong, many people lose the ability to repay debts. We saw a tremendous ripple effect caused when many people couldn't repay their debts during the 2008 economic crisis. Something similar could happen to a degree that could be ten times worse, and it bring down the whole economy like a collapsing house of cards. The result would at best be a global depression like we saw in the 1930's, and at worst a global economic collapse. The chance of such a thing seems worse when one considers that after the 2008 financial crisis, the world's nations patched on a lot of band-aids and used "extend and pretend" to muddle through, rather than making the hard choices needed to reduce the risk of similar problems in the future. 

7. A Global Pandemic

The risk of a global pandemic or plague is described in my previous How to Have a Pleasant Pandemic. The risk is very substantial because of three main factors: (1) global warming will increase the risk from certain types of diseases; (2) increased jet travel increases the chance of diseases spreading around the world; (3) there is a very substantial risk that mankind will be inflicted by some new disease deliberately cooked up in a laboratory by some malevolent party.

In regard to the third of these factors, consider that gene-analysis technology that used to cost millions of dollars is now available in desktop devices for a few thousand dollars. Technology leaps such as that may in a few decades make it so that any Tom, Dick, or Harry can whip up a new virus or bacteria in his garage, or genetically turbocharge an existing type of germ. Once that happens the risk of a global pandemic will rise much higher.

So that is my list of the seven likeliest disasters of this century. If it leaves you depressed, please read my vastly more optimistic post The 8 Luckiest Things That Could Happen in the Future.