Randomness, Probability, and Coincidence

Possibility lies unseen in the fabric of reality. It is possibility, and nothing more than that, which may, in fact, be the fabric of reality. Possible Universes, possible histories, possible phenomena, and possible resolutions of every kind “wait” within it. At every juncture where a decision is being made, at every point where there is a choice of futures, in every situation where a variable is at work, at every place where chains of consequence are intersecting, at every interval where uncertainty is about to be resolved, one of the most powerful and pervasive features of possibility is about to come into play: randomness. Many researchers and observers have come to the conclusion that randomness—the presence, in any given dynamic situation, of variables that will have non-predictable effects on the situation—is a ubiquitous feature of reality. (The word stochastic is sometimes used as a synonym for random.) Randomness is studied in large part through the mathematics of probability, which many people call chance. And randomness appears to be one of the “masters” of both the Universe itself and the human life which has emerged within it. It is a constant factor in the events, great and small, that comprise a human life.

These events often seem to occur out of nowhere. Seemingly impossible or unlikely things (such as a meteorite crashing into a house) actually happen. Good “luck” or bad “luck” just seems to follow some people. People sometimes hit it big in games of chance. Seemingly hopeless situations can unexpectedly resolve themselves in a way favorable to those involved in them. People’s lives are altered by chance introductions to other people. Individuals run into people they once knew and never imagined they would see again. Tragedies strike with brutal suddenness. And the story of life often seems incomprehensible at times, even senseless, littered as it is with the unexpected and the inexplicable.

To make sense of this apparent capriciousness, most humans seem to need an explanation of some sort for the events that engulf them, and for the way in which the world seems to work. A great many of these humans believe that all events have been foreordained. This belief is sometimes called a belief in Fate. Sometimes it’s called a belief in Predestination or Determinism. But the terms all mean the same thing: events happen in a certain way because they were meant to, and there was no other way they could have happened. Most humans are loath to believe that it might be otherwise. To suggest that randomness is at work in life often elicits responses ranging from despair to outright hostility. Many humans hate randomness because they want there to be a why. Especially when things are at their worst, the only comfort many humans can take is the conviction that “all is, or will be, for the best”. To invoke the apparently arbitrary element of chance into the situation is to say there is no why. And for many, such a conclusion is intolerable. For many people, the future needs to be predictable, at least in the general sense. If it isn’t, the implications can be terrifying for them.

Many humans are fearful of randomness because it robs them of their sense of being in control of things. Even worse, the existence of randomness robs them of the hope that some Higher Power (a god, Karma, ancestral spirits, and so on) is guiding life and has an Ultimate Purpose in mind for everything, a great Plan that will ultimately “balance the Universe” and make everything come out all right, however “all right” might be understood. They cling to the idea of predestination or fate because they desperately want to believe someone or something has matters under control, especially when the brutality and chaos of life are all too evident and all too near. (The belief in fate can be so all-encompassing that it erases the sense of personal responsibility; after all, if it was fated, an individual might say, “there was nothing I could have done anyway, so it’s just as well that I didn’t do anything.”) But there seems to be no getting around it: randomness is evidently real, and it would appear to be pervasive in the true sense, permeating the physical world, and upending all human expectations.

It is natural to ask a crucially important question: Is quantum indeterminacy the origin of randomness on the macroscopic level? The answer appears to be: in rare circumstances it can be, but since the behavior of subatomic particles is predictable at the group level, the structure and function of the macroscopic world seem to be generally consistent. Yet, given the Many Worlds hypothesis, it could be argued that on the broadest scale reality is being shaped by chance occurrences, namely, the random observation of an event which leads to the creation of an alternate history.

My own (amateur’s) hypothesis is that the classically-governed macroscopic world is composed of elements that behave in a quantum manner, but these quantum elements decohere in such a way that the consistency of the classical world is maintained. In other words, the uncertainty in the physical world is resolved in such a way, and at a “low” enough level, that it is not noticeable to humans. For example, it’s not how the beam of light goes from point A to point B, testing every possibility, it’s that it does go from point A to point B in a predictable manner. Randomness, therefore, chiefly lies in the classical domain. It is governed by the addition or multiplication of probabilities, combined with a sequence of “decisions”, many of them unconscious, that lead to definable outcomes. The one realm where quantum indeterminacy may play a role in randomness is, interestingly, the function of the human brain. Consciousness may have quantum elements because the neurotransmitters and neural structures upon which it is based may operate according to quantum rules.

I am not concerned here, by the way, with the mathematical quest to determine what a truly random number sequence is. I am concerned with randomness as it is found in ordinary human life, and, consequently, in the course of human history. In that course, as far as I can determine, every event is the inheritor, the descendent of so many other events, that tracing the line of causation quickly becomes impossible. When these events touch directly on the lives of conscious beings, these beings make a response of some sort to them—even if the response is to do nothing. Every event is the outcome of a series of yes/no, black/white, utterly binary “choices” that, laid on top of one another, yield a result that can only be called randomness. A thousand choices up a chain of events a simple decision was made—will we go this way or that way? From that choice sprouted innumerable others, each adding an unexpected turn. The sum total of those simple choices is the complex set of variables that produce the human perception of randomness.

Probability

Randomness is based on probability, or more accurately, a series of probabilities unfolding and manifesting themselves through time. Probability involves the sum total of things that can happen in a given situation—the situation’s universe of possibilities, or its sample space— and the frequency with which these possibilities actually occur. An event occurs because a whole sequence of probabilities played themselves out in the events that led up to it. In areas open to human study, probability estimates are the product of human record keeping, which is by no means infallible, of course.

Leonard Mlodinow, in his wonderful popular work The Drunkard’s Walk, explains the basic rules that govern probability. He summarizes them this way:

1. The probability that two events will both occur can never be greater than the probability that each will occur individually.¹

2.   If two possible events, A and B, are independent, then the probability that both A and B will occur is equal to the product of their individual probabilities. For example, if in the life of a person there is an event that has a probability of 1 in 50 over the course of a year and an event that has a probability of 1 in 5,000 over the course of a year, if they are truly independent events, the odds that both will happen to the same person in the same year is ¹/₅₀ x ¹/_5,000, which is ¹/_250,000. ²

3.  If an event can have a number of different and distinct possible outcomes, A, B, C, and so on, then the probability that either A or B will occur is equal to the sum of the individual probabilities  of A and B, and the sum of the probabilities of all the possible outcomes (A, B, C, and so on) is 1 (that is, 100 percent.)  In other words, as Mlodinow explains, when you want to know the chances that two independent events will occur, you multiply; if you want to know the chances of either of two mutually exclusive events  occurring, you add.³ If there is a 20% chance of person A appearing somewhere (a party, for example), a 25% chance of person B appearing at the same place, and a 30% chance of person C showing up, there is a 75% chance that someone from the set ABC will show up at the appointed spot. (The odds of all three showing up, however, would be only 1.5%.) Of course, as one of my mathematically-gifted friends has pointed out, in real life the odds wouldn’t necessarily add up or multiply that cleanly. A, B, and C might know each other, for example, and that might play into their decision to attend or not attend. There are conditional probability equations that are used to calculate real-life probabilities more scientifically.

The need to calculate these real-life probabilities has spawned a sophisticated subspecialty within mathematics, and the problems it deals with reveal to us the full complexity of the probability that weaves through our lives. Mathematicians working in probability theory are attempting to put into a rigorous form the work of the first people who studied the nature of chance, many of whom wrote about such matters as odds in card games and dice throws. The chief issues in probability theory include such matters as assessing what a truly independent event is, the definition and classification of random variables, determining the independence of random variables, the properties of expectation (which deals with a great many repetitions of a given phenomenon), situations containing an infinite number of variables, and Markov chains, which deal with probabilities of transitions from one state (status) to another in a finite universe of possibilities, among others. (The mathematics of these subjects I leave to those more gifted than I.)⁴

In any situation in which multiple outcomes are possible, we need to determine how many possible outcomes actually exist. For example, in a coin toss where the coin is allowed to land on the ground, the coin can either be heads or tails (assuming that it will never land on its side). No other outcomes are possible, so the universe of coin-flip possibilities is extremely small and well-delineated. But what of other situations? Let’s say we have an individual who has the financial means to travel to any part of the world he or she desires. This person is going to make a decision about his or her travel plans. How many possible outcomes are in that universe? Moreover, how many variables are at play in the actual making of the decision?

Or let’s consider a businessman who has to travel extensively during the course of a year, meeting potential clients, and sometimes winning the business of the people he meets. In that universe of possibilities there are: A. The number of stops the businessman makes in a year. B. The number of people he will encounter at those stops C. The varying emotional and intellectual receptiveness of the people he meets. D. His ability to communicate effectively with his potential clients, an ability which will be determined by such factors as his mental acuity on a given day and his relative physical health. E. The number of new clients he actually wins during the year. How many possible outcomes are in that universe, and how many of those outcomes rest on factors which cannot be predicted in advance? (And this isn’t even taking into consideration all the possible combinations of sales/non-sales on the part of the salesman.) The success rates of salespeople can be calculated, but predicting the success of any salesperson in any given year is a highly uncertain enterprise. And this situation represents an extremely small part of the sum of the human experience over the course of a single year.

The “bigger” and more complex an event is, the greater the probability of the unexpected occurring. When an event involves a great many humans doing a great many things over a large enough area over a long enough time, the odds are very, very good that some random events will occur, including those which can alter the ultimate outcome of the main event. These odds are good because the universe of possible outcomes is so huge in a “big” event. In a war the size of the Second World War, even if one could predict the probability of either an Allied or Axis victory, a seemingly straightforward binary choice,  the universe of possibilities was of such enormity that no one could have conceivably predicted the detailed course of the war, or the specific features of its outcome. There was no possible way that anyone could have foreseen the individual events of which the war was comprised, nor could they have known the outcome of specific human actions in advance. Human will had less to do with the outcome of the struggle than we might suppose.

Various areas of the Earth are governed by local probabilities. When people from these areas come into contact with those of other regions who have been shaped by different distributions of probabilities, the effects can be particularly unpredictable. Differences in the geographic settings of different cultures are a major source of these variable probabilities. Some people have been conditioned by harsh terrains or brutal climates. Others have faced more temperate conditions and far different challenges. Cultures with high population densities and heavy rates of interpersonal interaction will have different probability sets than cultures where human interaction is minimal. The point is, the action of probability is not uniform over the surface of the Earth or throughout all human populations. In the sparsely populated regions of the world the randomness of life tends to come from the natural elements that have caused the region to be thinly populated to begin with. In more populous regions, probabilities stemming from the actions of others might play a more dominant role. And it is by no means a certainty that people in sparsely populated regions are more at risk than those from heavily populated areas. In many ways, the greatest threat to a human is other humans.

Sources of Randomness in Human Life

So randomness, the expression of the probabilities that lie all around us, is the great unknown variable of existence. The sources of randomness in the human experience are:

1.   Sudden events in the natural world. These events may have a very long genesis but their actual manifestation is dramatically unexpected. Outbreaks of disease are included in this category, as are illnesses which strike an individual human unexpectedly. Other examples range from such relatively common events as wind storms, floods, lightning strikes, blizzards, cold snaps, heat waves, fires, hail storms, and the like to less common events such as earthquakes, volcanic eruptions, wild animal attacks, and landslides, to very uncommon and rare events such as strikes by meteorites or comets. The rarer the event—the greater the odds against its occurring—the more profound the shock and surprise when it does happen.

2.  Events which are caused by human volition. We might qualify this by saying “apparent human volition” because people don’t fully know the reasons for their actions, in my view. This volition may be a simple decision that carries no larger purpose (such as what to eat at a given time), it might be something a human does because of the will of another human (such as the actions of an employee), it may be something done in the course of ordinary social interaction, it may be something that a human wishes to do to benefit another human, or it may be completely malicious in nature. It might have elements of several of these motives. Most significantly, human volition itself stems from the unique, randomly expressed variables that make up the consciousness of the individual. No expression of human volition is identical to any other. This category is closely related to but does not entirely coincide with…

3.  Events which are caused by human miscalculation or error. This is the source of an enormous amount of randomness in human life. Human misperception—the false interpretation, by the relevant portions of the cerebral cortex, of stimuli coming into the senses—has been the source of countless random errors. People do not necessarily see what they think they see. They do not necessarily hear what they think they hear. They miscalculate distances, underestimate times before a collision, or make other errors of prediction based on misperceived stimuli. The failure of humans to anticipate negative consequences is a relative of this misperception. In failure of anticipation, sensory data may be accurately perceived, but the next step—action based on this perception—is not consistent with the sensory data received. The role of fatigue and intoxication in causing human error must also be considered, as well as errors caused by damage to the brain. The story of human history is so rife with error that no one can know its full impact. The accidents caused solely by human mistakes, as opposed to mechanical failures or natural interventions, are quite literally innumerable. Human mistakes are the great “wild card” of human history.

4.   Events that occur gradually over long periods of time. These are events that were not “willed” by anyone, nor are they the product of human incompetence. Neither are they sudden eruptions of nature. One sub-category of such events is the physical breakdown of man-made objects, a naturally-caused event less dramatic than those typical of the first category. Eventually, a machine simply wears out from the friction involved in its use. Metal becomes brittle. A house needs repair because of exposure to the elements. Such events, happening over long periods of time, can cause accidents, but more often than not such deterioration results in less drastic outcomes. Another subcategory is the slower processes of nature, such as erosion (caused by wind, blowing sand, or water), the shifting of tectonic plates, gradual alterations in the Earth’s climate, slow changes in the Earth’s surface features, the processes of natural selection that bring about changes in the genetic composition of a population in a given area, and other such examples. The sum total of the unpredictable changes brought about by these processes can be dramatic in the extreme. And a third subcategory is actually the hardest one to trace: gradual changes in individual humans, or in the attitudes of groups of humans, or in the interaction of human societies. These kinds of changes, totally unremarkable in most respects, can still manifest themselves in profoundly unpredictable and important ways. The cumulative impact on the direction of this world’s existence brought about by gradual processes  is enormous.

5. The action of the truly unpredictable. There are events which are so utterly unanticipated that they are what Nassim Nicholas Taleb calls Black Swans. Black Swans appear out of “nowhere” in the sense that most people believed they could not happen and made no preparation for them. And oftentimes, they are so unprecedented and unusual that no one could have predicted them. Taleb presents persuasive evidence that humans tend to grossly overestimate their control over affairs and their understanding of reality, and get surprised by Black Swans more often than they would care to admit.⁵

Arthur C. Clarke characterized the surprise and disorientation brought about by the discovery of unexpected scientific principles as failure of imagination—the (understandable) inability to foresee the radically unpredictable. Examples of such principles would be those which underlie the existence of x-rays, radiation, and nuclear energy. No observer, however scientifically sophisticated, predicted such phenomena would be uncovered because there was no way of doing so. By their very nature these principles were inconceivable, and their discovery constituted a highly random set of developments in the sciences.⁶

6.   Events which involve interrelated and interconnected aspects of all or some of these sources, acting in unpredictable combination. This is perhaps the most important category of all. Many, many variables operate simultaneously in reality. It is beyond human abilities to know all the individual probabilities of all the variables in a given event. Since the essence of the definition of randomness is unpredictability, this lack of knowledge is the basis of the apparently random nature of things, and manifests itself as the perception of randomness. The inability of humans to comprehend these often blindingly complex combinations limits their ability to take control of situations, or even to take effective measures for their own protection. These unpredictable combinations of random events are also the source of much of the astonishment humans feel as the unexpected plays out around them.

Many humans are surprised by unusual events because they overlook what experts in probability refer to as The Law of Large Numbers. In its essence, it states that if enough repetitions of a given event occur, then sooner or later every possible outcome will manifest itself in a statistically predictable way. This was first proved in 1713 by Swiss mathematician James Bernoulli and it has since been confirmed and its expression made more succinct. Less precisely known as The Law of Averages, it means that everything that can happen will happen, given sufficient opportunity.⁷ There are many long-shots coming in all the time, there are many of what I call clusters of probability expressing themselves at any given moment. In a human population of 7 billion, statistically improbable things occur every day. They are not miraculous in nature; they are occurring simply because they had enough chances to occur.

Humans generally (not universally) wish to be prepared for unexpected contingencies. In much of human society, security is defined by the ability to defend one’s self and one’s loved ones from the effects of randomness. (A friend and former colleague of mine has said that the human quest for security is largely based on the desire to find a safe place to sleep.) The acquisition and keeping of weapons is one of the oldest examples of the desire to be protected from randomness. The much more recent rise of insurance in the more advanced nations is another example of this desire. The nature of the complex societies that humans have evolved, with their governments, law codes, military establishments, and police forces, can all be seen, in one perspective, as an attempt to increase the predictability of existence. A whole statistical discipline has arisen based on the mathematical calculation of risk, a discipline which serves insurers, gaming interests, and those engaged in finance. For such people, the calculation of odds is vital to their economic survival.

The attempt to arrange one’s life in such a way as to avoid chance encounters with disaster is made harder by poverty, which exposes a human to the effects of bad probabilities to a far greater extent than those who are more affluent. Impoverished humans live closer both to the natural elements and to others who are in similar circumstances. They are exposed, therefore, to a high degree of both natural and human capriciousness. The social arrangements of a society might therefore be best understood by the distribution of exposure to randomness within the society’s population. But even for the most prosperous, the quest for absolute security is a futile one.

Coincidence

Coincidence can be understood as the simultaneous (or near simultaneous) occurrence, through the action of probability alone, of two or more events which bear similarities to each other or which  seem to be related. Coincidences occur because various intersecting chains of unexpected consequences occasionally produce very similar outcomes at about the same point in space-time. Because many humans have a poor grasp of probability (and numbers in general), they tend to see certain events that are merely coincidental as extraordinary. As John Allen Paulos, a professor of mathematics has said, a million to one shot comes in hundreds of times every day. Ordinary examples from human life abound:

--We are thinking of someone and at the moment we are doing so we get a phone call from that individual or we meet them on the street.

--We are reading about a situation in a distant country, and someone within earshot of us happens to mention that country.

--We happen to be a little short of the change we need for a purchase and we see a coin of the exact denomination we need lying unclaimed on the floor of the shop.

--We have a vivid dream about a disaster occurring, and a disaster very similar to that about which we dreamed actually occurs.

--We happen to be thinking of an elderly relative moments before we receive the news that that relative has died.

--We are playing a dice game. We perform a small ritual to “guarantee” that a seven will be rolled, and indeed we roll a seven.

Besides being the products of simple probability, what do such events have in common? Many people tend to see in them evidence that the Universe is mysteriously bringing events together specifically to convey meaning to us. We often hear the expression, “That was no accident.” Well, in almost every imaginable case, such events are accidents—the accidents of coincidence. ⁸

Many people claim—completely without logic—that events that have no significance, such as odd little coincidences, are deeply important, in part because these occurrences are so surprising and so unexpected, and in part because they fail to understand that causation is statistically explicable. Many people seem to reject the notion that  probability can touch their own lives. They insist that there must be “good causes” for everything. But just because a book shelf collapses when we are talking about the possibility of a book shelf collapsing, it does not mean that our mentioning of the subject caused it to occur in real life. All of this misinterpretation of coincidences is a symptom of apophenia, the belief that events and phenomena that have little or no real significance are tremendously important. All “omens”, all “prophetic signals”, all “ill winds”, and all “unlucky signs” are examples of this—and so is the conviction that coincidences are intrinsically meaningful. Again, ignorance of the Law of Large Numbers blinds people to the fact that the amazing is actually rather commonplace.

Some people believe in an alleged phenomenon called synchronicity, which was conceived by the psychoanalyst Carl Jung. From The Skeptic’s Dictionary:

His notion of synchronicity is that there is an acausal principle that links events having a similar meaning by their coincidence in time rather than sequentially. He claimed that there is a synchrony between the mind and the phenomenal world of perception.⁹

Furthermore, Jung felt himself capable of “interpreting” these “meaningful coincidences” through sheer intuition. There is no empirical evidence for any of this at all. In fact, Jung believed many bizarre things, and may have been afflicted with mental illness himself. 

Why do people want to believe in the significance of coincidences? Why do they focus on the seeming “success” of  systems designed to “beat the odds”? Why do they fall for scams based on human ignorance of probability? Paulos explains that many humans have a sort of mental filtering mechanism that emphasizes successes and passes over defeats. We remember the time we hit it big on a bet; we forget all the losses we incurred before and after it. In a similar fashion, humans remember all the “hits” of coincidence, forgetting all the thousands and thousands of times such seemingly extraordinary events did not occur.¹⁰

Is the world explicable by algorithmic means, or is reality essentially non-algorithmic? In many cases, we can know outcomes in the aggregate; we cannot know them in the particular, much as we can know the behavior of subatomic particles as a group but not the behavior of individual ones. But subatomic particles are much more predictable than the bearers of consciousness, for whom a number of behavioral options are open. It is the evolution of consciousness that in many ways has added a new layer of uncertainty to reality. The bearers of consciousness are affected by the apparent randomness in which they find themselves, but they contribute to this randomness by their very nature. They are much less in control of their lives than they would like to admit, and the hidden realities of randomness and probability dog them at every turn. But in crucial ways, the most dangerous aspects of this randomness come from their fellow humans, the actions of whom over the centuries have helped create the very uncertainties that humans most fear. Humans have helped unleash sequences of events that have played themselves out in wildly unpredictable ways across both space and time. It is to these sequences that we now turn in order to further understand the unseen realities in which we are immersed—and against which we so often struggle in vain