The Birth of Stochastic Science
I have seen in Richard Dawkins' work many references to the difficulty people have, when looking at an animal, in accepting that it is not the product of a top-down design, but the result of a random process — more exactly the upper bound of a random process, in which (roughly, and only roughly) the most successful mutations tend to make it. Yet my problem is that when those who accept the evolutionary argument look at a computer, at a laser beam, at a successful drug, at a surgical technique, at the spread of a language, at the growth of a city, or at an commercial enterprise, they tend to fall for the belief that its discovery or establishment partook of some grand design. And, in hindsight, some "explanation" will be given as to why it happened: there was a plot — it could not have been an accident.
Alas, we are victims of the narrative fallacy — even in scientific research (but while we learned how to manage it in religion, and to some degree in finance, we do not seem to be aware of its prevalence in research). The pattern-seeking, causality producing machine in us blinds us with illusions of order in spite of our horrifying past forecast errors. I hold that not only discoveries are also largely the result of a random process, but that their randomness is even less tractable than, and not as simple as, biological evolution. While nature might produce milder form of stochasticity, the environment for manmade discoveries is governed by a far, far more severe, wilder form of processes, those called "fat tailed".
Against what one might expect, this makes me extremely optimistic about the future in several selective research-oriented domains, those in which there is an asymmetry in outcomes favoring the positive over the negative — like evolution. These domains thrive on randomness. The higher the uncertainty in such environments, the rosier the future — since we only select what works and discard the rest. With unplanned discoveries, you pick what's best; as with a financial option, you do not have any obligation to take what you do not like. Rigorous reasoning applies less to the planning than to the selection of what works. I also call these discoveries positive "Black Swans": you can't predict them but you know where they can come from and you know how they will affect you. My optimism in these domains comes from both the continuous increase in the rate of trial and error and the increase in uncertainty and general unpredictability.
I am convinced that the future of America is rosier than people claim — I've been hearing about its imminent decline ever since I started reading. Take the following puzzle. Whenever you hear or read a snotty European presenting his stereotypes about Americans, he will often describe them as "uncultured", "unintellectual" and "poor in math" because, unlike his peers, they are not into equation drills and the constructions middlebrows people call "high culture". Yet the person making these statements will be likely to be addicted to his Ipod, wearing t-shirts and blue jeans, and using Microsoft Word to jot down his "cultural" statements on his (Intel) PC, with some Google searches on the Internet here and there interrupting his composition. Well, it so happened that the U.S. is currently far, far more tinkering an environment than that of these nations of museum goers and equation solvers — in spite of the perceived weakness of the educational system, which allows the bottom-up uncertainty-driven trial-and-error system to govern it, whether in technology or in business.
It fosters entrepreneurs and creators, not exam takers, bureaucrats or, worse, deluded economists. So the perceived weakness of the American pupil in conventional and theoretical studies is where it very strength lies — it produces "doers", Black Swan hunting, dream-chasing entrepreneurs, or others with a tolerance for risk-taking which attracts aggressive tinkering foreigners. And globalization allowed the U.S. to specialize in the creative aspect of things, the risk-taking production of concepts and ideas, that is, the scalable and fat-tailed part of the products, and, increasingly, by exporting jobs, separate the less scalable and more linear components and assign them to someone in more mathematical and "cultural" states happy to be paid by the hour and work on other people's ideas. (I hold, against the current Adam Smith-style discourse in economics, that the American undirected free-enterprise works because it aggressively allows to capture the randomness of the environment — "cheap options"— not much because of competition and certainly less because of material incentives. Neither the followers of Adam Smith, nor to some extent, those of Karl Marx, seem to be conscious about the role of wild randomness. They are too bathed in enlightenment-style causation and cannot separate skills and payoffs.)
The world is giving us more "cheap options", and options benefit principally from uncertainty. So I am particularly optimistic about medical cures. To the dismay of many planners, there is an acceleration of the random element in medicine putting the impact of discoveries in a class of Mandelbrotian power-law style payoffs. It is compounded by another effect: exposure to serendipity. People are starting to realize that a considerable component of the gravy in medical discoveries is coming from the "fringes", people finding what they are not exactly looking for. It is not just that hypertension drugs lead to Viagra, angiogenesis drugs lead to the treatment of macular degeneration, tuberculosis drugs treat depression and Parkinson's disease, etc., but that even discoveries that we claim to come from research are themselves highly accidental, the result of tinkering narrated ex post and dressed up as design. The high rate of failure should be sufficiently convincing of the lack of effectiveness of design.
But if the success rate is very low, the more we search, the more likely we are to find things "by accident", outside the original plan — or the more an unspecified original "plan" is likely to succeed. Looking at the swelling pipeline, something tells me that the discovery of cures, or near-cures for unspecified diseases is about to happen — except that I do not know which one, nor do I know where it is coming from. More technically, I see the sign of fractal randomness in these payoffs from the fact that results are more linear to the number of investments than they are to quantities invested — thus favoring the multiplication of small bets.
All the while institutional science is largely driven by causal certainties, or the illusion of the ability to grasp these certainties; stochastic tinkering does not have easy acceptance. Yet we are increasingly learning to practice it without knowing — thanks to overconfident entrepreneurs, naive investors, greedy investment bankers, and aggressive venture capitalists brought together by the free-market system. I am also optimistic that the academy is losing its power and ability to put knowledge in straightjackets and more out-of-the-box knowledge will be generated Wiki-style. But what I am saying is not totally new. Accepting that technological improvement is an undirected (and unpredictable) stochastic process was the agenda of an almost unknown branch of Hellenic medicine in the second century Mediterranean Near East called the "empirics". Its best known practitioners were Menodotus of Nicomedia and my hero of heroes Sextus Empiricus. They advocated theory-free opinion-free trial-and-error, literally stochastic medicine. Their voices were drowned by the theoretically driven Galenic, and later Arab-Aristotelian medicine that prevailed until recently.
This idea applies to so many other technological domains. The only bad news is that we can't really tell where the good news are going to be about, except that we can locate it in specific locations, those with a high number of trials. More tinkering equals more Black Swans. Go look for the tinkerers.