Are markets really random or is randomness just an artifact of methods used to study markets?
In this post, I will show why you can’t know that markets are random. Then, I will attempt to present randomness in a framework for understanding the entire universe.
Let’s get started.
What Do We Mean by “Random”?
Let’s call something random if it does not follow an intelligible pattern and events cannot be predicted or known.
You Can’t Know Markets are Random
To the degree that the stock market is random, nothing can be known about its future behavior. How then, could we possibly know that its random? The statement “the stock market is random” is a statement of fact about a system, that directly contradicts itself: if the statement is true, then the statement itself cannot be known because we cannot know what is random.
“The stock market is random” implies that the truthfulness of the statement “The stock market is random” cannot be known.
Markets might seem random because they are highly complex, adaptive systems with second-order chaos controlled by millions of evolving participants with different goals, ideas and strategies. Since a certain prediction of future randomness is also uncertain, we don’t really know if the stock market is random.
Randomness Is Not Intrinsic
Randomness is not an intrinsic property of markets (or any process for that matter), but rather a relative expression of uncertainty. A fair coin flip is not innately random— it is only random to me because of my (time-relative) uncertainty about the outcome.
Thinking of randomness as an inherent quality of something is a form of magical thinking— a coin flip will be decided by how hard I flick it up, where it lands, how it bounces, air pressure, etc. Attributing randomness as the cause of events is no different from attributing causation to invisible magical randomness-fairies that dance mysteriously behind the scenes. Calling things random is mistaking your perception/model for reality.
Just like Nature, True Randomness cannot be known intellectually, because randomness will always transcend our understanding (otherwise it would no longer be random). In the figure, the orange circle (True Randomness) is an absolute (infinitely large) that contains relative measures of randomness inside of it. The relative randomness of you perceiving a dice throw or observing the markets exists within the infinite randomness of the Universe.
Randomness is a convenient linguistic/mathematical crutch that all-too-easily dismisses the unknown as the unknowable.
Isn’t this just armchair philosophy?
If randomness is just an expression of our uncertainty and no-one has perfect clarity of the future, then doesn’t that mean that markets are still random if no-one can outperform them? Even if we can’t know for sure if markets are random, this doesn’t exclude the possibility that they are still random, right?
While markets are not knowably random in the absolute sense, markets events do appear random to even the most informed investors. From the author’s perspective, the market is mostly random (not entirely!). This relative randomness, however, is a primary feature caused by the competition in markets.
Since correctly predicting the future can result in big $$, markets provide strong financial incentives to correctly predict the future. Consequently, making money from predicting the markets makes the market even more unpredictable by correcting for the informational redundancy that lead to its predictability in the first place. Buying an “underpriced” asset will cause the price to go up and selling an “overpriced” asset will cause the price to go down (supply and demand). Just like humans, markets can be emotional and biased by misjudgements/ignorance. Price is a consensus-driven proxy for the imaginary thing called “value.”
From this perspective, randomness is not a philosophical musing but rather the result of an essential feature: the market’s predictability ultimately leads to its unpredictability.
The way you think of the markets affects the way you interact with it, consciously and subconsciously. The market is acting as a consensus machine trying to value an asset— there’s nothing random about a consensus, only our own (not necessarily accurate) uncertainty about how others will behave, and more importantly how we will behave. Being aware of the conceptual nature of randomness liberates you from the limited perspective of the mathematical tools that produce randomness and creates space for good judgement and intelligent decisions.
Helium uses probability estimates to predict stock movements, however given the unknown nature of counterfactuals in markets (“potential” events that did not happen), there is no way to know if an individual probability estimate was accurate because there’s only one data point. At best, we can calibrate our predictions over multiple predictions to see how well we do over time. For this calibration, look at the Helium scores.
Luck vs Risk
Suppose I make a highly risky bet that Boeing will go up tomorrow because I saw a cool movie last night where airplanes go ZOOM.
Lo and behold, the next day Boeing rises by 5%, presumable from some boring new financial contract or whatever. Amazing, I’m rich! Was my bet skill or luck? Obviously this must be a lucky bet because the reason for my success had no bearing to my original hypothesis.
If my bet had a 90% chance of failure (based on past occurrences), then clearly this was a lucky break. However, was my bet not skillful in its ability to capitalize on the potential volatility of a (random from my perspective) price move? Was putting myself in a position to get lucky from a relatively unlikely event a display of skill or hubris? If I can’t go back in time, wasn’t this outcome always going to happen?
Risk and luck are two sides of the same coin. Risk is just future uncertainty that I don’t want, and luck is future uncertainty that I do want. Finding a $100 bill on the ground is luck. Getting pooped on by a bird is risk. Who ultimately decides who is unlucky and who is lucky? We do.
Even though future prices are quite random to most people (despite many advertising pitches to the contrary), different market participants can have different amounts of knowledge, and thus different levels of uncertainty. A deep analysis of trends, psychology, trading, and business can (not always) lead to a small decrease in uncertainty about the future because future market prices do (not perfectly) depend on consumer/business trends, market cycles, and events in the past and events happening right now. While it is difficult to predict the future (especially far out), saying that it’s impossible is pessimistic by ignoring the remarkable ability of human insight, imagination, and reasoning.
Markets aren’t “random,” but rather a function of the collective participation of people/institutions who themselves have different perspectives, motives, and uncertainties regarding future prices. It is a mistake to conflate the difficulty of forecasting with the impossibility of forecasting. Markets aren’t random, but people are complex.
What about black swans?
Certainly black swan events (highly impactful, rare events) must be random, right?
Just like markets, events are not random in and of themselves. A black swan is just an event with significant ramifications that most (not all) market participants were not able to foresee. Hidden/tail risks may appear random, however, randomness is just a reflection of our ignorance to the highly complex tree of events that caused the event. From a linear time perspective, every past event was always going to happen and every future event (while currently undetermined) will always happen.
What about second-order uncertainty? How do I know whether my uncertainty regarding the market is correct and not subject to under/over-confidence?
Unfortunately, I don’t think the answer to this question can be known, because the problem keeps regressing: what about my uncertainty about my uncertainty about my uncertainty? This groundless, infinitely recursive problem of uncertainty about uncertainty, when taken to its logical conclusion (since uncertainty about uncertainty cannot increase our total certainty) seems to result in total and complete uncertainty: ultimately, we know nothing.
This perplexing result, however, eats its own tail: since knowing that we know nothing is, in fact, knowing something, it contradicts us knowing nothing. If we know something, then we know nothing about something. It appears as if we know nothing and something at the same time. If we know something, we know that we cannot know that we know it. If we do not know something, we cannot know that we do not know it.
Our direct conscious experience is an interesting example: empirically, we do not know what is causing our existence to come into being, however I cannot know that I don’t, in fact, already know the cause of my own existence.