The story had paused for more than two thousand years and with a surprise discovery was then suddenly back in play. In the 1940s and 1950s a sad mismatch of incentives after the discovery of the Dead Sea Scrolls led to the destruction of parts of the ancient biblical documents. That destruction was something that no one wanted and yet, with these priceless historical items, it was logical. Why?
A theme of my writing is that as we fail to recognize how a system pushes us into certain actions we fail to predict the resulting second-order effects.
In many cases, we even have incentives to ignore how our actions might create these effects.
What can we do about this? We can start by looking at examples of incentives creating a situation for unintended consequences.
The types of incentives I’ll look at in this post come from seeking safety improvements, pride and glory, and financial upside.
Safety Incentives
One category of motivating incentives is safety: real or perceived. Seeking safety means that people may be willing to spend money, or give up something else in pursuit of that additional safety, whether for themselves or others. Here are some examples of how seeking safety went (or could go) differently than expected.
Hospital report cards. In this scenario, hospitals choose which patients to serve based on likely impact on the hospital quality “report card” metric. The desired outcome is a quality rating that reflects how safe the hospital is.
The result is that hospitals sometimes focus care on healthier patients in order to improve the measure of patient outcomes. Less ill patients are more likely to fully recover. Sicker ones are more likely to have complications and therefore lower the measure of patient outcomes. Hospital “report cards led to a shift in the incidence of CABG [coronary artery bypass graft] surgery towards healthier patients” (from The Effects of Report Cards on Health Care Providers).
Autonomous national missile defense system. This is a concept that many nations are working toward in many forms and described as a subcategory of lethal autonomous weapon systems (LAWS). The appeal for these weapons is two-fold. First, there is the argument that human lives can be saved by having the autonomous weapons fight other. Then there is the argument that it is inevitable that this technology will be developed and therefore it is logical that each nation work to develop or acquire it. The UN, Holy See, and other groups call for banning LAWS but some countries, including the US, won’t sign. Since LAWS would be self-governing, while they may respond more quickly than humans, they may also respond in unintended ways.
Eradicate or genetically modify mosquitoes. I have written about this issue a couple times, both here and for TechCrunch. Many international groups are involved in studying the possibility to reduce or eliminate human deaths from mosquito borne diseases but they usually fail to consider what happens when things don’t go according to plan. Problems include the growth of cross-species diseases in places where humans move now that the mosquitoes are gone. We also recently learned that genetically modified mosquitoes have transferred their genes to the natural population.
Pride and Glory Incentives
A second category of incentives I list is centered around the pride and glory of those who act. Some examples:
Seeding the upper atmosphere with nanoparticles to cool the earth. I include this in the pride and glory category since the upside to an individual or group funding such a deployment may come from appearing to be the savior of the world. I take a cynical view here since these proposals come from individuals or small groups. Any individual influential enough to promote this technology is rich enough to move wherever they want to avoid potential effects of climate change. Rather than promote green energy, reductions in fossil fuel use, or other conservatory actions, the idea is to maintain growth and restrict some of the sunlight that reaches the surface of the earth. (I’m ignoring the other option of action by an institution or country that would financially benefit from lower temperatures.)
As a top-down and possibly irreversible alteration to the atmosphere, we should be super careful with proposals like this. If the models that predict results are (as usual) imperfect, how do you reverse the impact? How do you hold someone accountable for a bad implementation?
Universities grow to depend on financial donations. University administrators who receive these donations often do earn more, but not to any degree similar to what they would earn working in private industry. Most of the administrators’ personal value seems to be captured in the pride of raising the money and the glory of funding a bigger department. A recent example of this is the MIT Media Lab’s sourcing of donations from Jeffrey Epstein, even after he was noted to be a sex offender. The Media Lab kept him as an anonymous donor (though they still emailed with him openly) to satisfy the letter of the law.
Universities are in a fund-raising arms race and individuals have incentives to prioritize donations over ethics. Normal human behavior.
Compete using performance enhancing drugs. Elite athletes find themselves in a bind. Certain sports are high endurance and ultra competitive and have actually increased in competitiveness over the years. In some instances, athletes reason that it is worth taking performance-enhancing drugs and taking the risk of eventual exposure. That is, increase your odds to win. Or, create your only chance to win. In some cases, it can make sense to take performance-enhancing drugs and take the risk of future discovery.
Financial Incentives
Companies have little incentive to consider second-order effects of their actions, except perhaps where they may be the recipients of these effects.
A recurring theme behind company incentives is that a specific financial goal can influence people differently than intended (if there was intention). Consider the example below, where workers met the corporate goal but caused other problems.
“Sears set sales goals for its auto repair staff of $147/hour. This specific, challenging goal prompted staff to overcharge for work and to complete unnecessary repairs on a company-wide basis. Ultimately, Sears’ Chairman Edward Brennan acknowledged that goal setting had motivated Sears’ employees to deceive customers.” — Goals Gone Wild
The other problems were the creation of unethical staff behavior and overcharged customers. Short-term win, but long-term failure. Let’s look at a few examples of this type of incentive.
Pay loan originators and traders (drivers of revenue) for aggressive revenue growth. Offer loans to people unable to pay them back in order to meet origination quotas. Make trades that pay bonuses at year end but that cannot be clawed back. When the company’s financial positions explode, lobby for a public bail out.
Create pain killers with addictive properties. With the pain killer Oxycontin, drug companies set dosages to draw out patient craving and ultimately addiction. Sales reps educated doctors on the desired 12-hour dosing and “pill mills” emerged for easy prescriptions. The Sackler family behind Purdue Pharma’s Oxycontin has been found to have moved $1B in assets overseas, possibly in an attempt to avoid losing drug profits following legal proceedings. It’s an addiction-based business model that skewed corporate behavior.
Lower subscription service price well below the cost of service. In a well-known corporate example, movie-subscription company MoviePass lowered their pricing to $10/month for one movie in a theater every day (30 movies a month — well above $10 in costs). That incredible offer was influenced by a $2 million bonus to be paid if the company grew subscribers to 150,000 within 18 months. (It took two days.) Never mind what it did to the company. MoviePass shut down last week.
Pay people to find and turn in parts of the Dead Sea Scrolls. Well, you’ll see below…
A Timing Mismatch
The above incentives live within systems that didn’t link short-term goals with long-term outcomes. If these systems could make that linkage, they may have produced less extreme unintended consequences. The system would have a check on itself instead of being pulled in one extreme direction to pursue an incomplete goal.
In the above athletic performance incentive to cheat there is no easy way to reverse the awards won. Second place athletes (assuming they did not dope also) cannot truly take the old winner’s place and benefit from the pride and glory or financial rewards that they would have claimed. That time is past. Same for the loan originators and traders, those who designed hospital report cards, and depending on outcome, those who propose seeding the atmosphere with nanoparticles.
Back to those scrolls. The discovery of the Dead Sea Scrolls soon created a new market. However, the archaeologists who wanted the scrolls paid Bedouins, who had access to caves around the Dead Sea, by the piece.
The natural outcome?
In “the very first days, the Bedouin would tear the scrolls in pieces to get a higher price, because at that time they were being paid by the piece. Only later on de Vaux [one of the archaeologists] changed the rate of payment to the square centimeter price, in order to keep the Bedouin from tearing them up.”
“Also, to the same end of keeping large fragments intact, a baksheesh [bonus] was paid beyond the surface measurement for unusually large pieces.”
Another archaeologist reported that a person who grew up in the area “told him that ‘as a boy he used to find similar scraps of leather in caves and burn them, since he knew nothing about their value.'” – The Dead Sea Scrolls: A Full History
Again, the natural outcome.
Consider
- In a big decision, work to understand who has an incentive to act in a specific way. A person’s incentives are not purely financial, so think more broadly (the safety and pride and glory categories from above).
- It’s difficult to know enough pieces of a system and where incentives may act.
- Do decision trees that help estimate the risk and set the value of unintended consequences include a veto option? For decisions with large top-down and irreversible risk, the default answer should be to veto.
- Misunderstood incentives can destroy more value than they create.
