Writing about a range of unintended consequences from a single company is a new format for me. I typically have either looked more widely at effects from a new technology, policy, or event. But many companies create or house second-order effects.
As a start, let’s look at Amazon. Amazon was an easy choice to start with since the company is big ($321 billion in revenue in the past 12 months), has multiple business lines, and is a mix of software, hardware, and people businesses.
This article is long and it really only scratches the surface of some of Amazon’s unintended consequences.
Alexa
Over 100 million devices sold.
Amazon’s Alexa smart speaker has changed the way people gain information while going about their day. Alexa customers use hands-free voice as the input method to place orders, ask questions, play entertainment, and more.
For that convenience, Alexa users may find unintentional purchases, including multiple toilet paper orders initiated by people streaming a preacher’s service. Similar events have happened at other times, including unintentional cat food orders and a (somewhat intentional) doll house.
“Accidental” recordings have also happened, for example when a couple’s conversations were recorded and sent to someone on their contact list.
But these incidents seem minor compared to the next examples, which are more about speculative behavior change.
Similar to the way we might speak differently to someone who doesn’t understand our language well, do people adjust the ways they speak to a smart device? Using sarcasm, unusual words, uncommon terms, highly contextual terms — these all create more opportunity to misunderstand. But that risk of misunderstanding may be acceptable when we’re having a conversation with someone we know and like. When it’s a functional task, maybe less so. Also, given that Alexa is a female assistant, does that bring out different behaviors or train different behavior in its users when speaking to human females?
When people become frustrated with Alexa, or are just frustrated because they are having a bad day, Alexa responds with an apologetic voice.
This is called “frustration detection” by Amazon’s Alexa technology group. Frustration detection in other forms has been around for a long time, of course. The original frustration detection is people responding differently to another person’s frustration. Sometimes in an apologetic way and sometimes in an aggressive way depending on the relationship and personality of the respondent.
Less common frustration detection takes place during a customer service exchange. There, a frustrated customer will often (but not always) be treated with deference. The frustration that a customer expresses in a public forum also tends to be more controlled than what they would express in their own home.
As Amazon writes on its blog:
“As customers continue to use Alexa more often, they want her to be more conversational and can get frustrated when Alexa gets something wrong. To help with this, we developed a deep learning model to detect when customers are frustrated, not with the world around them, but with Alexa. And when she recognizes you’re frustrated with her, Alexa can now try to adjust, just like you or I would do.”
Does this change people when these exchanges are with a machine rather than a human?
Researchers have studied Alexa’s responses (as well as those of other voice assistants) for a while. This 2017 article gives a good breakdown of what was found related to “device harassment” back then.
Will being able to talk naturally, including with frustration, impact the way people — and especially observing kids — learn to interact with others? This is what some researchers wonder. I don’t believe that anyone knows the answer.
Rough estimates are that one-quarter of US households own a smart speaker, of which Alexa is the most common device. At 128 million US households, even a small impact on the one-quarter of them owning smart speakers can make a big difference in outcomes. As with many of Amazon’s unintended consequences, size alone accounts for a lot of the potential impact.
- Causes of the unintended consequences: Alexa interacts with and adjusts to abuse.
- Counteractions: Program Alexa to ignore, push back, or misunderstand the abuse? Do we even understand the effects in order to know how to respond?
Investing
The investment arm has a parent company.
The Amazon Alexa Fund raised $200 million for investing to “fuel voice technology innovation.” The fund invested in 91 startups to date.
As a driver of voice-enabled hardware, Amazon has beneficially supported startups that build on top of its own Alexa hardware. By pushing more investment into voice-related innovation, Amazon benefits and also gets to learn what others are doing in this space.
Amazon’s investment arm is different from other VCs in that its parent company is also potentially a competitor to its portfolio companies.
Related to that, Amazon’s investment arm has been accused of benefiting from and copying the work of startups that pitched to it or that it invested in.
The following is a series of quotes about this issue, from the Wall Street Journal:
“In some cases, Amazon’s decision to launch a competing product devastated the business in which it invested. In other cases, it met with startups about potential takeovers, sought to understand how their technology works, then declined to invest and later introduced similar Amazon-branded products, according to some of the entrepreneurs and investors.”
“In 2010, Amazon invested in daily-deals website LivingSocial, gaining a 30% stake and representation on the startup’s board. Former LivingSocial executives said Amazon began requesting data. ‘They asked for our customer list, merchant list, sales data. They had a competitive product and they demanded all of this,’ said one former executive. LivingSocial declined to hand over the data, this person said.”
“LivingSocial executives began hearing from clients that Amazon was contacting them directly and offering them better terms, some former executives said. Amazon also began hiring away LivingSocial employees.”
“‘They are using market forces in a really Machiavellian way,’ said Jeremy Levine, a partner at venture-capital firm Bessemer Venture Partners. ‘It’s like they are not in any way, shape or form the proverbial wolf in sheep’s clothing. They are a wolf in wolf’s clothing.'”
I have known of investors — typically in small markets where founders have few options — to use predatory techniques like this. With any corporate investing arm, startups seeking investment should be a bit more careful. But even with no intent, Amazon is large enough that issues like this might just happen because the company is so large it’s hard to know what’s going on in every corner of it.
In June 2020, Amazon announced that it is building a $2 billion fund to invest in clean energy. The Climate Pledge Fund, will invest in companies building transportation, energy generation, battery storage, manufacturing and food and agriculture innovations. This might be the next group of startups affected.
- Causes of the unintended consequences: Competitive nature of the parent company. Using pitch meetings as intelligence gathering.
- Counteractions: More care by founders in who to pitch and what information to reveal.
AWS
AWS is the most common cloud infrastructure service.
Amazon Web Services (AWS) is the most common cloud infrastructure providers, at 33% market share. That means that many companies use AWS for hosting company data, sensitive documents, customer credentials, bank statements, correspondence, logs, code, and more.
The pervasiveness of AWS means that engineers have a natural benefit to knowing how to use AWS, as opposed to, say, Microsoft’s Azure or Google Cloud. But that pervasiveness also means that many data breaches happen on AWS as well.
AWS-related data breaches include exposed data by a travel company (500,000 customer transactions exposed), by Accenture (“authentication credentials, certificates, decryption keys, customer information” exposed), and Dow Jones (2.2 million customer records exposed). More interestingly, there have also been breaches by departments of the military and their contractors (top secret information exposed).
Data breaches also come from Amazon engineers themselves (passwords and AWS key pairs exposed).
- Causes of the unintended consequences: Many employees and contractors with customer data access and authorization to edit entitlements to that data. Pervasiveness of use.
- Counteractions: Better management of a complicated process.
Marketplace Product Reviews
Amazon is the default search engine for products.
Through Amazon Marketplace, third-party sellers have grown to account for over half of items sold since 2017. When just about anyone can sell on Amazon it is necessary to judge quality communally. But that can lead to second-order effects.
One way that sellers give themselves an edge is to manipulate the reviews of their competitors. Examples of this manipulation are the direct: buying a competitor’s product and giving it a low review but also the more sophisticated: giving fake high reviews for competitors to later report them for review manipulation.
Types of attack include paying people to write five-star reviews for a competitor in order to get the product flagged, hijacking a competitor’s brand, gaining illegal access to competitors’ products through illicit access to a distributor’s account, and even buying a competitor’s products only to set them on fire, post pictures, and then complaining about the danger.
One of the reasons that Amazon could grow so large is that they enable third-party sellers in their market. But with an organization this large, that touches so many buyers and sellers, there is a necessary loss of individual control in favor of algorithmic or decision-tree approaches.
Another version of the fake reviews is to create a customer, when none exists, purely for the ability to give an item a rating. This method has sellers (sometimes international) send unordered items to US-based addresses in order to rate the products themselves. Such is the importance of product reviews that shipping free products can be worth it.
Additional consequences come from the way Amazon treats brands as opposed to sellers. A brand can still be an individual seller, but one who has registered with Amazon. Brand registration provides extra capabilities to remove someone they claim is infringing on their business. This can be the result even if the claim is made up.
Even without bad intent, Amazon’s algorithm can interpret genuine customer reviews differently than expected, including dinging sellers receiving the comment that the shoes they sold were “not as described” when they were just too small. Remedy: add instructions “recommending customers wear thin socks.”
- Causes of the unintended consequences: On the default search engine for products, sellers win by looking comparably better than the alternatives (not by being comparably better). Lack of personal relationship with the platform, which is not possible at scale. Amazon pushes the losses to sellers (financial) and buyers (time). As long as the problem doesn’t get too big, it’s tough to stop it.
- Counteractions: Sellers need to develop direct customer relationships.
Ecommerce Taxation
Governments don’t think of unintended consequences.
Perhaps a predictable consequence of changed tax policy is that when the EU increased taxes levied on tech companies, those companies simply passed the tax on.
Amazon paid £293 million in tax in the UK in 2019, with revenues of £13.73 billion. In 2018, Amazon paid £220 million in taxes in the UK with revenues in the UK of £10.9 billion. And this in turn is also an increase over the results from 2017, when Amazon paid just £4.5 million in UK taxes, on revenues of £8.7 billion. But you get the point of government frustration about not being able to capture tax revenues.
From Amazon’s Seller Services Forum – K Announcements:
“Earlier this spring, the UK government introduced a Digital Services Tax (“DST”). While the legislation was being passed, and as we continued our discussions with the government to encourage them to take an approach that would not impact our selling partners, we absorbed this increase.
“Now that the legislation has passed, we want to inform you that we will be increasing Referral fees, Fulfilment by Amazon (FBA) fees, monthly FBA storage fees and Multichannel Fulfilment (MCF) fees by 2% in the UK to reflect this additional cost. We will not apply the increased charges retroactively, but starting 1 September 2020, the fee types listed above will increase.”
From a forum commenter called British_Sports_Muse1:
“This tax was brought in to try to make companies like Amazon, who earn huge profits while contributing very little tax in the UK, pay what is due. They were not designed to be passed on to the sellers, who are mainly small businesses, who already have to pay 20% in VAT plus corporation tax in the UK. This is outrageous and shows the contempt that Amazon has for the UK and it’s marketplace sellers.”
- Causes of the unintended consequences: Government lack of second-order thinking.
- Counteractions: No good ones?
Logistics
Prime has 150 million members.
The last time I lived in a college dorm, getting mail was uncommon. A package, requiring pickup at the mail desk, rarely happened. This changed with Amazon Prime, discounted for students, as college mailrooms, never outfitted for many package deliveries, struggled to keep up.
Amazon deliveries affected college bookstores in a contrary way. Some of the beginning of the semester rush to buy books has now shifted to online purchases with deliveries.
Beyond college campuses, items once bought in person — or more often not bought at all — are now delivered. How does Amazon manage this? Through building a far-reaching series of logistics centers, delivery services and relationships, and more.
But in a different way than other delivery companies we’re used to seeing. For example, there is greater regulatory oversight to companies like UPS and FedEx than Amazon.
“Both [UPS and FedEx] are also heavily regulated by the government, and many of their trucks are subject to regular federal safety inspections and can be put out of service at any time by the Department of Transportation. But Amazon’s ingenious system has allowed it to avoid that kind of scrutiny. There is no public listing of which firms are part of its delivery network, and the ubiquitous cargo vans their drivers use are not subject to DOT oversight.”
Amazon’s quest for doing more, faster, cheaper, impacts the safety of the communities where their drivers deliver, through traffic accidents and even fatalities.
By working with small delivery operators, Amazon can avoid legal liability and also push the operators in ways that UPS and FedEx could not accept, in delivery volume, lower training requirements, and lesser vehicle safety. Many of those differences are required by the lower cost structure and higher delivery load that Amazon places on the small operators.
In their own way, Amazon drivers competing for more delivery orders game the system. By hanging their phones from trees near delivery stations and Whole Foods, syncing to another device in their car, and then parking as close as they can, the drivers trick the logistics system to assign them orders preferentially. If there were enough parking next to these locations perhaps the drivers wouldn’t need to go through the trouble.
A consequence of increased deliveries is also packaging waste, notably carboard boxes.
Cardboard increased from 15% of curbside recycling in 2004 to around half of the volume by 2019. Who pays for that increase? As I wrote in Visibility of Cost:
“Costs work differently depending on who pays and when they pay. These questions of “who” and “when” and innovations that change them are second-order effects that impact what society gets more or less of.”
Should ecommerce companies like Amazon pay for the costs of recycling their packaging? More cardboard is put into landfills today, as the market to recycle it fell.
Rates for cardboard recycling fell over the years, in part due to more shipments going directly to people homes, rather than to retailers. Compared to retailers, when individuals open their products and dispose of the boxes, more of the cardboard becomes dirty with household garbage and rendered unrecyclable.
- Causes of the unintended consequences: Prioritizing faster and cheaper.
- Counteractions: Regulation of small delivery companies. Push penalty for delivery accidents to Amazon.
Recruiting
Poor selection of algorithm training data.
As a large organization, Amazon is in a position to optimize processes. One common process is recruiting new employees. How could this process be made more efficient and more fair?
For a time, Amazon used a recruiting AI that they had built to tackle this issue. However, it turned out that the AI was biased against women candidates by being fed training data from 10 years of resumes and job candidates, “teaching itself that male candidates were preferable.”
Amazon disbanded the group that created the recruitment AI in 2017. The reaction to this one could of course be just as biased as it was formerly, or biased in a different way.
- Causes of the unintended consequences: Believing that people are the source of bias and by removing them we can be fairer. Not knowing what to pay attention to in AI training data.
- Counteractions: Early tests of results.
Consider
- Whose responsibility is it to watch for unintended consequences created by businesses? Is it within the authority or desire for anyone to monitor and act on them?
- Amazon’s most recent quarterly results are greatly improved from last year, even with $4 billion spent for safety during COVID.
- When it came to facial recognition AI, Amazon chose to withdraw from the market, rather than play a part in the potential misuse of such systems, which I agreed with. Perhaps withdrawal from complex areas makes sense.
- One person’s perverse result may be a company’s unexpected benefit.
I’ll continue to look into second-order effects stemming from Amazon business lines in future posts.