Image recognition is a set of technologies where we’ve seen great progress recently. Some applications help us gain advantages of efficiency, for example identifying items that may be debris, for removal from an agricultural field. There’s also accuracy, for example identifying tumors in cancer screens at a better rate than human experts. And some applications are for convenience, for example enabling users to unlock their devices with their faces rather than passwords.
These applications can lead to good outcomes just as they can also have unintended consequences.
Related to that, at the start of the recent wave of protests in Hong Kong, a journalist opened an article with a beautiful summary of how important facial recognition had become.
“The police officers wrestled with Colin Cheung in an unmarked car. They needed his face.
“They grabbed his jaw to force his head in front of his iPhone. They slapped his face. They shouted, ‘Wake up!’ They pried open his eyes. It all failed: Mr. Cheung had disabled his phone’s facial-recognition login with a quick button mash as soon as they grabbed him.”
It seems legitimate that we fear misuse of facial recognition. It’s a question of suddenly being able to do something at a scale that would be difficult or costly earlier.
But what about subtler abuses?
That brings me to a new report, titled Facial recognition technology can expose political orientation from naturalistic facial images.
From the report:
“Pervasive surveillance is not the only risk brought about by facial recognition. Apart from identifying individuals, the algorithms can identify individuals’ personal attributes, as some of them are linked with facial appearance. Like humans, facial recognition algorithms can accurately infer gender, age, ethnicity, or emotional state. Unfortunately, the list of personal attributes that can be inferred from the face extends well beyond those few obvious examples. A growing number of studies claim to demonstrate that people can make face-based judgments of honesty, personality, intelligence, sexual orientation, political orientation, and violent tendencies. There is an ongoing discussion about the extent to which face-based judgments are enabled by stable facial features (e.g., morphology); transient facial features (e.g., facial expression, makeup, facial hair, or head orientation); or targets’ demographic traits that can be easily inferred from their face (e.g., age, gender, and ethnicity).”
The research used dating site profiles that included a picture and a stated political orientation.
Think of how many people unwittingly contributed to the findings.
Too Fast, Too Far
As I wrote in The Emergence of Omniscience (Part 1 – Images):
“San Francisco became the first US city to ban facial recognition tech used in public.
“As the Board of Supervisors noted, there is a fine line between ‘good policing’ and becoming a ‘police state’, and the recent rapid improvements in facial recognition technology… have raised many concerns that the technology was advancing too far, too fast… It is important to note, however, that the ban on facial recognition technology for city agencies does not apply to private individuals or private businesses.”
“Sounds like the ban will spread to other locations but something makes me pause. That term ‘too fast.’ Facial recognition tech seemingly sprouted up suddenly, which was part of the reason behind the push-back. I say give it some more time and the use cases for the tech will overpower the privacy concerns.”
But back to the new research. The part that seemed to be going too far was this quote:
“Moreover, the accuracy of the human judgment is relatively low. For example, when asked to distinguish between two faces—one conservative and one liberal—people are correct about 55% of the time (derived from Cohen’s d reported in Tskhay and Rule), only slightly above chance (50%).”
This seems to be a stretch and not because that 55% should really be some other number. Instead, it’s a reminder of how research can mislead if you don’t read the original sources.
The 55% statistic, supposedly about identifying conservative and liberal faces, is no such thing. Following the footnotes leads to another study, which quotes yet another one from 1946, being the source of this 55%.
That original study is so old that I could only find parts of it online (the original: Allport, G. W., & Kramer, B. M. (1946) Some roots of prejudice. Journal of Psychology, 22, 9-39. The quote from Tskhay and Rule mentioned above is from Accuracy in Categorizing Perceptually Ambiguous Groups: A Review and Meta-Analysis).
The topic of that 1946 study was not identifying conservatives or liberals by facial images. Instead, the supposed 55% human accuracy rate comes from a very different question: asking people to mark yearbook photos as being of Jewish or non-Jewish students. The purpose was to test how much anti-Jewish prejudice comes from facial cues.
Techno Richelieu
After all this talk about facial recognition enabling characterizations of individuals, we should ask: why does it matter? After all, people take great efforts to announce who they are — politically and otherwise — and willingly.
I think the bigger problem is from what I’ve been calling the Techno Richelieu Effect.
Cardinal Richelieu’s supposed quote (though he may have never said it) was “If one would give me six lines written by the hand of the most honest man, I would find something in them to have him hanged.”
Today, everyone produces much more than six lines. Interpreting what we mean in our words, or the way our face looks when our images are captured, produces enough data for guilt, if needed, in two ways.
One is the guilt that comes from being a statistical match for unwelcome behavior, for example looking similar enough to someone who participated in a riot, a protest, a crime. Results: avoid working with someone with high statistical guilt, raise insurance premiums, reduce default rates of trust.
Another type of guilt is that of wearing an expression that statistically shows we thought something undesirable while being presented with information. This is a potential outcome of the political orientation-style research. But really, marketers already do this type of work to sell products.
Marketers have long been able to tailor ads to people based on user behavior. Psychological targeting (extraversion, openness) has already been used to improve ad click-through rates. The information used for the targeting can be as minimal as a single Facebook like.
Combined with perhaps exaggerated capabilities, should facial recognition of political orientation matter to us that much?
I believe that the real problem is not facial recognition being able to figure out someone’s supposedly true feelings better than people can manually. It’s being able to do so at scale (thousands in an instant rather than one at a time) and then putting statistical guilt on those identified. The Techno Richelieu effect means that improving such a process is enough.
The Shape of Faces to Come
Like CEOs matching their language to the latest dictionary version of a trading algorithm, people will act to avoid political orientation targeting. At least to a point.
- Does it matter? Something like political orientation is easy to find by other means, including online statements and images. Many people self-report their political orientation online. Matching the individual’s identity with their face while out at an event is becoming more possible.
- There are more subtle outcomes. Rather than something as broad as political orientation, assessing people for cultural or team fit could be a problem. A facial recognition assessment of job applicants might lead to different types of discrimination.
- Misidentification will happen. As in the famous Robert Julian-Borchak Williams case, this misidentification is as much a problem with a manual police process than it is with facial recognition tech.
- Goodhart’s Law says that once chosen, metrics can lead to unexpected outcomes.
- Read the original sources of research that catches your eye.