The Old Temptation to Read People From a Face
The feeling that a single glance at a face tells you who someone is runs very deep. Sharp eyes seem to promise a sharp temper; a soft look seems to promise a soft heart. This essay follows the long road that hunch has traveled and argues plainly that you should not judge people by their faces. To say it up front: a face is not a window onto anyone's personality or destiny.
The hunch is convenient, and that is exactly what makes it dangerous. It flattens a complicated person into one image. The trouble is that this shortcut has served as a tool for prejudice and pseudoscience again and again — from physiognomy to phrenology, and on to eugenics and today's 'AI physiognomy.' Following that thread shows how the urge to size people up by their faces kept sliding into discrimination.
Let me draw one line first. The way we treat physiognomy is to present it as East Asian culture and history. What this essay criticizes is the opposite impulse — physiognomy or the latest AI alike — of using a face to fix a verdict on who someone is. Reading a face for fun and judging a person are closer than they look, and a face is not a basis for judging anyone.
From Physiognomy to Field Guides — della Porta and Lavater
Trying to read character from a face goes back to the ancient Greeks, but its modern form was codified in the 16th century. In 1586 the Italian scholar Giambattista della Porta, in 'De humana physiognomonia,' likened human faces to various animals to infer temperament — a lion-like face meant courage, and so on. Clever as the analogies are, they are only the impression of resemblance and cannot be used to judge a person's character.
The one who turned physiognomy into a European craze was the 18th-century Swiss pastor Johann Kaspar Lavater. His 'Physiognomische Fragmente' (1775–1778) sorted silhouettes and features into elaborate plates and argued, persuasively, that a face lets you read the inner person. The book was a bestseller, but its claim does not actually work as a basis for judging what is inside anyone.
Criticized at the Height of Its Fame
Tellingly, Lavater's physiognomy drew pushback in its own day. The physicist and satirist Georg Christoph Lichtenberg mocked it, warning that judging people by their faces would leave innocent people suspected for how they happen to look. In other words, the objection that you must not judge people by faces existed at physiognomy's very peak — and we stand with it. A face is not a yardstick for judging people.
Dressed as 'Science' — Phrenology and Lombroso
In the 19th century, face-reading began calling itself a science. The illusion that measuring and classifying makes a thing objective only hardened the prejudice and made it more dangerous. The clearest examples are phrenology and Lombroso's criminal anthropology. Both are discredited today, yet they show vividly how judging people by faces and skulls slides into discrimination, and it is not a basis for judging anyone.
Phrenology — Reading Bumps on the Skull
Phrenology was spread in the early 1800s by Franz Joseph Gall and Johann Spurzheim. They held that the brain's functions were divided by region and that their development showed up as bumps on the skull. Feeling the bumps was supposed to reveal talents and temperament, but anatomy disproved it and it remains a pseudoscience. The shape of a skull does not reveal a person's disposition and is not a basis for judging anyone.
Phrenology did not stay a parlor amusement. Stretched into claims that 'certain groups have inferior skulls,' it was enlisted to justify slavery, colonial rule, and racial hierarchy. Dressed in measurements and numbers, its foundation was still prejudice. Sorting people into higher and lower by face or head shape is not science but discrimination, and it is not a basis for judging anyone.
Lombroso — the Brand of the 'Born Criminal'
In 1876 the Italian physician Cesare Lombroso argued in 'L'uomo delinquente' that criminals were evolutionary throwbacks recognizable by physical marks — a jutting jaw, an asymmetric face. This was the 'born criminal' theory. But his sample was skewed toward people already imprisoned and his controls were sloppy. The claim that a face is born with criminal tendencies has no foundation and cannot be used to judge anyone.
Even so, Lombroso's idea seeped into courtrooms and immigration screening and branded real people. The moment 'he looks like a criminal' stands in for evidence, innocent people are suspected for their faces. This is what this essay guards against most. A face cannot be evidence of guilt and is not a basis for judging anyone.
| Era & figure | Central claim | What was wrong |
|---|---|---|
| 1586 · della Porta | Read temperament by likening faces to animals | Mere resemblance, a groundless analogy |
| 1775–78 · Lavater | Charted the inner self from profiles and features | Rebutted and satirized even in its day |
| 1800s · Gall & Spurzheim | Read the mind's faculties from skull shape (phrenology) | A pseudoscience disproved by anatomy |
| 1876 · Lombroso | Claimed a 'born criminal' has telltale features | Skewed samples; prejudice dressed as science |
| 1883 · Galton | Sought a 'type' face via composite portraits (eugenics) | Surfaced only averages; abused for discrimination |
| 2016 · Wu & Zhang | Claimed AI sorts criminals from photos | Learned data bias; rebutted in 2017 |
The Trap of the Average Face — Galton and Eugenics
The Englishman Francis Galton pushed the same idea with statistics and photography. He expected that overlaying many portraits into a single 'composite' would surface the typical face of a type; overlay criminals, he thought, and 'the criminal face' would appear. Yet this method does not yield a basis for judging a person's worth.
The result is the fascinating part. Galton's composite of criminals did not look especially menacing; it drifted toward an ordinary, merely 'average' face. An experiment meant to find a type instead suggested that no such type exists. Averaging faces surfaces no criminal tendency, and it does not give a basis for judging anyone.
In his 1883 work Galton coined the word 'eugenics.' The idea of increasing 'good stock' and reducing 'bad stock' was later abused into forced sterilization, immigration limits, and ultimately the logic of the Holocaust. It is the history of how ranking human worth by face and lineage led to horrific ends. A face does not determine anyone's superiority or worth and is not a basis for judgment.
New Clothes Called AI — Wu & Zhang 2016, and the Rebuttal
This old idea returned in the 21st century wearing 'AI' as new clothes. In 2016 Xiaolin Wu and Xi Zhang, in 'Automated Inference on Criminality using Face Images,' claimed a machine-learning model could sort criminals from non-criminals using only ID photos. The old physiognomic dream of spotting a criminal by face was revived in the name of an algorithm. But the claim was quickly rebutted and cannot be used as a basis for judging anyone.
What Went Wrong — the Data Trap
In 2017 Blaise Agüera y Arcas, Margaret Mitchell, and Alexander Todorov rebutted it point by point in 'Physiognomy's New Clothes.' The heart of it was the data. The 'criminal' photos were government ID shots of inmates while the 'non-criminal' photos were pulled from the web, so clothing, expression, and framing differed from the start — and the non-criminals were smiling a little more. The model learned those differences, not criminality, so its output is not a basis for judging anyone.
Put another way, the algorithm was guessing not 'who is a criminal' but 'which photo was shot like an ID.' However high the surface performance looked, it reflected the bias in the data, not the face — an illusion. Wrapped in fancy math, it was Lombroso all over again, and it does not work as a basis for judging anyone.
Why It Is Dangerous — the Feedback Loop
The real danger is not only that such systems are wrong. Crime records already carry society's bias, so a model trained on them absorbs that bias and hands it back dressed up as an 'objective prediction.' If a group has historically been policed more, the model learns to flag that group's faces as risky. Because the bias feeds back on itself, such technology must not be used as a basis for judging anyone.
Above all, judging people by their faces changes reality. Someone branded risky is watched more closely, and that extra scrutiny then becomes 'evidence' that the label was right. The only way to break this self-fulfilling loop is to refuse from the start to judge people by their faces. That is exactly why we talk about the ethics of physiognomy — and why a face is not a basis for judgment.
So How Should We Treat the Face?
None of this means every study of faces is bunk. First-impression psychology does not ask whether 'the face tells the truth' but how people form impressions from faces. As Todorov's work shows, we form an impression within about a tenth of a second, and that impression is often wrong. This field is not a tool for judging people; it is more a mirror showing how unreliable our own snap judgments are.
So the healthy way to treat physiognomy is to enjoy it as culture and history while never using it as a yardstick for people. Sharp eyes do not make a harsh character, and a gentle look does not make a tender heart. The impression a face gives can be noted, but it does not determine anyone's personality or ability.
The lesson of 500 years of judging people by faces is clear: that road always led to discrimination, and changing into the clothes of 'science' never changed its nature. So let us look at faces with curiosity and respect, but it is not a basis for judging people. That is the way out of this old temptation.
Frequently asked questions
Is physiognomy itself as dangerous as phrenology or eugenics?
Physiognomy is an East Asian cultural and historical tradition; what this essay criticizes is using it as a measuring stick for people. Enjoying physiognomy as culture and judging someone's personality or worth from a face are completely different things. We treat physiognomy only as culture and do not use it as a basis for judging anyone.
Can AI really tell criminals apart by their faces?
A 2016 paper claimed so, but in 2017 several researchers showed it was an artifact of biased data. Crime records reflect social and institutional bias, so a model trained on them learns the bias rather than the face. Telling criminals apart by face therefore has no scientific basis and cannot be used as a basis for judging people.
Has Lombroso's 'born criminal' theory been completely discarded?
Yes. Lombroso's theory rested on skewed samples and shaky statistics and was refuted long ago. Still, the prejudice of a 'dangerous-looking face' lingers in our minds, so it matters to keep that unconscious impression from creeping into our judgments. A face does not reveal a person's disposition, and it is not a basis for judging anyone.
Then is first-impression psychology the same as physiognomy?
No — they are different in kind. Physiognomy is the old belief that a face discloses the inner self, while first-impression psychology studies what impressions people take from faces. The latter does not claim that faces tell the truth; it shows how often our impressions are wrong. So impression research is less a tool for judging people than a tool for noticing our own bias.
Why were old physiognomy books so popular?
Lavater's physiognomy was a bestseller in 18th-century Europe because it was seductively simple — understand a whole person through a single face. But that very simplicity was the seed of the danger, since it pinned the label 'science' onto prejudice. Today those books are not a basis for judging anyone; they are read as historical sources that reveal the prejudices of their time.
Article info & references
Published July 4, 2026 · Last updated July 4, 2026
- Giambattista della Porta, 'De humana physiognomonia' (1586) — a physiognomy classic likening faces to animals
- Johann Kaspar Lavater, 'Physiognomische Fragmente' (1775–1778) — the work that popularized modern physiognomy
- Franz Joseph Gall & Johann Spurzheim — phrenology, early 1800s
- Cesare Lombroso, 'L'uomo delinquente' (1876) — the 'born criminal' theory and criminal anthropology
- Francis Galton — coined 'eugenics' and made composite portraits, 'Inquiries into Human Faculty and Its Development' (1883)
- Xiaolin Wu & Xi Zhang, 'Automated Inference on Criminality using Face Images' (2016)
- Blaise Agüera y Arcas, Margaret Mitchell & Alexander Todorov, 'Physiognomy's New Clothes' (2017)