While society has shunned those who are sexist and racist, many of us are committing the sin of "face-ism", say experts.
Research has shown that people often make sweeping judgments of others based on their facial features.
For instance, individuals with feminine-looking or naturally happy faces are consistently thought of as more trustworthy.
Competence, dominance and friendliness are also associated with specific facial traits.
Writing in the journal Trends in Cognitive Sciences, US researchers warned "face-ism" can lead people to make rash decisions, from voting for a particular politician to convicting someone of a crime.
Dr Christopher Olivola, from Carnegie Mellon University's Tepper School of Business, Pennsylvania, said: "Although we would like to think our judgments and choices are rational, impartial, consistent, and solely based on relevant information, the truth is that they are often biased by superficial and irrelevant factors.
"This is a troubling human tendency that needs to be corrected, or at least mitigated, because faces are not valid predictors of a person's traits."
Dr Olivola and co-authors Friederike Funk and Alexander Todorove, from Princeton University, said numerous studies had shown that people form "face-ist" impressions of aspiring leaders.
Political candidates with naturally competent-looking faces were more likely to win elections than those who looked less competent.
"Babyface" features characterised by a round face, large eyes and a small nose and chin are generally seen as a sign of incompetence. The reverse is true for strong, mature faces.
Company chief executives who looked competent had a better chance of being hired by large successful companies, even when they performed no better than less competent-looking rivals, said the researchers.
In the military, having a dominant, masculine face appeared to be a ticket to higher rank attainment.
Shifty-looking individuals were also more likely to be convicted of crimes. Research has shown that they tend to have sunken cheeks, furrowed brows and eyes set close together.
In contrast, a smiling face with prominent cheekbones and high eyebrows suggests trustworthiness and honesty. People with these features have been shown to be more adept at securing financial investments and loans.
Research suggests "face-ism" can be combated by providing more relevant information, such as the past behaviour of a political candidate, said the experts.
"We need to guard against letting our choices be biased by superficial cues," said Dr Olivola.
"In some contexts, educating people might be sufficient to reduce facial stereotyping. In other contexts, however, more research will be needed to identify the best way to mitigate the biasing influence of facial appearance."
