What can we tell from the face? There're mixed data, but some show a pretty strong coherence between what is felt and what’s expressed on the face. Happiness, sadness, disgust, contempt, fear, anger, all have prototypic or characteristic facial expressions. In addition to that, you can tell whether two emotions are blended together. You can tell the difference between surprise and happiness, and surprise and anger, or surprise and sadness. You can also tell the strength of an emotion. There seems to be a relationship between the strength of the emotion and the strength of the contraction of the associated facial muscles.
The thing that I find fascinating about facial expression of emotion is our limitation in the ability to control them. They come in two separate forms. The first is in the distinction between spontaneously and deliberately induced facial expressions of emotion. What I mean by spontaneously is, those facial expressions that happen as a result of some stimulus intended to elicit emotion of some valance. Deliberately induced facial expressions are those that happen as a result of a directed facial action task—if someone says, "I want you to smile on command, or frown on command."
It turns out that the spontaneously induced and deliberately induced facial actions emanate from separate upper motor neuron pathways. The deliberately induced facial expressions come from the cortical motor strip, whereas the spontaneously expressed emotions emanate from the phylogenetically older, extrapyramidal motor system. What this means is that if you’ve got damage to the cortical motor strip, you lose the ability to make facial expressions on command but retain the ability to make them in response to some emotion-eliciting stimulus. On the other hand, if you’ve got damage to the extrapyramidal motor system, you lose the ability to respond emotionally to a stimulus, but you’ll be able to make a facial expression on command. That, I found very fascinating.
There’s another limitation that’s been found by Ekman’s group. What he has found is that a certain number of facial expressions can be thought of as reliable. Reliable facial muscles have two properties. Less than 10 to 15 percent of individuals can do two things with these reliable muscles. The first thing is to create them in the absence of that associated emotion. The second thing is to stop them from happening in the presence of the associated emotion.
Those two findings are the ones that made me jettison all the previous questions that I was asking about how we make facial expressions of emotion, because this struck me as extremely odd. Why would we express something as private and as subjective as our motivational and emotional states somewhere as conspicuous as the face? Wouldn't the smell of fear just give our opponents some advantage? Wouldn't a poker face be better? I went from asking these "how" questions to these "why" questions. There are three classes of answers to these questions. I’ll get to the one that I’ve been focusing on recently, last.
The first class comes from a physiological function. This comes from Darwin's Expression of Emotion in Man and Animals. He stated that facial expressions have a physiological function. That is, they allow us to respond to recurrent stimuli that happen within our environment. For example, the widened eyes that you see in fear expressions allow us to get more information from our periphery when we need that; the scrunching of the nose and the protrusion of the tongue allow us to expel noxious stimuli from our nose and mouth. There's good evidence to corroborate this suggestion.
None of these classes are mutually exclusive, not by any means. The second class is a communicative class of functions. Darwin didn’t talk about this communicative class of functioning for a very specific reason. At the time that he was writing Expression of Emotion in Man and Animals, a lot of creationist folks, including Sir Charles Bell, thought that facial expressions were a God-given way of communicating emotion. Darwin, for obvious reasons, wanted to distance himself from those conceptualizations because he had the whole evolution by natural selection thing that he wanted to propagate.
Darwin didn’t talk about any communicative function of facial expressions at all, and that’s why these questions have lagged a bit behind. To his credit, he did imply a communicative function with his antithesis principle. The antithesis principle states that expressions have the form that they do because that form opposes the form of opposite expressions. For example, the upward lip corner turning in smiling differs from the downward turning in sadness, the outward turning in fear, and the dimpling action in contempt. That would be Darwin’s explanation for something like a smile.
That's the second class—the communicative class of functioning. This makes a lot of intuitive sense, but it still didn’t answer the question that any adaptationist would ask: What is the benefit to the signaler of expressing something on their face? The benefit to the receiver seems more clear. It allows the receiver to predict what that person’s emotional state is and what they’re going to do next. The benefit to the signaler wasn’t so clear to me.
That takes me to the third class of functioning, which is related to the communicative functioning, and what I’m going to call the commitment function of facial expression. What I’m about to say is very ubiquitous; it spans economics, biology, psychology, and it’s been put forth by Robert Frank and Jack Hirshleifer, and previously Thomas Schelling in the Sixties. The idea is that we’ve got sets of emotions that Adam Smith called moral sentiments. These moral sentiments function by competing with calculations that stem from rational self-interest.
If you take someone that has strong guilt feelings or is capable of strong guilt feelings, this person’s not going to cheat on their spouse, not because they’re afraid of getting caught, but because they don’t want to; they know that the guilt feelings are going to be aversive and it’s going to outweigh whatever benefit they’d get from cheating on their spouse. Similarly, take someone who is capable of great acts of revenge. They’re not going to need a formal contract to get revenge, they’re going to get revenge because they want to, because it’s going to feel good and that good feeling is going to outweigh the negative consequences of seeking revenge.
This is still problematic because the good feelings that we get when we remain faithful to our spouse and the good feelings that we get when we exact revenge are in and of themselves very real rewards. But we are living in a material world of material payoffs and for these incentives to be viable, they must have incurred some material benefit. That is to say, it’s no good for me to say, "Look, you should have trusted me when we could have cooperated before, because I would have," and it's no good for me to say, "Look, you should not have harmed me a while ago," when the harm is already done. The idea is that for these incentives to give material payoffs, they need to be honestly communicated beforehand.
From this view, this answers the question that has been gnawing at me of what’s the benefit to the signaler? The benefit to the signaler is that they can now honestly express threats and promises. From this viewpoint, what I categorized before as limitations in our ability to control facial expressions now becomes a necessity—a defining feature of the conceptualization of facial expressions. This is the idea that I’ve taken, that I’ve tried to find empirical evidence for. I’ve done a couple things so far, so I’ll talk to you about those just briefly, and then I’ll talk to you about what I’ve been doing very recently—the data that I’ve collected just last week—and some of the questions that I have moving forward.
The idea is that facial expressions should enhance the credibility of our threats and promises. In a study we had out a couple years ago, we had people play a one-shot prisoner’s dilemma with an acquaintance period beforehand. What we found is that most people gave verbal promises to cooperate during this prisoner’s dilemma, probably 75 to 80 percent of those individuals. Some of those folks promised with genuine, difficult to fake smiles—Duchenne smiles. The verdict’s out on how difficult it is to fake a Duchenne smile. I could do it right now, but you could argue that it’s a costly signal.
What we found was a couple things. Those individuals that had their verbal promises paired with Duchenne smiles were more likely to cooperate with their partners. Furthermore, their partners predicted that they would be more likely to cooperate with them, which suggests that there’s some encoding of these promises in facial expression among signalers, and some decoding of these promises among receivers.
The other side of that coin is looking at threats and promises. What we did next is we had people play an ultimatum game. An ultimatum game, I’m sure a lot of you are familiar with, but quickly, you’ve got a proposer and a responder. The proposer has control of how much he’s going to divide the pie between the two of those individuals. The responder can accept that proposal, in which case both individuals get what the proposer decides. Or the responder can reject it, in which both individuals get nothing.
What we did is we had threats to reject incredible offers—high offers, offers that responders wouldn’t normally reject. We paired them with either neutral or angry, difficult to fake angry facial expressions. What we found was that those threats that were paired with difficult to fake anger expressions resulted in high proposer offers. What we think that suggests is that facial expressions, specifically difficult to fake anger expressions, add credibility to our threats.
What we want to do now is to see what other facial expressions might have this commitment function. I don’t think that all of them would, but all the facial expressions would have some communicative function. In addition to threats and promises, we’re also looking at requests. Again, this is the data that we’ve just got this week, and we’re still writing it up. The idea, and some people have positioned this, particularly Ed Hagen, and Randy Nesse, that sadness functions as a bargaining tool that would allow us to extort—I’m hesitant to use that word because it has a negative connotation, but I don’t mean it that way—resources from con specifics that would have some stake in our fitness.
We just did an MTurk sample in which we had two people playing a two-person threshold public goods game. We had one of the individuals state a request saying, "I can’t contribute my fair share in order for us to get the threshold for the payout. Can you give me more?" What we found was that this verbal statement, when paired with difficult to fake sadness expressions—the reliable facial muscles of the triangularis, which lowers the lip corner and the medial frontalis, which gives you that triangulation of the brows in worry and sadness expressions. Think of the Woody Allen expression. He’s one of those 10 to 15 percent that can do that at will. But now that he can it means absolutely nothing, right? You don’t infer any concern from him. That goes back to the point that Robert Frank made a long time ago that if the facial expressions that are characteristic of specific emotions were easily and readably fake, they would no longer be characteristic of those specific emotions.
We've looked at threats. We’ve looked at promises. Now we feel we’ve got some preliminary data that there’s a communicative function for sadness. The other emotions, I’m not so sure that they’ve got a commitment function, but some of the things that we’d like to do are to look at disgust. Disgust is hypothesized that it functions physiologically for the signaler, but the receiver benefits because it lets them know, "Look, whatever this person is exposed to, I might be exposed to as well."
Often times you’d imagine the EEA, in the Pleistocene era, that people are sharing meals. Such an adaptation that will allow us to figure out whether someone might be infected would be very beneficial. One of the things I’m thinking about doing—you’ll have to forgive me but this is in its planning stages—is of having some shared meal between participants. I was thinking about having Jelly Belly jellybeans, one of the nasty flavors put in there, and see how the recipient or the receiver responds when the person eats that disgusting one.
What I would anticipate is that there might be some contagion effects. If you saw an individual with disgust, you’d find that in the other participant as well. You might find the same with something like surprise. Surprise, again, has this physiological benefit to the signaler, but to the receiver, it tells them, okay, this person is watching out for something dangerous, and I am in their proximity. Maybe there’s something I’ve got to watch out for as well. That might be a way to look at some of the surprise expressions.
Finally, there’s this idea that facial expressions, insofar as though they are self-conscious, might form as a way of communicating common knowledge between other individuals. What I mean by common knowledge is the knowledge that I know that you know that I know that you know something, ad infinitum. It turns out that if you have just, I think that you think that I think that I think, that the recursive nature explodes that doubt into something that’s very different than common knowledge. There are some shortcuts to it, like direct eye gaze. I think Hugo was talking about this earlier. When you look at someone directly in the eye, it’s very off-putting sometimes. It’s often used in sexual come-ons and threats. But there’s no denying that you’ve made eye contact with that individual.
Another one could be tearing. Tearing is hypothesized to be a handicap that might serve as a cue for common knowledge because the other individual can be absolutely certain that you’re crying. They can be absolutely certain that you know that they’re crying because it impairs your vision in a specific way. Some people have argued, Provine specifically, that tearing functions as a handicap to make it difficult for us to make attacks on other individuals and defend because of the blurred vision. Also laughing. Laughing is an intuitively strange thing to make that response in response to sometimes seemingly innocuous comments, but that could also be a signal for common knowledge. Any of the motions could be insofar as the only requisite is that they are self-conscious and the other person knows that they are self-conscious. Things like crying, laughing, direct eye contact, blushing are good candidates for that.
That is the question. Those are the putative answers. I’m still trying to find more evidence for this commitment function of facial expression, and asking questions about why we express emotions on our face. It’s very satisfying, because sometimes you have questions, and then they never get answered. But here I feel as though we’re making some headway, which is extremely satisfying.
THE REALITY CLUB
HUGO MERCIER: This is fascinating work, but there is one thing on commitment and emotions that some people get slightly confused about. I’m not saying you’re doing this, but some people are. It's the idea that as soon as emotions are not under voluntary control, then that solves the evolutionary problem of how they can be stable. We know that’s not the case because, evolutionarily, the question that’s relevant is not whether they are under voluntary control or not but whether we could evolve in such a way that they could be expressed in different contexts. Once you have shown that someone who displays anger can reliably express this in behavior, you’re still only going to have half of the picture.
If you want to know how the communication of any given emotion remains stable, not in the sense of why people cannot take advantage of this in terms of controlling their own emotions, but why we didn’t evolve in such a way that we could take advantage of that. One of the solutions we put forward in the paper with Guillaume Dezecache and Thom Scott-Phillips is that the receivers of the emotion are vigilant towards the emotions that aren’t being displayed. If you think the source isn't as worthy, or the emotion doesn’t feel right in the context in which it is being expressed, you’re not going to respond to it in the way that the sender intends it to. I’m curious if you have done any experiments, or can you think of experiments in which someone displays an emotion and they don’t act in the way that is expected of them? Then the next time they display the emotion they’re not taken as seriously as they were, which would maintain the stability of the communication.
REED: That’s a good point. I would expect that there would be some limits on our ability to control our facial expressions, and then some limits on receivers’ ability to decipher how accurate they are, based on exactly what you said, because there’s going to be an arms race between the two. Why haven’t we evolved to completely lie? There are limitations, but the limitations need to be either that everyone has a limited ability, or that only certain people can do it. As long as it’s a heuristic that works more than half of the time, then we’ll have evolved for that. Then we’ll have adaptations for it.
There are studies in which they have displayed film clips, or stimuli intended to elicit various emotions, and then instructed people either to display a different emotion or not display any emotion at all just to see how good people are at controlling them. Forgive me, I don’t know, it’s mixed for all the emotions, but it does say that we have some limited ability to control them.
Then the other part of your question is how does that affect receivers if they know people have the ability to control them?
MERCIER: Even if people had no ability to control emotions at all. If you look at signaling in animals, at least the vast majority of it is not under any voluntary control, we still have the question of how can it remain stable. Why don’t animals evolve to send signals when it’s advantageous to them and not to the receivers? The emotional signals could be completely impossible to fake, and that still would tell us nothing about how they remain stable. You still need the receiver to be able to tell when she should respond to the signal. In a way then you could nearly see how controllable the signals are as being nearly orthogonal to the evolutionary question. Answering the controllability question doesn’t buy you the evolutionary stability that you’re looking for.
FIERY CUSHMAN: I was going to jump in and say that one way to make it not quite orthogonal is to turn the question around, say, given that we’re going to have to construct an honest signal, would it be a viable approach to put it under voluntary control? It's interesting to think that if you were to hand the keys to the car over to voluntary control, voluntary control would surely crash the car so quickly that it wouldn’t be a useful, honest signal.
MERCIER: At least the vast majority of our communication is under voluntary control and it’s mostly honest. So it can be done.
CUSHMAN: Fair enough. Yes.
DAVID RAND: But not in situations where it conflicts it, where conflicting interests exist. Right?
MERCIER: If it’s purely a non-zero sum game you’re playing then there can be no communication, whether it's emotional, voluntary, or whatever. The format of the communication is irrelevant to the evolutionary question. It's not completely irrelevant, but it’s not a necessary condition.
RAND: It seems to me that a point is “What keeps the signal honest?” One answer, which you were suggesting but I think you could test in a straightforward way, is that when someone gives us a signal that they will be honest and then they betray you, you’re more mad than if they had just betrayed you without giving the honest signal.
MERCIER: I’m going to do that. If nobody could steal that, it would be great.
JOSHUA KNOBE: I was thinking about the evolution of involuntariness. It seems like you’re saying that there’s some adaptational advantage that we have from having it not be under voluntary control —that we can send these honest signals. So, if you brought in pairs of people, some who have it under voluntary control and some who don’t, you should expect the people who don’t have it under voluntary control to show certain advantages. That is to say, if I can’t fake sadness but Hugo can, then with people who know me and people who know Hugo, I should be able to get what I want in some cases where he can’t.
It would be an easy way to test this hypothesis with this advantage because we can actually observe people who are doing it.
REED: I completely agree with you, and I am stealing that idea.
RAND: Specifically it would be that the person that can fake it would be at an advantage when interacting with strangers, but the person that can’t fake it would be at an advantage when interacting with friends.
DAVID PIZARRO: This is why reputation is generally spoken of in this context. Robert Frank goes way out of his way to say reputational effects—like communicating, "Hugo is a horrible person"—tend to dominate. There is something about signaling in emotions that is different from the signaling that we often see in the animal world, which is, the emotional signaling is directed at one person. You being angry when you’re talking to me, means something more than just you being angry. Both of those are showing the same facial expression. Is there a way in which humans are better at decoding angry? Angry directed at me means you’re going to screw me over. Angry directed at everybody else means: "This is my buddy. Don’t mess with us." That's just how complex we are. The signal itself isn’t telling everybody the same thing.
REED: That’s another wonderful, empirical question that you could do with eye gaze or something along those lines.
PIZARRO: I feel like somebody has done angry expressions looking at and looking away from, and people have very different reactions to them.
LAURIE SANTOS: Yes, this is true even in monkeys where you get different neural responses for direct gaze facial expressions versus sideways gaze facial expressions. The idea is the same.
