On Focus Groups, Elections, and Predictions
Focus groups, a method in which small groups of subjects are questioned by researchers, are widely used in politics, marketing, and other areas. In education policy, focus groups, particularly those comprised of teachers or administrators, are often used to design or shape policy. And, of course, during national election cycles, they are particularly widespread, and there are even television networks that broadcast focus groups as a way to gauge the public’s reaction to debates or other events.
There are good reasons for using focus groups. Analyzing surveys can provide information regarding declaratory behaviors and issues’ rankings at a given point in time, and correlations between these declarations and certain demographic and social variables of interest. Focus groups, on the other hand, can help map out the issues important to voters (which can inform survey question design), as well investigate what reactions certain presentations (verbal or symbolic) evoke (which can, for example, help frame messages in political or informational campaigns).
Both polling/surveys and focus groups provide insights that the other method alone could not. Neither of them, however, can answer questions about why certain patterns occur or how likely they are to occur in the future. That said, having heard some of the commentary about focus groups, and particularly having seen them being broadcast live and discussed on cable news stations, I feel strongly compelled to comment, as I do whenever data are used improperly or methodologies are misinterpreted.
The problem, put simply, is that focus groups are frequently misused (as are survey results, for that matter). The observations are interpreted erroneously; for example, as an indication of how many voters in the general population would switch their opinion after being presented with a particular argument, or as evidence that this argument was the reason for (cause of) the opinion change. Given how often focus groups are used and interpreted, it may be helpful to discuss these issues briefly.
Focus groups, in qualitative methodology, are used for very specific purposes. Regardless of how the participants in such a group or groups are chosen, the goal of convening focus groups is to get insight into how the participants react to certain stimuli in the presence of and interacting with other group members. That is, they are useful in gathering information about people’s views on a given issue in a collective context.
What this implies is that views and feelings of focus groups’ participants cannot be interpreted outside of the context in which they were observed. That is, they are solicited as much by moderators as by other participants’ opinions, and the group dynamic is as important as (if not more important than) individuals’ responses. This group aspect is what differentiates focus groups from in-depth interviews, also a common qualitative approach. The latter are conducted in order to gain insight into an individual’s subjective interpretation of events, whereas the former is to observe subjects’ reactions while interacting with others.
Focus groups are informative if we are interested, for example, in what vocabulary, figures of speech, or cultural reference points members of the group use to connect/explain themselves to others, while in-depth interviews are useful in understanding how they explain their opinions and actions to themselves.
For the reasons highlighted above, focus groups cannot provide information regarding individual people per se. The data should be analyzed in the context of the group – participants’ reactions may be different in a different group, even one consisting of seemingly similar persons. (And, of course, reactions could also be different in a group not broadcast on national TV.)
Moreover, focus groups are more useful for illustrating differences than consensus in individuals’ opinions. People interacting with each other have a tendency to converge toward a common opinion, and to suppress dissenting views. Consensus, therefore, may be artifactual. This is especially important in light of the fact that focus group members are usually selected into a given group based on their similarities rather than differences.
The reactions solicited during a focus group interview also cannot provide information regarding shifts in subjects’ opinions, feelings or beliefs. You cannot say that how a subject interpreted a given stimulus is why the subject formed (or changed) his or her view. The narratives offered by participants can only help to reconstruct the process of “at the moment” interpretation of events/stimuli presented, as focus groups provide a one time observation. For example, when a participant says that watching a presidential debate made him or her angry (or happy, or hesitant, or whatever) because of a certain word or expression a given candidate used, you cannot say that the word or expression itself caused the anger. There is a range of alternative reasons why the person could react with a given emotion. For instance, that person could have been angry with the candidate in general, and the word or phrase might simply have triggered that feeling. You can, however, probe for meanings that subjects associate with a given term or expression, which may become useful when thinking about, say, possible communication strategies.
Another, perhaps more fundamental reason why causal inferences based on what participants say in focus groups are not warranted is the following: People often do not know why they do what they do, or why they think what they think. That is not meant to be condescending. It is an established scientific fact. Human thought processes are unbelievably complicated. Plenty of research into decision making (here, here, and here, for example) demonstrates that what people identify as the reasons for their actions often are not consistent with the range of possible objective reasons. Moreover, it is unreasonable to expect individuals to be aware of how their interpretations are constrained by the opportunities they were given (here, here, and here, for example).
Put simply, when people react to stimuli, such as a political advertisement or a debate, they can be asked how they justify that reaction. The purpose of focus groups is to explore those justifications (words, cultural references, symbolism, etc.) expressed in the presence of and while interacting with other people, not to identify why they actually feel the way that they do.
As mentioned above, focus groups are a form of qualitative data, and no qualitative data can ever be used for making generalizations about a given population, regardless of the focus group’s composition. (Though, in the case of cable news broadcasts of focus groups, viewers aren’t even told anything about how the groups are selected or who the members of the group are.)
There are many ways to select qualitative samples, but none of these qualitative sampling methods gives any basis for generalizations to populations from which these groups were chosen. In other words, qualitative generalizations can extend to qualitative ideas, such as when you want to build a map of connotations brought up by a given term, but that doesn’t mean those connotations are shared by the population in general. It follows, therefore, that focus groups do not provide any basis for claiming that a given stimulus would have the same effect for persons different from those in the focus group.
To be sure, there is a very strong temptation to draw strong causal conclusions from focus groups. When a person tells you, for example, why they support a given candidate, it is natural to take them at their word. But our reactions to people and events are highly complex – even we are not fully aware of all that goes into them. Focus groups can be useful in showing us how people explain their reactions to others, but they cannot show us why people think what they think, nor can they help to predict how others will think or react in the future. It would be wonderful if all those who present and discuss focus groups would learn the difference.
I think this is pretty good, but you mae a fundamental mistake.
Acutally, qualitative data CAN be used to generalize to a larger population, but it cannot make STATISTICAL generalizations. And the reason is not really that the data is qualititative as much as it is that the sample size is so small.
Image a VERY large scale survey (i.e., NOT semi-structured interviews), using a rigorously constructed sample, with a open response item. The data that comes back from that open responsen item would be — by definition — qualitative. But because the sample is large (and contructed to be represenative), you COULD generalize to the larger population, statistically. In fact, that is exactly why you would do that.
So, if we can’t make statistical generalization (e.g., proportions of people who think X) to larger population, then what CAN we do?
Well, we can use qualitative research to study potential mechanisms or the nature of processes and then generalize those processes or mechanism to the larger population. In fact, that’s the primary purpose of rigorous qualitative research.
Darwin used qualitative data. He developed a theory about a process and a basic mechanism, and generalized that quite broadly. And it is one of the great scientific accomplishments in human history.
Generalization. Based on qualitative data.
I think your point is valid, but from my read, what Kinga is saying more to do with the notion that we can NOT use focus groups to generalize a political message to the wider audience. Where as we can take product that someone might use and get impressions and a decision if that person might buy a product in that moment as a legitimate method of market analysis, it is less likely that this method is working in our society today for political sensitivity and message definition.
Mostly, as this method has been playing out over the past several decades on the political front to be somewhat helpful and useful, the general population is starting to get wise to this method. And as such, they might not be honest in such a group since they may be discovering the relationship to the political group that the person asking the questions might not have their best intentions in mind and people asking the questions are more interested in whether they have successfully manipulate the subject.
The only real world example that I have to say as a data point that might prove this to be true would be the election of Donald Trump, all of the surveys and focus groups pointed to the other candidate as the winner. Those focus groups were wrong. Not in a small way, but in a much bigger way.
The challenges we face today now is that the political class and the political consultants are and will be at a loss in the next several years to what will be most effective in manipulating the population. As such, to use focus groups to determine if your form of manipulation will work and guarantee as "WIN" politically is now in question. This was the point of this article, and form my perspective, this is BRILLIANT.