Speaker 1: How can researchers design survey questions that are clear, neutral, and that accurately measure public opinion? For starters, there are some basic principles that carry over from everyday conversation. When you're talking with someone you've just met, you wouldn't assume that they're a policy expert, so you wouldn't use jargon like a continuing resolution or assume that they know what something like the Trans-Pacific Partnership is. Similarly, survey researchers can't assume that respondents know about certain things, so they generally avoid asking questions that ask about obscure topics in a way that provides only limited information. Do you favor or oppose the Trans-Pacific Partnership? Respondents may have heard something about that issue, and they might even have an opinion about it, but they might not know it's called the Trans-Pacific Partnership. So if the survey doesn't define or explain the agreement, it can't measure the opinions of those kinds of respondents. At the same time, though, survey researchers also want to avoid offering too much information in a question, as in this example. In general, that's not a good idea. It's unlikely people will pay attention to that much information, and those who do are likely to give answers that are influenced by how the issue is described, meaning that they might give a different answer if asked about the same issue in a different way. A better way to measure public opinion on this topic would be to remind respondents what the TPP is without overwhelming them with arguments for or against it, such as, do you favor or oppose the free trade agreement that the U.S. negotiated with 11 countries bordering the Pacific called the Trans-Pacific Partnership, or TPP? So your question shouldn't have too much information, and it shouldn't have too little, kind of like the three bears with their porridge and chairs. We want it to be just right. Of course, survey questions also shouldn't lead people to give one answer over another. Consider a question like this one. This question could lead people toward a potential response in two ways. First, it primes them with assertions about a scandal and corruption at the United Nations. Second, it pushes respondents to answering that the U.S. should stop paying its dues. Another fairly common mistake in survey questions is the use of a double negative. One example would be asking, Whatever your overall view of the new health care law, do you favor or oppose provisions in the law that prohibit health insurance companies from denying coverage or charging some customers higher premiums based on pre-existing conditions? What does it mean to favor prohibiting denying coverage? That's confusing. A clearer approach would be to describe the current law, and in a separate sentence, ask if respondents favor or oppose it, like this. Questionnaire designers also want to be wary of acquiescence bias, which is the tendency for people to simply be agreeable. Studies have shown that questions posed as yes-no or agree-disagree tend to overstate how much the public endorses something because generally people's default position in conversation or surveys is to be nice and agreeable. So if you ask a question like, Do you agree or disagree the best way to ensure peace is through military strength? you will get a much higher share of the public's reaction. You will get a much higher share of the public agreeing that military strength is the best way to ensure peace than if you ask the same question and offer two alternative statements like, Which statement comes closer to your views? The best way to ensure peace is through military strength, or diplomacy is the best way to ensure peace. This forced-choice format doesn't offer the respondent the chance to simply say, Yeah, I agree. Not only is the wording of each question important, but so is its placement in the order of the survey. A question asked early in an interview can influence how people interpret a later question or what considerations they have in mind when answering a later question. This issue is called a context effect, and it's why most polls that measure attitudes about the president's job performance or how people intend to vote in an upcoming election, they ask those surveys at the beginning of the questionnaire. That way, when people evaluate the president, they haven't been primed to think about any particular dimension, such as their handling of health care or racial relations, and the survey can get a cleaner overall measurement. After the questionnaire is drafted, two final steps can make a huge difference. First, have people of different backgrounds review the questions. Get input from people with different political views, different religious traditions, varying levels of education, and from different parts of the country. This helps guard against asking questions that are inadvertently confusing or biased. Also, be sure to pre-test the questions with the actual population that will be surveyed. That way you can observe respondents, grapple with the questions, and you can fix points of confusion or other problems before the survey goes into the field. So you can see that writing clear, neutral questions is actually a lot more difficult than it might seem. Certainly no single person can write a perfect questionnaire. But we can get closer to perfect by thinking carefully about these best practices. Microsoft Mechanics www.microsoft.com www.microsoft.com