Speaker 1: Let's talk about how to avoid some mistakes when designing questionnaires, when designing your survey. Let's start off by asking, first of all, what should you ask? What are the right things to ask? The main thing that you need to be asking are relevant questions. Do they measure the construct or the variable that you want to measure? And you want to avoid questions that are interesting but are irrelevant to your hypotheses. Now, you're supposed to ask demographic questions at the end so that you can better understand who is participating in your survey, and they're not necessarily related to your hypotheses. But apart from that, you want to avoid irrelevant questions. Secondly, you want to make sure you ask questions that are going to provide accurate responses. And to do so, the questions need to be simple, as simple as possible, so that most people can understand them. They need to be understandable. They need to be unbiased, not pushing people in one direction or in the other. They need to be unambiguous so that everybody interprets them in the same way. If they're ambiguous, people are going to interpret it differently. And they need to be non-irritating. They shouldn't tick people off because they're obnoxious or they're offensive or things like that. So, what are some specific things that we should avoid? Here's a list. The first thing is we need to avoid overly complex items, like suppose people were asked to agree or disagree with a statement. My boss has the personality necessary to lead a team in the development of new products and services appropriate for diverse markets. Huh? That's really complicated. Who knows what the personality necessary is to lead a team in the development of new products and services appropriate for diverse markets? People are going to just say, well, I like my boss, so yeah, he must or she must. So a lot of times, when you ask complex questions, it's just going to be reduced down, do you like whatever we're talking about or do you not like whatever you're talking about, which is very different. This is one of the problems with a lot of leadership measures, is they ask complex questions about somebody's supervisor or boss or leader or something, and people just respond with, well, yeah, I like the person, so I'll say something positive about them. So you want to avoid complex questions. You want to avoid leading questions that imply a certain answer. For example, if you ask the question, how fast was the car going when it slammed into the truck, you'll get a lot different response than if you ask the question, how fast was the car going when it accidentally tapped the truck? By using the word slammed, you're implying that it was going fast. By using the word tapped, we're implying that it's going slow. So our choice of words can influence the response that people will get, because a lot of times people don't know how fast the truck was going or they don't know something about something, so they'll look at the words and try to guess what the right response would be based on the vocabulary, so that would be a leading question. Now we also want to avoid loaded questions. Those are emotionally charged or socially desirable items where the socially desirable answer is available. For example, is freeing drug dealers worth the death of many innocent children? Oh, everybody would say no. But if you ask, should we continue to put nonviolent drug offenders in prison, which could be asking the exact same thing as the question here, people will give a different response because they're loaded questions, they're emotionally charged, and there's a socially desirable answer, so you want to ask neutral items. Okay, now D is you want to avoid ambiguity, unclear questions or choices, an incomplete range of choices or categories that are not mutually exclusive. For example, indicate your age, A, 20 to 29, B, 30 to 39, C, 55 and above. Ooh, there's all kinds of issues there. What if you have people under 20? Where do they go? What if you have people in the 40s or their early 50s? Where do they go? 55 plus is really a broad range. You might get far more people in that category than the others. There's a lot of ambiguity and inconsistency in that question. E is double-barreled questions. Double-barreled questions are questions that go off in two different directions, like a double-barreled shotgun, they kind of go off in different directions and you hit more things with a double-barreled shotgun. And these often contain the words and, or, that, which, because they have two clauses in it, or two subjects, or two objects. So for example, do you like your boss because of his or her leadership abilities? Ooh, that's double-barreled, but maybe I like my boss because, but it's not because of their leadership abilities. How do I answer that? Or maybe I like my boss's leadership abilities, but I don't like him or her as a person. And so people don't know what direction to take this question because it's going off in two different directions. That's a double-barreled question. Burdensome questions. Here's a list 10 words that characterize all your failed relationships. Oh, nobody wants to, wants to answer a question like that. So, cause that's really, that's burdensome. That would take a lot of effort to think of, okay, I've got to think of all my failed relationships already. That's kind of a bummer. And then words that characterize them. Well, there was me and me and me and me. Oh no. It's not a question that you're going to get good responses because it's just too hard. It's too burdensome for people to, to answer. And then you'd want to avoid questions that won't generate variants where most people will mark the same thing. So, how often do you help your coworkers? Never, occasionally, or often? I think almost everybody would say often. We're not going to get much variance there. We need to ask, have choices that would give us a very wide range of choices. Never, occasionally, often, very often, constantly. That would probably give you more of a range of answers. You want to avoid inapplicable questions that don't apply to the person that's taking the survey. For example, the question, was your first pregnancy easier or more difficult than your second pregnancy? Some of you could answer that, but probably most of you couldn't answer that because you haven't had two pregnancies. This assumes that you've had two pregnancies. And if you haven't, then you can't, can't answer that. So, if you do need to ask questions like that, you need to use what's known as skip logic when you design the survey, where you skip questions if somebody doesn't meet the conditions. So, only the relevant, only people that meet the conditions have the relevant questions. Now, generally, you have to pay for surveys that have skip logic. Google Forms, as of this year, doesn't have skip logic in it. So, you can only ask items that are relevant to everybody. Or you give instructions, skip the next item if it doesn't apply to you. But that doesn't work very well because people try to answer it anyway because they don't read instructions. Some people don't read instructions very well. Now, you want to also avoid ranking questions with more than four choices because they're burdensome. It's another example of burdensome. What I mean by ranking is saying, oh, list your favorite, your least favorite, such and such. So, an example would be rank the 50 states from the state you most like to the state you least like. Oh, that would take somebody forever. And they get to North Dakota and South Dakota, and they don't know which one they would rather live in. And it's just too hard for them. They could spend forever on it. So, avoid ranking questions. Now, if they have four choices, people generally can rank four choices. But when it gets up to five, six, seven, it gets really hard for people and too burdensome. You want to avoid force answering. That means requiring an answer, typically marked by a star in the survey. Open-ended questions, where people don't want to type in things, or controversial questions. So, if you had, what is your gender? And you just had two choices, male and female, there are a few people that are going to get really frustrated with that and really upset. And they don't want to do either, and they'll just give up on the survey. So, you don't want to make those questions mandatory. And for questions like, what is your gender? I typically add a choice, male, female, or a third, other slash prefer not to state. I don't ask for a fill in the blank, because TMI, too much information. That's not what I'm studying. But that allows everybody to choose something that they want, male, female. And then the third choice would be other or slash prefer not to state. And then, just avoid anything that's going to avoid, lead to missing data. Don't give people the choices, other, none of the above, no opinion. For example, do you like fried chicken, a yes, no, or other? Oh, a lot of people will choose other, and you can't do anything with that. Just have it, yes or no, or have it be a scaling question, going from strongly disagree to strongly agree. And then the people that don't really have an opinion, they'll give us right in the middle, neither agreeing or disagree. And you can still use their data. That works a lot better. So, this is an overview of some of the mistakes that you should avoid when designing questionnaires.