Speaker 1: Using Blueprints to Align Learning Objectives and Assessment. Hi, my name is Glenn Johnson. I'm an Instructional Designer here in the Department of

Speaker 2: Statistics at Penn State. And I'm Whitney Zimmerman. I'm a STAT 200

Speaker 1: instructor and course coordinator. And this is going to be our virtual poster. We'd like to start off with talking about how we align everything by using this diagram that shows how the learning objectives are related to the activities, which are related to the assessments, which can point back to the learning objectives. Everything should be aligned. We just want to make sure that everything is coordinated. We take a blueprint approach to figure out how to put this together and really make it work. The course that we're going to look

Speaker 2: at today is the online version of STAT 200. This is one of the undergraduate level introductory statistics courses that we offer through Penn State World Campus. It's a completely online course. What we have here is a screenshot of the first page of our online notes for a regression lesson. On the first page of each lesson, we like to begin with a quick paragraph or two that introduces students to the topics that will be covered and how they relate to previous lessons. We also clearly post the lesson learning objectives. These objectives are written in terms of what we expect that students will be able to do at the end of the lesson. With these learning objectives clearly written and presented to the students, we are communicating to them what the key points of the lessons are. Students are instructed to read the online notes and then attempt the homework assignment in MySTATLAB. MySTATLAB is an interactive Pearson product that we use for homework assignments. Instructors can select questions from a large pool of questions from our textbook and other Pearson books. Then students complete the questions at their own pace. These are the questions that I selected for my students to complete last semester in lesson 12. Our lesson learning objectives do not directly match up with the sections of the textbook, but later I'll show you how I try to match our learning objectives to the MySTATLAB questions. Each learning objective should be covered in the online notes and then again in either the MySTATLAB assignment or the student's lab assignment. This ensures that students who are following the recommended sequence of activities are exposed to the content at least twice before they take the quiz. Similar to how I matched up the learning objectives to MySTATLAB questions, I do the same with the lab assignments. The lab assignments are different from the MySTATLAB assignments in that they tend to emphasize the use of Minitab Express, which is the statistical software required in this course. So we've talked

Speaker 1: about homework and lab activities. The quizzes will build these quiz questions up and use Respondus as a way to manage all the quiz data. The questions, the answers, the feedback, and that kind of stuff really helps us manage this data really nicely and efficiently. Then we can then publish it out to Angel or Canvas or whatever course management tool you're using. It becomes a really convenient way to do that. You can see, pretty small there in the title section on that Respondus on the left-hand side, that there are different question sets or different categories, and that's reflected then again in the Angel and in Canvas as well. To show you what my alignment documents typically

Speaker 2: look like, I just take a list of the learning objectives, and sometimes if there are objectives that cover multiple topics, I split them up into multiple lines, and I check off whether or not they appear in the MySTATLAB assignment, lab assignment, and quiz. Here we can see that each learning objective appears in at least the MySTATLAB or lab assignment. Most appear in both. And all learning objectives are covered in the lesson quiz. Again, I do this to make sure that students are able to practice each task before being asked to do it in the quiz. There should be no surprises at the end of the lesson when students sit down

Speaker 1: to take the lesson quiz. So what we have in the notes is then aligns with what we have in our homework, our lab assignments, and our quiz. After the lesson is

Speaker 2: complete and all students have submitted their assignments and quizzes, I like to analyze the quiz results to see if there are any themes. This informs the materials that I make to help students review for the midterm and final exams. It also informs changes to the online notes. Here we have the results of the lesson 12 quiz. The two statistics that I typically focus on are discrimination and difficulty. The discrimination index in ANGEL is the point by serial correlation between the total quiz score and whether or not the question was answered correctly. Other course management systems may use different methods for computing this. What I look for are primarily negative discrimination indices. This typically means that there's something wrong with the item. Either it's incorrectly coded or just a very confusing item. Because what a negative discrimination index means is that the students who are getting the question correct are those who are doing worse on the quiz overall. The difficulty index, which is how these lesson 12 items are sorted here, is just a measure of the proportion of students who got the item correct. With the questions labeled by topic area, I could see that students seem to be struggling most with computing R from R squared. The discrimination indices are high and I've looked at the actual items and they seem fair, so my conclusion is that these items are difficult. I will share some more examples of going from R squared to R before the final exam and I should update our online course notes with a few more examples of this. Next semester I will also add another question on this learning objective to the MyStatLab assignment so they can get more practice. These learning objectives and lesson quizzes then serve as the basis for a

Speaker 1: midterm and final exam blueprints. So once we've got all the quizzes made and you can see on the left-hand side we've got the learning objectives there or the categories for those learning objectives for Lesson 7, Lesson 8, Lesson 9, and so forth. We can then put those together, choose those learning objectives that we want to cover in those midterms or in the final, and then use a spreadsheet like this and you can see on the right we then assigned our different instructors that we have teaching our different sections. Many Hands make light work on this and then we can, it's very easy for us to, you know, add new questions, maybe discard a few so that final is refreshed each semester. And it's really easy then in Respondus then to go from the data that we have to publish it out into a Word document once that quiz or that midterm is put all together so that the instructors can review it very easily going through Word to read all the questions or just the ones that they had updated and that kind of stuff. So it really makes it an easy way to kind of use the blueprint from each quiz to help construct our midterm and then the final. And here's an example of the analysis that's very similar to the quiz item analysis only for the midterm and you can see on the left-hand side the question areas on general theory, interpretation of confidence intervals, again reflect those different areas that we've identified as learning objectives and those questions that were associated with those. And then it's easy for us again to end use the difficulty or discrimination indices to determine whether or not the questions are good or whether they're difficult or whether students should have learned these and we just need to present them differently or give more examples in our online notes. So I think the analysis is kind of brings it full term so you've got the learning objectives informing what the activities and the assessments are and then we kind of go back a full term here and took a look and see how well we're doing and how well our students are doing. Well that's it for our poster here. If you have any questions here's our contact information. Don't hesitate to reach out and connect and call or email and ask your questions. We'd be happy to share any of the information related to what you saw here in our in our poster. Thank you.