Speaker 1: Hi everybody and welcome. Today we're going to talk about how to write measurable learning outcomes. Learning outcomes are the basis of any course design and it's the first step in creating a good course. So let's talk about what basic course design is. We begin with a course and the course runs from the beginning into the end. And as we begin planning our courses, we create our outcomes. An outcome is basically you determining what do the students need to do in the course. In relation to the outcomes, you also create assessments and the basic purpose of any assessment task is to distinguish students who have mastered the outcomes from those who have not. So any of your teaching strategies, assignments, and assessments are going to be driven by your course outcomes and that way we begin the development of the course with the end in mind. Now it's important to note that you can't have an outcome that doesn't have an assessment. You can't determine what do students need to do without measuring that they can actually do that. And likewise you can't have an assessment without outcomes. You can't just create a quiz or create an assignment that doesn't measure something specific as outlined by the outcomes. I call those orphan outcomes or orphan assessments. You don't want an outcome that isn't measured and you don't want an assessment that doesn't align with one of your outcomes. And this is at the course level. At the program level it's essentially the same thing. At the program level you're determining what do students who complete this program need to demonstrate? What do they need to do? And then the assessments, how can you determine that they have actually done that? Now the program level, everything's going to be higher level. The outcomes will be more general and the assessments are not going to be individual discussions or quizzes. It's more likely going to be something like a capstone assignment or something summative. Maybe a project that spans multiple courses such as a thesis or a dissertation. So now let's look at how we can start writing our outcomes. Outcomes need to be specific and they need to be measurable. So you don't want to write broad goals with no way to measure them. Let's look at a few examples of outcomes. For good or for bad you can determine, we'll talk through them. Here's the first one. Students will understand how an engine works. Is this good or bad? Think for a moment. And in reality we'll talk about that word understand and how it really has no place in outcomes at all because you can't really measure understanding. You can measure something that the students can demonstrate but you can't measure understanding or learning or knowledge. The students actually have to do something, demonstrate something. So I'm going to scratch that out. Say understand how an engine works probably isn't good but you could refine it somewhat. You can say students can demonstrate that they know how to disassemble an engine or students can identify certain components of an engine and how they operate and how they function together. Let's take a look at another outcome. So students will critically reflect on certain concepts from the reading. First and foremost I'm going to scratch out that word critically because it really has no meaning. And if it does have meaning then you should describe the meaning. What does it mean to critically reflect? So I'm going to say students will reflect on certain concepts of the reading. And I think that I want to dive further into that. I think that we can write that better. Reflect on concepts of the reading. I think we can turn that into something different like what does reflect do? How are they going to assess and analyze the concepts of the reading? And I'm assuming here that we're defining what those concepts are. But for now I'm going to scratch out that outcome as well. I think we can do better than that. So let's look at another one. Students will apply certain concepts to a real-life setting. Now I really like that word apply because that's something that we can measure. If they applied or if they didn't apply, I can assess that. And of course I'd specify the concepts. And then we're looking at to a real-life setting. That's where it gets a little bit shaky because what does a real-life setting mean? A real-life setting could be going on a fishing trip. It could be playing video games all day. It could be watching an extended cut of the Lord of the Rings marathon. You know real life means different things to different people and different people have different lives that they live. And so I want to be a little bit more specific. So I'm going to scratch that out. But we're in the right direction. So I'm going to say let's apply principles of servant leadership to a workplace setting. And I could even specify which workplace setting. It could be perhaps a managerial role or a meeting or a certain project. But the gist is that this is a bit more specific than the previous outcomes that we've looked at. And we're using an action word, apply principles. And we're specifying which principles and we're specifying where. And so this is specific and it's measurable. So the reason I like that word apply is because it falls very cleanly on Bloom's taxonomy. And for some of you this might be a review, but Bloom's taxonomy is a foundational component of creating learning outcomes. And we divide it into a hierarchy from low-level behaviors and principles to high levels of critical thinking. Now some people distinguish Bloom's taxonomy into three different peers. The lower level would be the entry levels of critical thinking, such as the BA level, the bachelor's level. The middle levels, applying and analyzing, would be like a master's level. At the upper echelons, you have the doctorate level where they're creating and they're evaluating. Now this was presented by Bloom as a taxonomy, as a hierarchy and a pyramid shape. But some people treat it more like a staircase. So they'll liken this pyramid model to a stairway where outcomes and objectives progress to higher levels of thinking. And so the verbs that we use when we create our learning outcomes are very important. Let's take another look at this Bloom's taxonomy and look at some words that we can apply. So at the low level, at the remember level, that might be an intro level course, a freshman level course, where you're just diving into a topic. You might have words like list, refine, or tell. And at the mid-levels for apply or analyze, you might use verbs like examine, investigate, organize. At the highest levels, you might be designing, producing, inventing, etc. So let's look at this, and let's analyze an apple at the various stages of Bloom's taxonomy. We're going to look at the early stages of development and apply the principles of learning to an apple for a small child. And so at the lowest level, we have remember. The child needs to remember certain things about apples, what an apple is. Apples can be red, they can be green or yellow, they can be small, they can be sweet or sour. Essentially, a kid needs to remember these certain attributes about an apple. They have to remember what an apple is and the characteristics. Next, they'll advance to the understanding level. So you might want to understand when is an apple right? In what climate do apples grow? And maybe nutritional things. Is an apple nutritious for us or not? And then they'll advance to the application. Are there different ways that we can eat apples? Maybe can we use apples for baking? How can you apples into a real-life setting? And then analyzing. So how is the skin of an apple different from the inside of the apple? Do we need to wash apples? Are there pesticides involved? And what happens when you add cinnamon to apples? Next, we'll move up to evaluating. So now we're looking at older kids, and we might assess how does apple nutrition compare to other fruits or foods? And how can you assess an apple in terms of the vitamins? Is it affordable? Do people have access to it? What does it taste like? And in terms of taste, where do apples score? And are there other foods that score higher than apples? And finally, we'll move up to creating. And so a kid might say based on my understanding and analysis, can I create something new with apples? Can I grow an apple orchard? Or can I write a cookbook that features apples in the recipes? So from this example, you can see that there are various applications of the same concept. We're studying an apple, but we're studying the apple at different levels of Bloom's taxonomy. Now let's talk about how we can incorporate technology into our learning outcomes, and how does that apply to Bloom's taxonomy. Now you always want to write your outcomes before you choose whatever technology you want to integrate, because educational technology is never the solution to a problem. It can be part of a solution, but you want to start with that curriculum alignment. Start with your outcomes, and then determine is there a technology out there that can facilitate what I'm trying to do in the classroom that can help the students to achieve those learning outcomes and demonstrate mastery. So this time let's work at the top. What can students do? They can engage in blogging, direction, producing, and there are various technologies that can help with that, such as maybe Canvas Studio or Panopto, some kind of screen recording, even PowerPoint, VoiceThread, EduBlogs. They can use Microsoft Sway to create an ebook type of interaction. There are various technologies that allow students to engage in creative outcomes. Next we'll look at evaluate, so maybe posting things, moderating. So you might have students use Microsoft Forms or SurveyMonkey or Poll Everywhere. At the analysis phase, you might engage students in mind mapping or curation of some sort. So they might use mind mapping tools like Padlet, or they can use other tools like Kahoot or quizzes. At the application level, you might have students do editing and uploading things, so Nearpod, PowToon. Again, Padlet would be good for that as well. And at understanding, you might have students doing tweeting, hashtagging, so social media, Twitter, or you can have them create things in Adobe Spark or Canva to demonstrate their understanding. Remembering is a lot of rote memorization, so bookmarking, copying. There are tools like Wacklet, Bulb, ThingLink, Padlet, and in reality, platforms like Canvas Studio or Screencast-O-Matic, any kind of screencasting software could be used at any one of these levels of the Bloom's Taxonomy. But I just want to reiterate, you always want to draft your outcomes before you choose your technology, because the technology should not be the focus. The technology is only a vehicle for transforming student thinking at different levels. So the question you have to ask is, how can edtech support different cognitive levels? Now I want to shift gears and look at the Bloom's Taxonomy in a different way. This is a table format. Now we're already familiar with this x-axis of remember, understand, evaluate, create, but in addition, some researchers have formulated different knowledge dimensions as well. You have rows A, B, C, and D. So we're looking at factual dimensions, and that would be dealing with knowledge, terminology, details, as well as conceptual dimensions. So we're looking at principles and theories, and then there's procedural, so techniques, methods, and then the metacognitive. So we're looking at strategies, context, and conditional knowledge. And so this gives us a grid table-like format, and a good way that we can assess how our objectives are falling across the entire course. So let's look at this. I'm going to show an objective here. The objective is that students will develop a new classroom behavior chart utilizing student-inclusive methodologies. And I highlighted the verb develop, and then the direct object classroom behavior chart. So I'm going to isolate those and identify that develop falls on number six on create for Bloom's Taxonomy. So in the cognitive process, I would call that six. The direct object would be the classroom behavior chart that aligns with knowledge type C, which is procedural. So they're creating some kind of process, and so I know that this objective falls on the table in category C6. And so let's mark that on the table here. I have one learning objective or learning outcome that falls on the creative column and the procedural row, so C6. And then let's just pretend like I have other objectives for this course, and I'm going to map them all out. So let's do that right now. You can see that they populate in the lower levels of Bloom's Taxonomy, and they tend to reside more in the factual, conceptual, and procedural knowledge dimensions. So this tells me a few things. And again, this isn't a real course, but this is telling me that most of the verbs are low-level Bloom's Taxonomy. I'm going to ask the question, is this an entry-level class, such as a bachelor-level class? Or if it's a BA class, is it one of the entry-level 100-level classes? Because the learning outcomes seem to be falling over there. And if it's not, if this is a senior-level class, then that tells me that I need to go up and revisit those outcomes so that I can move them up on the Bloom's Taxonomy so that they fall higher in the analyze, evaluate, and creative dimensions. And I see that that first outcome, the C6 that we talked about, it seems to be an outlier from the other objectives. And so I have to ask myself, is this correct? It could be. It could be a 101-level class where there's a lot of remembering and understanding, but at the end of the term, they have a capstone project where they have to simulate and they have to create something. So in that case, that lone outlier outcome might be appropriately situated. However, if it's an upper-level class, then I'm going to see a lot of these Category 1 and 2 on Bloom's Taxonomies, and I'm going to have to rewrite those so that they assimilate more of those higher-level Bloom's verbs. And I'm also noting that there isn't anything metacognitive, and so I want to evaluate, is that accurate? Is this class focusing more on factual, conceptual, and procedural dimensions, or should we be visiting that metacognitive? So I can't imagine a class where all of the outcomes would be evenly spread out over the entire table, because in reality, students graduate. We have a cognitive graduation. As they progress through a program, then they start applying higher-level cognitive skills, and when they get to those higher levels, then they're going to be doing less of those lower levels, less of the remembering and understanding, because they should have that foundation already. So this table is a good approach for you to get a high-level overview of the course that you're developing, and either verify that the outcomes are good, or you can adjust to accommodate where the outcomes should be. And so this is a basic overview of how to write measurable learning outcomes. Outcomes are the foundation of good course development. So I appreciate you being with me today, and if you haven't already, please subscribe to our channel, visit us on social media, and if you have any thoughts or insights, feel free to leave them in the comments below. And as always, I want to wish you happy teaching and learning.