Speaker 1: Thank you. I'm going to start off with how we got here. About two years ago, I got my performance review from my manager. It went relatively well. Some good things I did. Some things I needed to improve on. I started walking out of the room. And then she said, oh, one more thing. This is never good. She said, I want you to work on this thing. You have a, quote, verbal anomaly. And I said, what's a verbal anomaly? And she explained that sometimes when I was watching presentations, or watching videos, or something that was like talking about something that I was relatively passionate about, and I didn't like the way it was going, I would make this sound. And it sounds like this. Oh, no. And I didn't really realize that I was doing it out loud. Which, by the way, if you ever have this problem, don't do that. And she said, don't do that. And whenever you feel like you need to do that, go write the presentation that you thought should have been given. And that's where this comes from. So shortly after that, I was put on a project where I was told to sort of go around the world, visit agencies and advertisers that were doing something unique with data. And then if they would explain to me how they got there, how that process worked, how the team worked, I would help them with more data whenever I could, whatever I could find to help them accelerate, essentially. It was sort of like a, you know, tit for tat kind of thing. I found that a number of things were happening around the world. And they were happening in these little petri dishes of teams. It was pretty crazy. There was a team in Germany that was being really smart with search data. There was another. There was a team in Los Angeles that was doing really smart stuff with video. So we started experimenting ourselves and I provided data wherever I could, within our policies, of course. And one of the things that I found is that people would keep asking me for data. They said, you have data, we want to use it. And it was sort of this general request. And what I would do is I'd say, okay, fine, let's sit down. I'm gonna grab a whiteboard and I'm gonna say, literally draw a spreadsheet and say, what do you want that to look like? What, just show me, like is it, what numbers, what is in that data at the granular level? And about half the time, they couldn't actually define it. They couldn't actually say what they wanted, because it was an awareness problem. And of course, in the back of my mind, I said, no. The other half of the time, I found that the data was available, publicly, was easily used, and was accessible to almost anyone, almost always for free. And this was really frustrating, because these were really advanced advertisers, but they weren't connecting the dots on some of the stuff that we had and public open web data. So I'm gonna make the case, essentially, that one, is that open data is helpful, and it is out there way more than you may expect. I'll show you some examples, but also talk about the use of machine learning and artificial intelligence to sort of process that data in smart and helpful ways. The argument here is that essentially, rich data sets are really important, because they often give you insight, not necessarily performance. I think measuring performance and measurement in general media is critical to spend. If you're spending, you should be measuring. But there's tons of insight that's available that isn't necessarily used for measurement. There's a number of things that go like this. So the idea here is that lots of open data processed the right way is a major business differentiator. I've seen it happen around the world. We're seeing more and more of it, but right now I'd argue it's sort of like a Petri dish stage. And you can basically future-proof your business, which I think is probably what a lot of us are worried about, moving into a more complex, more always-on world. A couple definitions, one is data science in general. These are buzzwords that have been thrown around a lot lately. It's mostly about finding insight, I think. You can process with machines a lot of optimizations, but I think finding something interesting is where data science comes in. Machine learning, which also gets often mixed up with artificial intelligence, they are a nested sort of argument, is good at predictions and classifications. And so that's a part of possibly finding insight. And the last part is when we use the term AI, we also mean is that it produces an action. And that's why, for example, at Google, we say machine learning, we talk about processing data. But when we talk about the Google Assistant or the Google Home, we say AI. Because it produces interactivity and it produces an action based on what you've said. So, a couple things. I'll show you what we're doing to leverage AI, some of the experiments we've done. And these are all things that can be done by any agency advertiser that has an open either cloud account or data science team. I'll show you how you can build your own, a couple things. I'll talk about the ability to build your own insight machines. How machines can process emotion, which is kind of fun. How we're experimenting on how to measure a brand as a whole. And essentially, how we're processing insights for video, which is kind of like the most complex frontier. So, let's get started. I'll talk mostly about ads. This is usually in the paid sector, but not always necessarily. A couple things. One, how many of you know what natural language processing is? A couple of you, good. NLP or NLU sometimes it's called, I think is one of the most underutilized technologies in advertising. It's somewhat utilized, there's just so much more we can do. What it does is basically machine learning deconstructs a sentence into something that's much more consumable by a machine. So, it may understand why things are negative or what words you use to describe certain words. It does still struggle a little bit with sarcasm, which is a major complaint about natural language processing. My counter to that is most people struggle with sarcasm. So, we've used NLP to try and figure out what works in text ads. Which is not exactly the sexiest of all creative, but it is the most functional. And we found that, for example, when we looked through airline ads, I took billions of impressions, hundreds of thousands of different ad copy variants, and found there were patterns associated with click-through rate. One of those was using the word price or airfare, as opposed to language like enjoy savings. Enjoy savings is a corporate language that nobody really uses in the real world. And it turns out when you're in a very price sensitive category, it's important to be direct about price. We also found that, for example, mood detection. We said if the ad had an imperative tone, which means it's kind of very action oriented, it performed better. That makes sense, right? Nothing shocking there. We've also used this to optimize on our platforms for our pixel hardware line. We found, for example, that talking about words that have to do with quality versus exclusivity. Quality tends to do better. And verbs, talking about what it does or what it can do, versus what is described as, which is seen as a lot more wishy-washy to most consumers. We had a speaker earlier talk about getting smarter with keyword intelligence. This is one of the smartest ones that I saw on that little trip, was if you take keywords from your e-commerce site, client site, whatever, anywhere where there's a user ability to enter in their own text. And you map them on what they search for in the same session. This could be on an e-commerce site, for example, or a brand website. And you map them together with graph theory. You can find patterns in how consumers think. What we did, this agency took that data, mapped it out, and mapped it over a year, and then found that there were certain patterns. That cluster in the center left there is actually Christmas lights. And they found that Christmas lights was also often searched for with Smart Home. What people were starting to do is say, I wanna put up Christmas lights, but I wanna be able to turn them on with my Google Assistant or something like that. Or I wanna be able to time it or run it from my phone, which is kind of a clever DIY project. They actually used this data to change around their physical stores and test it out. Does the way people search reflect how they shop in a store? And it did. You can also do sentiment analysis with natural language processing. This is relatively well known. It's been around for a few years. I'm surprised how few people do it. We basically take YouTube comments, which are available, again, publicly, on our API, and process and say, this was positive, this was negative. Here's how they talked about brands, actors. Here's how they talked about the product. And they basically can find all this insight and track it over time automatically. And this is all free. The only thing you have to do in the cloud API is if you're going really crazy on it, I think it's like pennies, right? This is not gonna break the bank. We use this to evaluate how people feel about, especially movie trailers in the entertainment industry. We know if they like this actor or they like the character. We often map the character and the actor that plays it to see if they're who's, is the character bringing the actor up or is the actor bringing the character up? Which is kind of an interesting thing. We also look for words like video, videos, ads, and ads, commercial, commercials. We look for the singular versus the plural, because it'll say, I hate commercials, but I love this one. Now we know that this was unusually effective, at least they'll tell us that, right? The next one we're starting to do is to try and process emotion. I wanna be clear, machines do not feel emotion, they just read it in others. But what we can do is kind of get an idea of how things are being perceived. And the reason I love this so much is because it's all about efficiency, not necessarily about insight in this case. When we trained a TensorFlow model to understand the emotion that was being expressed in emails, customer service emails, we were able to say, okay, this person's angry, this person's sad, this person's happy. We're able to do that with relative confidence. What we did with that data is we said, let's organize our customer service team by emotion. And say, okay, Mike is good at dealing with angry people. We're gonna send all the angry emails cuz he's really good at dealing with angry people, right? And it turns out if you service the emotion they're having first, you actually get better performance, or sorry, customer satisfaction scores, about 15% better. So it's sort of interesting, we're using machines to say, okay, what is the leading emotion that this customer is having? And let's deal with that and then solve their problem, right? Which turns out, they like that too. One of the experiments we did was around using machine learning to try and map out a brand as a whole. And this is actually a ridiculously complex task. What we did simply was take the website copy of 100 different brands, map them together with machine learning, and said, don't score them. That's dangerous, I believe, in general. But actually map them out and see how they're similar amongst each other, and find the best fit. So for example, a lot of the telecoms were put together, retailers were kind of close together. And we found that, for example, Uber and Nike were really, really close together. And I thought this was a bug. I thought we had made a mistake. Went through, wasn't a bug. The machine said, Nike and Uber are really, really similar. And we couldn't really figure out why, because Reebok was one of the brands, and she was very, very far away from Nike and Uber. What we discovered when we really dug down into it is that there's four things that those two brands talk about, Android, iOS, time, and distance. And to a machine who didn't have Reebok pumps in the 90s, never saw the Air Jordans and things like that, and didn't have any of these experiences with these brands. All they saw was that Uber is an app company, and Nike is an app company, cuz they have Nike Run Plus. And they saw them in the modern day, in their own words, as very, very similar. So we can use this to say, if you're a brand, who do you admire? And we can actually track it in a relatively unbiased way, if you're getting closer or further away from that brand. One of my favorite things about that is that you can track it over time as well. So you can actually say, over the past year, we have become more like whatever goal brand you may have. My favorite experiments that we've been done are around using machine learning to watch video. And I'll say the technology right now exists somewhere like a toddler. It can point things out in a video relatively literally. So it'll say, at three seconds in, I saw the car. Five seconds in, I saw the cat. There's a dog, right? It's relatively literal. We're getting better with cultural things. So for example, for a long time, we had trouble discerning the difference between a baby shower and a birthday. Cuz visually, they were somewhat similar, but there was some cultural nuance in there that we had to train a little. And we're getting better at that. And in the next couple years, it'll become very, very good if it continues at the current pace. So what we do is we try and use this technology to watch ads that no human could possibly watch in a lifetime. So hundreds of thousands of ads, in this case, YouTube ads. And I love this simply because the idea is that when we have a little skip button, we have a metric now we can measure. And this, of course, this data is available in the YouTube Data API and the Google Cloud Video Intelligence API. You can do this yourself. What we can detect is casting, for example. We look at what gender we think they are. We look at the estimate age. At one point, we tried that. It was not fun. And we also look for music types. So you can identify colors, brands, and sort of cultural events or sort of things that are recognizable to the average person. And what we do is we have the machine watch them all. And it goes crazy. It literally takes tons of computing power to do this. Because videos are not just, think about it like this. A 15 second ad is 15 times 30 frames. And a frame is essentially a display ad, right? So it's processing a lot of data. But what's kind of cool is we can find these interesting sort of patterns and look and see in a given category what seems to work and what seems to not be working. I'll show you some examples. So the idea is that we can have better and worse practices, as we say. You don't say best and worst, because it's not necessarily like that. There's a lot of variability in creative. Talented creative directors have patterns, and those patterns tend to work. So I'll show you a couple examples. One is we broke down things into different product categories in most cases. So for example, when we looked at lipstick ads in the beauty category. And I'll tell you, there's a lot of lipstick ads that have been out there in market for a long time. And they're actually relatively similar. They're quite straightforward. It's all about the color that you're choosing. If you don't like the color, you don't like the lipstick. It's pretty simple. When we did analyze the things that weren't completely obvious, like showing lipstick in a lipstick ad is not insightful. It's true, but it's not helpful, right? What was, it did have to seem a lower skip rate, it seemed to retain audiences better, was actually fire or flames. I know what you're thinking. You're saying, I don't, how many of these ads could possibly have fire? There's a lot, I'm telling you. And this is part of the insight process, is I say, okay, okay, fine. I can prove that relatively that's true, maybe on average, if you were to randomly run it, you would probably do better off having fire or flames in your lipstick ad. But I asked people why. And I didn't get really a good coherent response until someone said, it's hot. And I was like, you'll see this theme continue on. When we looked at chocolate ads, and this was like, this was a great project. Because of watching chocolate ads, you get so hungry. We found showing the chocolate works, sure, but what really seems to work is showing any sort of clouds or smoke. Again, why? I don't know. I can't tell you why. I can guess. My guess is something like it's dark chocolate, the opposite is light and fluffy clouds or smoke, maybe. What's funny is that steam did not work. So I don't really know how to process all that. So we really wanted to throw things at the machine that we thought would kind of throw it off. Humor is really hard. Humor is one very culturally hard to get right. And machines don't really get it right. But what was associated with comedy movie trailers was one of the questions. We thought, this is gonna be a failure. We're not gonna be able to get anything else. And there was a very clear, distinct winner. In comedy trailers, bachelorette parties do the best. Now, that doesn't mean if you put a bachelorette party in your trailer, it's guaranteed to work well, no. But on average, across billions of impressions, it leans that way, which is kind of fun. Anything with formal wear seemed to do well. So the more formal the event, the more ripe for comedy it was, which kind of makes sense, right? Formality leads to humor. A couple of other things that didn't seem to work. We noticed that lens flares. Lens flares peaked in 2014 and slowly declined after that. I call that the J.J. Abrams effect. Black and white film noir doesn't seem to work that well, especially on mobile devices, which is kind of interesting, right? So there's something about black and white, it's kind of like basically glare at that point, and it's on a smaller screen. It seems to be correlated with people skipping more. So when I was going through that long project with the beauty and lipstick ads, I'll tell you, with this kind of type of research, what happens is 99.9% of what comes out of it is absolutely boring, useless, or inactionable. It will tell you that lipstick works in lipsticks. I'll tell you cars work in car ads. Shocker, not helpful. But when the machine does quirky things, you can find interesting things. So when I was in that beauty project, it told me that models and supermodels do really well in beauty ads. I would hope so, right? Like you would hope. But I didn't know actually what the difference was between a supermodel and a model. And I don't know enough about the industry to answer that question, so I asked a number of brands. And they told me, it's about face symmetry, it's about height, it's about cheekbone depth or prominence or something like that. And those, maybe, I don't know, I literally can't prove that, right? One of them told me supermodels get paid more. I was like, that's not wrong. But the machine wouldn't know that. So when we put side by side, and this is actually what I do for a job. I put side by side all the models and all the supermodels in these ads, side by side. I had to try and figure out what was the difference. And there was only one real distinct difference. And that was simply that there was wind in her hair. So while I love this technology and I think it's literally going to change the way we think about advertising and marketing, sometimes it's just better to just watch them and get your insight the old-fashioned way. So I wanna stress that when people talk about AI and machine learning. But it was so simple. So I'll wrap up and then I'll have a little bit of time to cover some questions. But you can do this yourself, and I wanna emphasize this. Almost all of this data is available for free. And some of the machine learning processing, you may have to pay if you're really going wild. But the fact is that any agency, anyone with the talent can do it. And I think that's important because talent is actually the biggest hindrance here. I think we've heard that today a little bit, right? I teach at Columbia University in the master's program for applied analytics. And it's amazing to see some of the talent that's coming out of universities right now is unbelievable, and they are completely lost. They're super smart, they're talented, they can do stuff that I can't even imagine doing. And then they say, should I work in finance or marketing? I was like, marketing, it's way more fun. And they're like, why? I'm like, because you can point to something you did and show your mom. That was the best argument they had heard, which is insane. We also know that, for example, these are really talented kids. And I say kids cuz they're relatively young, that's true. But they're really talented, but they're completely lost. And I think this is a management issue. We're losing good talent to other industries simply because we've done things the way we've always done things, and we're not even explaining why at this point. So these kids come in and go, why don't we do it this way? And no one gives them a really solid answer, and they leave. And I don't really blame them at that point. So one of the things we know is, for example, psychological safety is important. You have to explain why we do things the way we do. That's when they're very smart kids coming out of school, but they need that guidance. They need good management, and they need to be told, you know what would be interesting if we could do this, right? Be much more exploratory. Once they get into it, they do amazing work that literally could, I've seen students just messing around with things that would rock industries if they got that put out at scale, and they don't even know it. I think this is a management issue, and we really need to take our talent seriously. So a couple things when it comes to talent is, one is understand who is the actual end user of this insight. Is it an advertising brand? Is it an internal agency? Is it an external agency? Is it designed for public consumption? That sort of thing. The work that we did took months and months of engineering to put it down to a couple slides, right? There's a very, waterfall effect is very real. But understand what, who's the end user of this data or this insight, and then process that from that lens onward. Just like you would a campaign, you understand your audience first, right? The same thing happens with data and insight research. They can do lots of different things, these students coming out of school. They can model predictions, they can do classification of images, like we did with the video ads. And you need to say, you know what I'd love to do, is this possible? And often they'll find a way. There's a number of times where I've eaten my own words. I was like, I don't think that's even possible. Student comes back two weeks later going, I think I did it. And I look at him, I'm like, my God, you did it. Like, it's amazing what they're doing. And there's a huge gap between what their knowledge set is and our management style with said people. When it comes to that talent, I think we're moving past the era of the analyst. I've gotten a lot of flack on this one. I think we're moving to data engineering and data science. It's more complex, it requires much more technical knowledge. If your analyst work is based on going to someone's third party user interface, you're probably not getting the most out of your data. I think there's much more advanced ways to do this. Cloud options are amazing, some of the stuff they automatically do out of the box. These students are literally saying, should I use this service or that service? And sometimes I haven't used one of them, and I'm like, I don't know, tell me. Go try both, right? One of the students was like, well, it's gonna cost me $75 to set up the account. Give me my credit card, go. That's worth $75 to me to find out if you're right or not, right? They have to be able to code. I think this is a problem we have in the marketing industry, is that we think that coders are for building things or tools. I argue actually they're the ones that are gonna get the most value out of the data that you could possibly have. And there is data everywhere, if you're looking for it. I'll end with this and basically say AI will fundamentally change how we do creative. I think we're already seeing how it changes measurement. We're able to do data-driven attribution now for really well-tagged sites. But I think telling what is happening is the absolute fundamental baseline where you need to be. Artificial intelligence can't tell you why. I don't know why steam doesn't work in chocolate ads. I don't know why fire seems to work in lipstick ads. I don't know. I have to do a lot more research to find out that answer, right? The amount of engineering that went into getting to that point is actually quite a lot, and it's getting easier and easier as we go. But I think we need to spend our time essentially figuring out the why. This is inherently a human endeavor. Knowing that there is a bias towards darker chocolate, even in a milk chocolate ad, I think is a human insight. It turns out it just looks better on camera cuz it's shinier. Maybe, right? That's a human endeavor, and I don't see any technology out there that can do what humans do when they say, maybe this is why. The answer is usually just test it. But you have to know what to test first, and that's inherently a human endeavor. Thank you. Thank you very much. Thank you very much. Thank you. Thank you very much group. Thank you, everyone. Thank you. You're welcome to think about it. Okay? Thank you, everyone. God bless you, but we have to get a Ш TD, and that was our round two about the American Dream Coach. And I just want to thank everyone for getting the show on block, and I want to thoroughly congratulate Shawna and