Speaker 1: Hey, everybody, welcome to another episode of Tim and Tim Talk. It is so great to have you join us and we have a great episode in store for you. But first, I can't do this without my amazing co-host, Tim Kerbaviz. Hey, Tim, it's great to be here with you again, as always.

Speaker 2: And what are we talking about today?

Speaker 1: Well, Tim, what really makes this episode pretty exciting is we're going for a new format here. We're going to do a deep dive into a particular project. And this is actually an amazing project. Google I.O. is the is the premier Google developer event. Normally, this is done. It's a sold out event with thousands of people throughout the globe. But this is the first time that Google I.O. was 100 percent virtual. And so there is a there's a couple of main components to this. There's the keynote, which was done on site. But Tim, the the Google events team challenged us to come up with solutions and tools to do a 100 percent virtual first production for them. And Tim, what were the our main roles and responsibilities that we had to achieve?

Speaker 2: So we were asked to produce these live broadcasts of live Ask Me Anything AMAs, which are live Q&A panel discussions and live workshops, which are presentations and coding demonstrations that involve live slide and live coding demonstration content from the presenters laptops. And so we had these components that were fully live and, you know, involving people all over the world. What were some of the other challenges that we faced or questions we had to answer when we got to this?

Speaker 1: Right, Tim, let's just call this the project of a thousand questions, problems, solutions. I mean, I think every time we made a churn, a new question came up, a new decision had to be made. The first one, the first one that you and I had to decide was, is it going to be a cloud based or is it going to be an on prem? Now, because of the newest technology, we can do cloud, but do we go the traditional route with on prem? Tim, let's sort of let's take our viewers through our thought process of why we ultimately went with one decision versus the other.

Speaker 2: So I think we should define some terms here, Tim. We're talking about when we talk about the on prem versus cloud, we're talking about basically the control, the video control rooms. We're talking about, you know, taking video inputs, processing them through a video switcher and then outputting them to the stream. And, you know, with those inputs, we had our presenter content. We had our cameras from the presenters. We had our live screen share content from the presenters. And then we had some prerecorded video elements. And we had this across four simultaneous tracks across multiple days. And so we had this challenge.

Speaker 1: Not only days, right? But I mean, overnight, 24-7, what system is robust enough to handle 24-7, right?

Speaker 2: Right. And we were looking, of course, you know, Google is a cloud based company. We were really looking at whether a cloud first approach would make sense and actually hosting this with a virtual switcher in a cloud environment. And there are a lot of advantages to that, right? You get the reliability of a cloud system like Google Cloud, where you have essentially, you know, a data center with a whole team behind it, powering that technology. And, you know, you have the ability for people all over the world to control it. And so we had the potential to have our crew distributed all over the world. And so we had the potential to have people in multiple places controlling systems together and collaborating over the internet. And that's a model that works really well for a lot of productions. This production had some unique challenges. And we were building some unique systems, right, Tim? And so... So many of them. Right. And so, you know, that became almost a training challenge. And so one of the reasons we went with, in the end, an on-prem solution and built studios out, four studios, like, in fact, the studio behind you, right, was one of those studios, right? And so, you know, we built out systems like that, you know. But, you know, we really wanted to make sure that we still took advantage of these digital tools and technologies. And so, in addition to actually using some cloud production for some of the interstitial content, we used the same technologies that we would have used in the cloud environment, NDI, SRT streams, you know, software-based processing in the on-prem solution, including some of the hardware you see behind you is some of those software tools. And we did also then process those streams through cloud-based distribution networks that we built for the show.

Speaker 1: And I think it's worth talking a little bit more about the advantage, because it's such... The ability to do a cloud-based virtual switching production is exciting. It's revolutionary. It's basically... It's reached its stride in the past couple years. But as you mentioned, Bespoke Solutions meant that we had to train crew. And it's just... It was a significant challenge when we looked into the logistics of how do we train each individual position with our Bespoke Solutions. Because we're not talking of a small crew. We're talking crews of dozens and dozens and dozens. So, essentially, we didn't have months to train crews to do the solution and also make sure that they had the resources at their place to do it. A few of our technicians do have remote control capabilities, beautiful setups. But not every... Some folks just have a laptop and multiple monitors. So it's easier that we provide everything. And we also provide the training to bring them up to speed on our specific solutions.

Speaker 2: And to mitigate those issues, obviously, we weren't in Google's data center. But we did set up a system with multiple redundant internet connections and a whole system of failure checks and balances. So this is to make sure...

Speaker 1: One of the decisions that we... One of the checkboxes that we looked at was reliability. And because we're in our control room, in our studio, we have dual ISP connections. We have a gigabit symmetrical fiber line. So that's huge right there. And then, Jim, what's the second solution?

Speaker 2: So this was something, obviously, we talked about for a while. And actually, that came together for this production was going with a fixed wireless, a WISP, point-to-point wireless solution that has a dedicated network connection, business class network connection over the air from a nearby tower that gave us essentially a secondary ISP with a totally separate path leaving the building. So it wasn't following the same path. And so if the fiber got cut, if something happened, we still had that reliability of that.

Speaker 1: We had some ridiculous conversations of what ifs, right? I mean, one of them with fiber on a pole. Well, what if a truck hits the telephone pole and knocks it down? That's a one in a million lightning chance, but it's still a chance. So now with the second ISP leaving a completely different path, essentially using the sky instead of the ground, we now have a robust backup solution. So we felt comfortable with having those dual streams.

Speaker 2: And Tim, this was in addition to the crew on site, we had presenters and audience all over the world. What were some of the challenges we faced with this globe-spanning event?

Speaker 1: So Tim, are you saying presenting to a global audience and bringing in global presenters is a challenge? Not that I have any experience in that, but maybe a little bit. Well, you're right. So how do we bring in... This is a true, 100% authentic virtual experience. We had no presenters come in and we had no audience locally. 100% virtual through and through. And so with this, we're going to APAC, we're going to EMEA, we're going to the Americas. So how do we work on a global 24-7 time zone where we're presented basically from California, from the Bay Area? So Tim, we looked at some interesting problems of, first, how do we communicate? Because the presentation, and once they deliver, that's the easy part. It's all of the legwork and preparation to get to that point that is the challenge.

Speaker 2: Definitely. And one of the first challenges was, how do we get all of these presenters all over the world, who are often presenting with people in their own panel from a different time zone, how do we get them in rehearsal? And we needed rehearsals for them, we needed rehearsals for our crew. How do we schedule that? We looked at... There's some very popular tools like Calendly, which are designed to have one-button press to schedule a meeting. But I think we ran into a challenge with Calendly in particular because it was a third-party tool. What was that challenge we ran into and how did we address that?

Speaker 1: So if you're a production company or if you're a producer in a large company, you are very familiar with what you are and what you aren't allowed to install on computers. And basically what you are allowed to install is practically nothing and what you aren't allowed to install is practically everything. And so the problem with Calendly is it wanted to read permissions and that permissions basically opened up a huge chain of review process that unfortunately we just didn't have time to do that. But the events team at Google, amazing folks, absolutely great to work with, they did some digging around and they actually stumbled upon a tool. So Google has a similar tool internal to their ecosystem that basically does the same function. So that's awesome. And this is a big lesson learned is the best tool for the job is a tool that you can use. It's not the best tool that is in existence. And so once we discovered that this tool was there, it worked extremely well and it made communicating with presenters so easy. So Tim, let's talk about that. How did we deploy this tool and how did we use it?

Speaker 2: So this was a tool built into Google Calendar that basically let us create blocks of time on our studio calendar and then create time slots within those blocks for the presenters to click a link in their calendar and choose what time worked best for them. The big advantage there was if it didn't work anymore, they can click another link and reschedule their own time block. So Tim, amongst the challenges with scheduling, with calendaring, these all seem like documentation challenges. And I know we had a lot of documentation challenges with this show. What are some of the ways that we created and managed documentation to make sure everyone was always on the same page, both literally and proverbially?

Speaker 1: Yeah, and this was the biggest challenge I would say that we had to overcome. And that is integrating the client communication. As you said, they had mission critical information within their documents. But what was also mission critical for our production team oftentimes banded over multiple documentation. So we needed to develop a system that looked at each individual doc and then we curated that essentially into a nested view. Now that alone is good, but not good enough. Because in a show like this, there's constant changing, there's constant movement, there's rescheduling of time. There's presenters for whatever reason that no longer can present. There's new presenters that are being added. So how do you keep this information now that is constantly basically a living document? So essentially all of our documents for the crew were basically what we call living documents. It pulled information from another doc and it absolutely worked really, really well. It took a long time to perfect the system. But once we did, it was super easy to distribute just the relevant information to the crew. Because the crew, while we're running four or five studios at a time, they only need to know specifically what's relevant to them and just the minimum information to do their job perfectly.

Speaker 2: Totally. And we had dashboards for every crew position that showed them just what they needed to see, just when they needed to see that. And among those dashboards, in addition to show information for the show crew in the control room, we had dashboards for our recording engineer who was QCing recording. We had dashboards for one of the things that I was doing was managing the streams and monitoring all of the various streams as they started and stopped. And then we had another set of documentation for training our crew. Let's talk a little bit, Tim, about that and how we made sure that all our crew was on the same path towards success.

Speaker 1: Well, we refer to these checklists as a pre-flight checklist because you know why, Tim? I mean, it is as important. Anytime you're going in the air, you want to make sure that you're doing every single check. And even if you've done this checklist 100 times or 1,000 times, you're still making sure you look at the pre-flight checklist to get every single step and not rely to memory. Because the crew is absolutely amazing, but we want to make it easy for them. And they actually really appreciated the fact that they knew exactly what their roles and responsibilities were. And by systematizing, we could also deliver a continuous experience, a continuity amongst all of our control rooms. So it didn't matter if speaker one went into control room A, B, C, D, they had the exact same experience at the same time. And that's not easy, right? Because we have to train dozens and dozens of technicians to a specific formula just for this show. So by documenting, by spending enough time training the crew up, we had great success with that because we had the time.

Speaker 2: Exactly. And one of the other documentation challenges that we faced was presenter facing, right? And the Ask Me Anything panels were answering questions live from the audience as they were asked on Twitter and in the chat on the live stream. And we were actually bringing those questions up live on air in a box at the bottom of the screen. How do we manage handling those questions?

Speaker 1: Well, it's interesting, Tim, is like, not only did we bring those questions in, but we didn't use a fancy purpose-built broadcast solution. This is going to shock some people, Adobe Premiere. I will say that again. We used Adobe Premiere to bring in live questions. In a later episode, we'll talk about how we use that. But I think what's important in this one, Tim, as you mentioned, is Q&A. How do we communicate? Because oftentimes these hosts, they should not be looking through the questions and filtering out that. That's, each segment had a, essentially, like a project manager to look through questions for them so they could focus on the relevant ones. And then our doc was very simple. We're going to go into this in a little bit later, but essentially red light, yellow light, green light. What does that basically mean? What question has been asked? What question are we asking now? And what question are we asking next? Those sounds simple, Tim. But when you're working with a large team, that can actually become a very complicated process with live questions coming in all the time.

Speaker 2: That's a super complex process, Tim. We've tried to simplify it as much as possible. Speaking of simplifying, we've talked a lot in this episode about some of the setup and the documentation process for this show. There's so much more to talk about on this production. I think though that we have to, we've run out of time here. What are we going to talk about in the next couple episodes and how we're going to make this really interesting for our audience?

Speaker 1: Well, yeah, well, so we're going to talk about the challenges of bringing in each presenter. We're also going to talk about the challenges of how do we share content? Our initial idea, Tim, that we really, really loved, we had to throw it away. It was way too complex. We'll talk about that. So we'll also talk about the live Q&A. We'll talk about how we used Adobe Premiere. And we'll also show you that doc that we used to achieve that. And Tim, lastly, what's near and dear to your heart is we're going to talk about the thing that scares live engineers all across the globe, starting and stopping the stream. And now let's times that by not even figuratively, hundreds of times. Tim, how did you do that?

Speaker 2: Super scary process. We'll talk a lot more detail about that. For now, I think we better say goodbye. Remember, we're available on all the social platforms at Tim and Tim Talk. We will see you next time. My name is Tim K. And I'm Tim Kerbavis. And we talked. See you next time. Bye. Bye.