Speaker 1: Welcome to our capstone difference presentation. My name is Shalini and I'm here with my colleague, Elio Belfiore. Today we'll be sharing our translational research journey focusing on the development and implementation of an AI-powered chatbot designed to optimize the efficiency of obstetric pre-anesthesia unit at the Ottawa Hospital. In this presentation, we will walk you through our project's journey from identifying the need to developing an innovative intervention. We will cover the research methods we used, the development of our prototype, the steps we took to make it suitable for implementation, the impact of our work, and finally, our future directions and recommendations.

Speaker 2: Okay, let's jump into the background of our research now. Obstetric anesthesia plays a vital role in childbirth, encompassing pain relief during labor, vaginal delivery assistance, C-sections, and other obstetric procedures. C-section often happens to prevent maternal or fetal injury. It requires careful consideration of anesthetic techniques based on various risk factors, patient's preferences, and anesthesiologist's judgment. The graph here shows C-sections rates per 100 live births across Canadian province in 2021. Globally, the World Health Organization recommends that C-sections should not exceed 15% of all deliveries. However, in Ontario, the C-sections rate closely mirrors the national average of around 30%, reflecting broader Canadian trends. Over 90% of C-sections today are performed with anorexial anesthesia, which has greatly improved maternal safety. However, the fact that mothers remain awake during these procedures can still lead to significant anxiety. Additionally, a recent CIHI report highlights an increased risk of maternal morbidity and higher hospital costs compared to vaginal deliveries. Preoperative anxiety is a common issue among pregnant women, driven by fears such as death, postoperative pain, complications, medical errors, and not recovering from anesthesia. This anxiety is particularly pronounced among first-time and high-risk mothers. Many of them attribute their anxiety to a lack of information about the procedure, not knowing their anesthesiologist until the last minute, and having unanswered questions. Imagine a mother going to surgery feeling informed and confident because she knows what to expect. She's met her anesthesiologist ahead of time, and all her questions have been answered. This isn't just a nice act, it's essential for improving surgical outcome and ensuring a smoother recovery. But what happens when we don't address this anxiety? It can lead to numerous negative outcomes, including patient noncompliance, surgery cancellations, unanticipated perioperative complications, prolonged recovery times, and increased infection risks. The good news is that this anxiety is preventable. But how? By simply providing valuable information to the patient. This is why pre-anesthesia care units were created. They play a crucial role in optimizing health status to guarantee better operative results, inform and educate patients, and to offer opportunities to clarify questions. Research highlights the benefits of such setups in reducing anxiety and improving outcomes. We want to explore the current state more. For this, we set three objectives. Understanding the problem in the current system, gathering insights from stakeholders about the current state, and finally, identifying interventions in the current state. Our collaborating institution for the study is the Ottawa Hospital. At the Ottawa Hospital, which is a primary referral base for Eastern Ontario, about 7,000 births are managed each year. The obstetric anesthesia unit handles a variety of cases, around 57% labor epidurals, 25% scheduled C-sections, and 18% emergency C-sections. This variety creates a repetitive cycle of questions from patients, limiting the effective use of consultation time. Imagine performing pre-anesthesia consultation for all these people. While essential, this consultation can become time-intensive, especially in busy hospital settings, where the anesthesia staff work in a broad sort of tasks. So, how is this being managed? Hospitals in Ontario typically rely on three primary approaches. Referral-based systems, nurse-led consultations, and digital tools. Referral-based systems prioritize high-risk patients, leaving many without access to thorough pre-anesthesia information. Some hospitals assign nurses to conduct these consultations, which helps reduce cancellations, but can lead to heavy workloads and limit patients' choice. Digital tools like websites, FAQs, and info videos offer some assistance, but they can often lack engagement and personalization. Each approach has its strengths and weaknesses, but none fully address the needs of all patients. For our second objective, we interviewed anesthesiologists and anesthesia nurses, and we discovered that patients often asked redundant and non-clinical questions, such as those related to procedural logistics and general concerns. While important, these questions could easily be answered through resource materials. One physician even suggested that an AI could triage these non-clinical questions, freeing up their time for more critical work. For the third objective, our research led us to an innovative initiative at Toulouse University Hospital in France. They developed a digital assistant, a chatbot, to enhance pre-anesthesia consultation. The study concluded that using a digital assistant before the consultation reduces the duration while ensuring high quality and patient satisfaction. As we look at the challenges, it's clear that gaps in the current system are causing significant issues. Women and mothers often lack timely, adequate information about obstetric anesthesia, leading to preoperative anxiety. This anxiety triggers post-traumatic disorders and strain resources. So the desired state is that all obstetric anesthesia patients have access to comprehensive pre-anesthesia information at all times during their care period. The problem preventing the current state from becoming the desired state is that there is an efficiency in delivering detail and timely information. Hence, there is a need for an intervention to ensure all obstetric patients receive consistent, comprehensive information, reducing anxiety, improving outcomes, and optimizing healthcare provider workloads. As our project, we took on the intervention part by creating a prototype as part of our capstone project. Our proposed intervention is an AI-powered chatbot that engages with patients using artificial intelligence, provide ease of use with user-friendly interface, and offer 24 by 7 accessibility, providing support from pre-anesthesia consultation to post-surgery. Our chatbot aims to enhance patient experience and build trust by giving evidence-based answers. This brings us to a research question. Can an AI-powered chatbot be implemented in a Canadian hospital institution? Now let's look into how we did this. Our project followed four key steps, identifying the need, building the prototype, validating with experts, and engaging with hospital stakeholders to ensure that the chatbot meets DOH standards. We created a live AI-enabled chatbot, recorded its responses, and then presented for preliminary validation. Once the preliminary validation was completed, we proceeded interacting with key stakeholders to understand what the chatbot should meet to comply with the hospital ethical and privacy standards. We also submitted our REBE application, which will be necessary later on when we test the intervention with patients. We then proceeded to the next step, which is partnering with a web development group to make our chatbot a viable product. To represent our project in terms of Toronto Translation Thinking Framework, which is a guide and set of practices based on experiential learning methods, it falls within the abstract and experimental stages where new ideas were created and then translated into design and the creation of the prototype.

Speaker 1: Let's dive into the step two of our project, which is the prototype development. Our prototype chatbot development took place in four steps. Step one was to train the chatbot. Step two was to configure the chatbot. Step three was to conduct performance testing, and step four was preliminary validation. This process was completed with the support of our technical lead who programmed the chatbot for us. We trained our chatbot initially using the SOAP FAQ documents and the Ottawa Hospital's Guide for Obstetric Anesthesia Patients. The information was compiled into one document and uploaded to the chatbot's knowledge base. Then we configured our chatbot and made it AI-enabled using Lama Index. It is the framework we used to connect the chat GPT model to our chatbot using API key. This framework uses advanced method in natural language processing called Retrieval Augmented Generation, RAG. Below is a simple diagram to show our chatbot's configuration. When a patient asks a question, it first finds the answer from the knowledge base, which is the information we trained the chatbot with, and then the response is prepared and sent to chat GPT, the generator, to make it more personalized and friendly. The response is then given to the patient. Here's a mobile view of a working chatbot. It includes some accessibility features at the top and a pop-up message to inform users about the possible risks associated with this chatbot. Here's a web app view of this chatbot. We conducted two performance tests and found that in performance testing 2, there was significant improvement in terms of cost-cutting and efficient use of chat GPT model. The last step was preliminary validation. Ten anesthesiologists were presented with the chatbot's responses to questions and were asked to rate them. The data analysis indicated that 60% found the responses accurate, 80% rated them as useful, and overall, the responses were positive. But in terms of completeness of responses, there was a need for improvement. The third step in our project journey was engaging with hospital stakeholders. During our REB submission, we were asked to undergo third-party technology assessment and approval from the tech specialist of DOH. We then connected with key stakeholders, including data security and privacy lead and data governance and compliance lead. These stakeholders guided us with the necessary steps to make our chatbot compliant with hospital standards. These steps include designing the chatbot with clear functional and technical requirements, establishing a comprehensive data management plan, risk management strategies for AI behavior and ongoing quality control. These requirements demand high-level technical expertise. That brought us to the final step in our project, which is the web development partnership. Our web development partner is KeenEthics, an ethical web development group based in Eastern Europe. Our partnership developed as we provided them with necessary user stories and engaged in several discussions. After several iterative cycles, we identified all key requirements. This includes compliance with SOC 2, Service Organization Control 2, and WASP web application security project standards. SOC 2 ensures that patient data is securely handled, stored, and transmitted according to strict security protocols, while WASP focuses on safeguarding our web application against vulnerabilities and attacks. To meet these requirements, we have implemented several critical features. For example, data security measures include pin login or multi-factor authentication systems. Access controls are managed through an admin panel with roles assigned to super admins and admins. Confidentiality is ensured through data encryption, and audit trails provide comprehensive logging of user interaction and admin actions. Regular compliance audits are enabled through version control, API management, and regular updates. If we incorporate these features into the chatbot, let's visualize the front-end prototype. Welcome to the Harness chatbot system. This is how the login page might look like for general users, such as patients. It includes a user ID and pin system, which will be provided by the admin. This is a visual representation of the chatbot interface itself. This is the Harness chatbot admin super admin login page. Admins will use their institutional email and create their own password. Admin profiles can only be created by the super admin. Here, a multi-factor login system can be enabled. Welcome to the Harness chatbot studio. This is the landing page, which shows analytics. On the left side is the navigation bar, and on the top is the quick actions toolbar. On the top right features accessibility options, notification settings, and an alert system. There is a filter to set time frame to derive analytic data, which shows total conversations with the chatbot, engagement rate, resolution rate, escalation rate, and abandonment rates. We also have a customer satisfaction score, which will allow the users to rate their experience at the end or during the conversation. The chatbot can be disabled if needed. In manage user section, we have three options. Generate user ID, which will be done when the patient is registered for obstetric anesthesia. Add admin, which is what we are viewing currently, where only super admin can create and modify admin profiles. And manage members, where authorized personals can view and download information on user login and admin actions taken to date. The next slide is configure, where only authorized personals can access this feature. It involves setting up environments, defining parameters, and configuring how the model should behave or interact with the data. This includes setting up intents, adjusting thresholds, managing API keys, and configuring privacy and security settings. Under security and privacy, we have toggled switches for every setting, including multi-factor authentication, data encryption at rest and in transit, enforced HTTPS connection, and so on. These are just for presentation purposes. This may actually differ when we are launching the product. Let's look at the technological pathway we anticipate our project will take. So far, with the support of translational research project grant, we completed developing the prototype, conducting preliminary validation with clinical experts, submitted RAB application, and developed toolkit, and partnered with Keen Ethics to work on the chatbot design requirements. With this, we have completed our graduate project deliverables. Our next step after this, after our graduate timeline, could be finalizing the design requirements, estimating the cost, and applying for more funding during fall 2024. Following that, we aim to have minimal viable product ready for user acceptance testing. We will then conduct internal and pre-launch testing. Following successful test outcomes, we plan to present the results to TAHOIST stakeholders, make final adjustments, develop risk management and ongoing monitoring plans, and launch the product by May 2026. The project closure is expected around the end of 2026. Our capstone project has had a significant impact in several key areas. Firstly, it has highlighted the importance of leveraging AI to address gaps in patient education and pre-anesthesia consultation. Secondly, it has set the groundwork for future integration of AI tools within the hospital settings, showcasing the potential for improving efficiency and patient satisfaction. In terms of knowledge dissemination, we have completed several key activities, including poster and oral presentations, a project website, and an implementation toolkit to assist clinical researchers. Ongoing efforts include abstract publication and an upcoming presentation at the annual International Obstetric Anesthesia Conference. Our future directions and recommendations include expanding the chatbot's application to other clinical areas for broader patient education, using machine learning to analyze chatbot responses for user satisfaction and behavior, and advancing the chatbot's maturity based on Boston Consulting Group's AI Maturity Pathway.

Speaker 2: This project would not have been possible without the advisors and mentors we gained during our capstone journey. We are grateful for the opportunity to have networked with so many people who contributed to the success of our project. Here's a stakeholder map of the individuals highlighted in their position from each institution and the work they have contributed to our project. We want to give a special thanks to our PACT team for guiding us whenever we had to pivot, helping us define our project goals and deliverables, and providing a supportive environment. A heartfelt thanks also goes to our wonderful TRP community members and our cohort members for their invaluable support and collaboration. Lastly, our technical lead has been instrumental in developing this prototype, turning our ideas into reality. Finally, thank you to the audience here today for joining us and taking the time to listen to our capstone defense presentation.

Speaker 1: Thank you.