This transcript is auto-generated and may contain inaccuracies.

Truc Taylor:
I think we’re going to go ahead and get started so that we have enough time for our speakers today. So welcome everybody. My name is Truc Taylor. I am a director with Guidehouse. I currently lead our health AI and data practice there. I will be your moderator for today, and I will shortly introduce our wonderful speakers. So today’s session will be around, and I apologize, I’m yay high. So I’m also looking to leverage AI to advance public health data infrastructure.

So in this session, we’re going to focus on three primary goals. So first, we’re going to build a foundational understanding of AI together so that we make sure we’re all speaking the same language. Then we’re going to explore real-world examples of generative AI and public health through case studies that highlight its potential and impact. We’re also going to examine the challenges and best practices involved in implementing Gen AI solutions effectively and responsibly in public health settings. So first, we’re going to go through that understanding of AI and public health. Then we’re going to talk through our speakers and introduce them, allow them to present their use case, and then we’ll go to Q&A. If you have any questions, please hold until the end.

So let’s start off with an understanding of AI. We’re going to start by grounding ourselves in some core definitions. So first, artificial intelligence, or AI, with a lot of vowels, refers to computer systems that perform tasks requiring human-like intelligence, such as learning, reasoning, or decision-making. Machine learning is a subset of AI that enables systems to learn from data and improve without explicit programming. It’s commonly used to find patterns or make predictions. Narrow AI focuses on specific tasks within limited domains, such as image recognition or search engines, and it’s been used for decades. Generally, AI describes highly autonomous systems that resemble human-level intelligence and can handle a wide range of tasks, so large language models are an example of general AI, and they are a major step in this direction.

Generative AI, or what we hear as Gen AI, creates new content, such as text, images, or music, by learning from existing data, and it powers tools that can generate essays, artwork, and more. So continuing with our definitions, let’s look at a few more technical terms, and again, just bear with me. We’re just level-setting on what we all understand by these terms. So large language models are trained on massive text data sets and can generate human-like language. Text chunking or vectorization breaks text into smaller parts and converts them into numerical formats that AI can process. RAG (retrieval-augmented generation) is an approach that combines retrieving relevant data with generating responses to improve accuracy in context. AI models also use two types of variables, so think deterministic variables that have fixed values, as well as probabilistic variables that vary based on likelihood, helping models handle uncertainty.

Okay, so now we’re going to look at this diagram, which helps us visualize how different areas of AI can relate to one another. So artificial intelligence is the broadest category. It includes any technique that helps computers mimic human behavior, inside which you have machine learning, which uses data and algorithms to help systems learn and improve without being explicitly programmed. As we mentioned earlier, deep learning is a more advanced form of machine learning that uses neural networks. So think layered systems model loosely based on the human brain to process that complex data, like images or language. And finally, generative AI is a subset of deep learning that creates new content, text, images, and code based on patterns that it’s learned. So each layer represents a more specialized and powerful capability within the broader field of AI.

So now that we’ve seen how AI technologies relate, let’s look at how they’ve progressed over time. So this technology highlights this timeline, highlights key milestones from the Turing test in 1950, which introduced the idea of machine intelligence to IBM Watson in 2011, showing AI’s ability to understand language. In 2020, GPT, standing for generative pre-trained transformer, marked a big leap in generating generating human like text. In 2023, generative AI had gone mainstream, and it’s now being used across industries and in daily life. Finally, in 2025, several weeks ago, the US launched a national AI action plan, signaling a major step in AI strategy and governance. So these moments really show how AI has evolved from concept into everyday impact and reality.

So now let’s introduce our distinguished speakers who will share with us their public health journey in AI and its implementation across different public health use cases. So first off, we have Dejan Jovanov. Dejan is the Chief Data and Informatics Officer at the Illinois Department of Public Health, leading the state’s data Modernization Initiative and advancing public health innovation. With over 18 years of experience, he specializes in health informatics, systems architecture, and clinical data standards. Dejan holds a BS in Automatics and Systems Engineering, multiple certifications, and continues to drive data-driven strategies to improve health outcomes. Please help me welcome Dejan.

Dejan Jovanov:
Hello. Sorry for the loud sounds. So today we’re going to talk about the measles outbreak simulator dashboard. So that’s one of our AR projects that we are doing at the Department of Public Health in Illinois. It’s in what we are trying to do this is to changing the response from reactive to proactive. So, as I was introduced, my name is Dejan Jovanov, or Dejan, if you want to give some French connotation, the hand with Spanish. But it’s there one of from with the Macedonian flavor where I’m coming from, and I have been with the department more than 12 years, serving in different capacities, and now I’m DMI Director and Chief Data and Informatics Officer.

So, we have built this simulator dashboard in April 2025 as our way to respond to the additional increase in measles cases around the state. So we took some school vaccination rates data. We got the enrollment data, put some additional model parameters, and we are trying to see how in the likeness of the outbreaks in the schools are. So this whole solution was inspired by the University of Texas at Austin epi engage measles outbreak simulators, and it’s a part of our overall initiatives to transform our data echo systems to be more proactive, more predictive than reactive.

Traditionally, in public health. We are reactive. We react after the things happen by collecting data, analyzing, and then pushing something out. This is like trying to reverse that process. It’s meant to support school administrators and staff in outbreak prevention and control efforts and to inform the public. So let’s talk about the data competence that goes into this simulator dashboard. So we have contact with the Illinois State Board of Education, and we have gained immunization data, along with the enrollment counts. We have also put together some modeling parameters, like the basic reproduction number and what that means. It tells the average number of people an infected person will spread the disease to in a fully susceptible population, and as measles has one of the highest error values of any disease, it’s very contagious.

So we have another parameter that we put into the model, which is the average latent period, and that is telling us the number of days. The person is infected with measles before they become infectious. Also, we are talking about the average infectious period. So that is giving us how long someone can spread the disease to others after becoming infectious, and at the end, we are defining the minimal outbreak size. What does that mean? That means how many new cases we need to have in a certain area to say that we have an outbreak? All of these parameters and information are put in a model, and the model is called the SEER model, and it’s defining the name come for the population group.

How this model is defined so that it divides the whole population into four different groups. The first group is susceptible people who can get the disease. The second group is exposed, so this is people who have been infected but aren’t yet infectious. So the third group is infected, people who can spread the disease. And the fourth group is removed, so people who are no longer infectious, either because they recovered from the disease, they were vaccinated, or, in some cases, passed away. The model also defines how people can move through all these groups. So we have an exposure rate. Exposure rate is how quickly a susceptible individual becomes exposed. Then we have a latent rate, which is how quickly an exposed individual becomes infectious. And then we have also the removal rate, how quickly infected individuals recover or are removed from the chain of transmission.

In our model, we also put the vaccination rate, infections are initial infections, the contact rate, the basic reproduction number, and some of these. It’s hard to understand the model, even for me, who is working, but I’m not the data scientist here. But it’s still hard to understand. So we default some of the values to follow the CDC measles clinical diagnosis facts. So right now, we have explained the model, and I will be explaining the calculation process.

So this model is very demanding, so demanding in processing power, demanding in references. So in order to make it faster response on the dashboard, and although most cost-effective. We have pre-calculated some of the schools using the default parameters, and we have, right now, this is probably a month ago, so we have a 7000 cash result, and our peak was when we promoted this on the news. So the same night, we have almost 2,500 new cash results coming. So we are not catching all the results. So this is not how many visits we have. It’s like, if you ask for the same parameters and 1000 people came back, they will not cash it again.

So, how is the process? You put the request in the hash board with infectious parameters; they go into our machine learning services realm. So we are hashing all the information that we receive, and there’s an intelligent process that is comparing that with the hash result. And if you are specific with how you set up the parameters, and we don’t have them, then we’re going to call our machine learning algorithm to calculate the new values, and at the same time, we’ll populate the cache table. And if you come back or someone else comes back with the same request. We don’t need to calculate again and then return the results to the dashboard, in the case that we already have the already the hash parameters there. So we’ll just pull the result and pass it to the dashboard.

So, this is what I didn’t want to experiment with life, so I put some screenshots. I am old enough to still remember Microsoft XP when Bill Gates was presenting that end crash, so I didn’t want to risk that. So. It’s telling you two stories that I’m old enough, and I still remember some of that. So this is the screenshot of the dashboard. It’s live. You can all visit it and try to play with the information. So we have all the Illinois schools, pre K till 12, already there. So you can go and choose from the pick-down list, the school, the district, the county, or you can enter your own parameters if your school, or you want to just try something different, or you want to experiment.

So after you put all this information into the dashboard, you have the opportunity to set the epidemiologic parameters. So we are having a message there to advocate not playing, because you can play with them, but you need to know what you’re doing. So we’re saying that if your local health departments are happy, you understand how the disease is spread, you understand how this model works. Yes, you’re welcome to play, but if you want, if you’re just like a made-up happy, you can play and receive some crazy results that you can call us, and we need to explain.

So after this, everything is entered into that it’s calculating the chances of the outbreak. So what does that mean? So the model is creating like 200 scholastic simulations using the inter parameters from the chances of exceeding your minimum parameters for outbreak. The value in this example is 58, so that means that from 200 simulations that we did with moving some of the parameters you selected, there are 58 results of these 200 that show that the outbreak will be higher than your threshold. So the threshold is a movable target. The default is 10, but that’s exactly how this works for the chances of the outbreak, the number of outbreak cases we expected.

So we are taking those simulations that are higher than a minimum, and we are taking the range of those to be 95% of the median. And the graphic representation is taking like 20% of the 20 random simulations, and it highlights the one that is closer to the parameters that we calculated before. It’s more visual, so people want to see something, rather than having any additional value. Because if we show we try to show a different number of simulations, and the graph looks crazy, we defaulted to 20. So what is next? This is the model that can be used for many different diseases, like all the diseases that have a clear incubation period, and have a vaccination in. It can be the prediction can be calculated with this model, but they have all these different specifics.

So we are trying to adapt the model to add additional information that goes like age groups, vaccination status, seasonality, and trying to divide that number, what are the chances that the outbreak is if you’re vaccinated, unvaccinated, and all these different information or epi parameters that go inside. So we are trying to build this model and build an additional dashboard that will go out because this draws a lot of attention from not just the public, but also nationwide. People were trying to figure out how to call and talk to us about how to adapt this model to work with our data. That’s all I have today, and I believe the question is later, right? Okay, thank you.

Truc Taylor:
Thank you. Day on. So next we’re going to move to Stephanie Meyer. Stephanie is an epidemiology supervisor. Advisor in the emerging infectious diseases epidemiology and response section at the Minnesota Department of Health. She holds an MPH degree in Epidemiology from the University of Minnesota, with over two decades of experience in public health and infectious disease surveillance. She leads a multi-disciplinary team conducting data analysis, visualization, and outbreak response. Her work has spanned food-borne and respiratory pathogens, health informatics, artificial intelligence applications, and public health. Okay, Stephanie, are you still on with us?

Stephanie Meyer:
I’m here. All right. Yours. Great. Thank you. And can you see my slides? All right, the slides are projecting.

Truc Taylor:
Yes, assume yes.

Stephanie Meyer:
All right, good afternoon again. I’m Steph Meyer from the Minnesota Department of Health. Apologies for not being there in person, but glad I could join you today. I supervise an epidemiology and data team in our infectious disease epidemiology Prevention and Control division, and today we’ll be talking about Claire, which is our large language model pilot project for medical chart reviews. So, just a little background. COVID-19 is one of many reportable diseases in Minnesota, and we acquire data from providers across the state and conduct disease surveillance in a largely centralized manner. So Minnesota is also part of the emerging infections program, and as part of that work, medical chart abstractions are often necessary, and I should note here that I’ll use the term abstraction and review interchangeably and often in this little presentation.

So what I mean when I say abstraction is that we’re condensing the elements of a chart into the data that we need to make determinations about population health. So this is how we get statistics that tell us things about how your underlying heart condition might put you more at risk for a certain infection or outcome, for example. So when we say that a certain percentage of people who had severe COVID-19 infections were also obese or had kidney disease or had other problems, we’re just looking at that data from medical charts to figure out the statistical associations and implications for the health of the people in our state, in our country, or even worldwide. So COVID-Net is one of many programs that require these chart reviews, and COVID-Net is part of that emerging infections program and includes laboratory confirmed covid 19 associated hospitalizations among children and adults.

So, just to level set our understanding, I’m not sure how many of you have seen a medical chart, so I thought I’d include some snapshots here. When we’re reviewing a medical chart, we look for answers to specific questions and then try to determine the course of a patient’s illness or hospitalization. So this might include getting admitted to an intensive care unit or being discharged to a rehabilitation center, or treatments that a patient was given, or just underlying conditions that affect what’s happening to them. So we collect all this data, and we enter it into a red cap database, and all the participating sites across the country collect this data uniformly and record it, and that allows us to do analysis and understand those types of statistics that you hear on the news, like 25% of covid 19 cases had an underlying condition like obesity or cardiovascular disease.

And you can see in this example that just like in your hospital stays, if you’ve ever been hospitalized, data is recorded every day. So somebody comes into your room, they check your blood pressure, you know, your pulse, your temperature, all those things get recorded over and over again for the duration of your stay. There might be a paragraph about why you were admitted to the hospital. There might be other codes that they use for billing and other purposes. So all those things are recorded in your medical chart, and most charts will also have long lists of medications, as you can see here. So some of those might be pre-existing prescriptions. Some are new medications or treatments, and they can be listed over multiple days, multiple pages of your chart. Each treatment might already have been administered, or it might be something you’re taking home, and the details and nuances of the dosage and timing can be important in relation to an infection.

So the data from that chart is then entered into a database, along with specific answers to detailed questions. So this is just a snapshot of what we refer to as an abstraction of the complete data set from the chart. So this is, this is our data management tool, red cap, and you can see that there are specific values like height or weight or answers to specific questions about the patient’s hospitalization that we’re answering based on the information in those medical charts. And manual medical chart review involves multiple staff, as you can imagine, and the COVID-19 net program includes just a sample of hospitalized cases for chart review, but you can see that still, 1000s of chart reviews have been completed each year, with over 20,000 of them done in the early part of the pandemic. And the cost of full-time staff to review 1000s of charts is just not tenable, particularly as we look at funding limitations and other barriers.

In our current process, the manual medical chart review is done by trained epidemiology staff or public health graduate students, and some charts are over 100 pages long. They can take two to three hours to review. And we’re conducting these reviews in a side-by-side manner, with a medical chart open on one screen and your abstraction tool or database on the other. And you’re just typing the data from the chart into another system, and the data is then sent from the Minnesota Department of Health to the Centers for Disease Control as part of our cooperative funding agreements. And if there are any data errors identified, those problems are manually reviewed, and charts are sometimes reabstracted.

So in the future, we’re trying to generate a document or a file that contains all the relevant text from a medical record, and we want to use large language models to sort of read that chart and extract some key data elements, and this might include using standard code or prompt engineering and selecting multiple large language models. And a selection of that data would then be validated against manually collected data. And the results of this process would be used to improve our abstraction pipeline. So in our Claire pilot project, we identified 150 COVID-19 medical charts from one health system, from the 2223 respiratory system. All the charts were redacted PDF files, and all of them had been manually reviewed by MDH staff with answers entered into a REDCap database. And we partnered with Minute at MGH, our IT infrastructure here at the State of Minnesota, along with Amazon Web Services, to evaluate our options. The analysis of the results from those text extracts of abstracted charts from the AWS pipeline was compared with our manual abstractions.

So the basic process developed with our Amazon partners included multiple steps, and it begins with a medical chart going into an Amazon S3 bucket or a cloud storage service. So this triggers an AWS step function state machine, which is really just a pipeline that groups and organizes a series of tasks that need to occur either dependently or sequentially within our larger Amazon environment. And from there, the first step is text extraction using textract and the Amazon tool. We convert the chart to a machine-readable format and then store the output in our S3 bucket. Then, next, we move into text chunking and vectorization. So we broke the extracted text into smaller text chunks using Amazon’s large language model Titan, and the chunks of text were converted into vectors, which are just numeric representations of text using embeddings. And this part is really key to semantic searching. So we’re not doing keyword matching, but we’re doing concept-level matching.

So this process is letting us rapidly match relevant chunks of data with field definitions or prompts. So the vector representations are stored in this semantic index, and we’re organizing chunks of text by meaning rather than by keywords. So that allows us to retrieve the most relevant pieces of information based on their conceptual similarity. And that leads us right into prompt-driven large language model searches. So we use different LLMs, giving them a set of field definitions defined through our covid net protocol, along with highly specific prompts to help them search efficiently. And the large language model might go to the vector store and then try to match with these defined concepts.

So these two steps, the text chunking and vectorization, combined with this prompt-driven LLM search, are the retrieval augmented generation, or rag piece, like you hear people talk about a rag approach. That was our RAG, and I’m just representing it here in a more granular view. So then lastly, we get the field match JSON output, which we had saved back as a file into our S3 bucket. So again, that gives us the full process. You know, from left to right, we’re converting the files into a machine-readable format. We’re chunking that text into vectors and using prompts and various large language models to search that vector store for the concepts, and then saving an output that is the answers to our questions.

And throughout the pilot project, we really looked at various data elements and corresponding accuracy. So we divided our data into four categories, and the first of these was structured data. This is data collection that might really be better suited for standard code. Think of things like weight or height that are more likely to be deterministic variables. The next was prompt driven search, and this is data that might require decision-making based on additional information. So you could think of things like classifying a type of antibiotic or an antiviral treatment. Next was probabilistic matching. So this was data that we thought might be better suited to a large language model or other AI resource. So this might include just concepts and nuanced data. You could think of things like a family history of a certain type of disease.

And then lastly, we had summarization variables. These were data that might require some manual review by staff, but a summary from the chart that has citations could really help streamline the work. It should just be noted here that other large language models might help us in that process to further streamline the data analysis from the summaries. And this is a key piece where we want a human in the loop. We want a portion of this work to be the creation of this augmentation tool. We’re not necessarily replacing all the needed people, but the speed with which we can do these reviews matters, as does the quantity of reviews. And we looked at the accuracy by each of these categories. So if you’ll recall, we had these 150 charts that we ran through our pipeline, but we had also manually reviewed those, like staff had looked through all those charts and already done the abstraction.

So we were comparing our output from our large language process to the manual review, and both structured data variables and summarization variables had the lowest accuracy score using just the strictly large language model pipeline. And this is sort of what we expected, structured data that aligns with our thinking that perhaps standard code would be better for a better method to extract this data and then for summarization variables, we also had some lower accuracy scores, and that underscores our desire for just additional review and maybe a shorter summary with a citation that would help us streamline that that process.

In summary, the Claire project was unfortunately paused due to funding turbulence, as many of us have experienced over the past six months. So with adequate funding, we’re hoping to continue this pilot with more cases, more file types, and more providers, and we’re hoping to look at charts in other disease areas and explore the use of large language models with electronic case reports as well. Large language models combined with deterministic code might be a smart hybrid approach to medical chart review and could really result in a more efficient use of staff time. And large language models could thus expand our ability to use medical data directly from providers to better understand population health. Thank you.

Truc Taylor:
Thank you. Stephanie. So last but not least, we have Jim Daniel, who is the public health leader for Amazon Web Services, state and local government team, helping health departments and providers modernize their IT infrastructure. Previously, Jim served for almost a decade with the US Department of Health and Human Services, where he promoted public health innovation, including the development of the immunization gateway and consumer access to immunization information systems. Jim also served as the CIO for the Massachusetts Department of Public Health and holds an adjunct position in drug regulatory affairs.

Please help me welcome Jim Daniel.

Jim Daniel:
Okay, great. So I’m not going to dive deep like Dan and Steph did into a specific topic, but instead, I’m going to talk about some other public health use cases for Gen AI that we’ve been exploring at Amazon Web Services, working with some other customers, including electronic case reports, intelligent document processing, chatbots, and data analytics. So before I dive into all of that, though, I do want to make a point that as I talk about all of these, you’re going to think about the fact that a lot of these tools are looking at PHI, and what are we doing to? Protect PHI when we’re using AI, and when you’re using AWS services to run AI with our bedrock or other tools, you’re actually doing it in a way that it’s running strictly in your own account. You’re almost like creating copies of these large language models in a way that just you’re using them, so your data is not being used to train the large language models.

That’s something that we get asked all the time, which is, how are you protecting our data? How is it not being used to train the models? So it does sort of give some limitations, because the way it’s set up automatically, you’re not training the models. So I’ll talk about, if you do want to train them, some ways we have around that. But the way the security is set up, though, is that it’s all running in your own account, and that data isn’t being used to train models.

So the first use case that I want to talk about is electronic case reports. Steph talked a bit about that as well, and it’s really similar to the use case Steph mentioned. So, how many people have actually tried to read an electronic case report themselves? Has anyone actually looked at one? Okay? So you know, they’re kind of a mess, right? We had a lot of health departments come to us and ask us to help them process their electronic case reports, because the volume is so high, it’s just you get 10 or 15 for every portable condition. Some of them are 20 megabytes. They’re huge files. How do you process these? And our first approach was the Health Department said, Well, why don’t we just put it all in a data lake, and we’ll just query the data fields that we need. We’ll use it comes with nice XML tags.

Why can’t we just use those XML tags and query the data? But it turned out that there’s not a lot of standardization for the names of those XML tags, and within those fields, there’s no standardization of vocabulary, so it’s really hard to set up just a normal process of putting that into a data lake and querying it. So our idea was, why don’t we use generative AI and actually find those public health actionable data elements that are important to public health using Gen AI? So if you think about that, what does that mean? We can look at them. We can look at all the syphilis cases, and think about the data elements that we need to find to prioritize all of the syphilis cases that are coming in, find the ones that are for women of childbearing age, that the pregnancy test is not there, or that there’s a positive pregnancy test for those are the ones that you really need to prioritize and make the phone calls for. You don’t necessarily have to call everyone, but can we use Gen AI in a way to help make the epidemiologist’s life easier and prioritize their work?

None of this, and I should have started with this, but none of this is about replacing jobs. None of it. This is about making it easier to do your job and making it more efficient. Instead of having to look at every individual syphilis case report, you can use Gen AI and help identify the ones that you need to find. So that’s just an example for ECRs, but there are multiple other things you can think about for Hepatitis A. Find the ones that are food handlers; those are the ones you need to call immediately. So if you think about the way an electronic case report looks, it’s just a big XML file. You know, there are human-readable forms as well, but it’s going to take someone like 15 or 20 minutes to read through all of that and find the information that you need, since we can’t just automatically query it.

So, instead of hearing Steph talk about this, we’re using prompt engineering to ask the questions we want to know and get the data in a standardized format. Just ask the questions, basic demographics, is this person pregnant? Is this person a food handler? We can just ask all of those questions, depending on which disease it is, and we can even say for the question, Is this person a food handler? Only give me yes, no, or unknown, so you get the very specific data elements and vocabulary you want, which you can then push directly into your disease surveillance systems for case management. You can even ask for it in the format that you would like to have it exported in, so that it’s ready to export. You can ask for it in JSON format at CSDE. Someone asked us if we could generate a mif file for the Maven customers, and we gave them an example of a Maven mif file, and asked it to produce a MIF file, and it was able to do that.

So, taking advantage of what Gen AI can do is a way we think we can really automate and make the process of electronic case reports much easier. I will say this is very much in the early stages; we’re working with customers to validate the process. Think about the ways that we really need to use all of our tool sets to make it the best. What Steph showed for how they’re looking at electronic health records, that’s exactly the kind of process that we need to go through for electronic case reports. Thinking about how we chunk these big. Pieces of data, thinking about rags or our knowledge bases, how we could use those, because even for something like syphilis, you might want to know, was this person appropriately treated for syphilis or not? Those are the ones you want to call right, the ones who would not have been appropriately treated. But you don’t necessarily want to read through all of the medications.

ECRs include a lot of medications that public health doesn’t want to look at, like behavioral health medications and other things that we don’t want to see. So instead, we could use a knowledge base or a rag to say, here’s what we mean by appropriate treatment for syphilis, and then just get that answer, yes or no, was this person appropriately treated for syphilis? You can again put directly into your disease surveillance systems and use that for case management. So another use case that we found that’s really important to public health is intelligent document processing. I think no matter what we do, public health is always going to get a ton of paper forms. I think in the infectious disease world and immunization, which I know probably most of you guys are from, we’ve done a really good job of getting away from paper, but in other parts of public health, there are definitely still a lot of paper forms coming in, vital records, WIC, there’s still a lot of paper coming in.

With intelligent document processing, it’s a little different than what you might think about with what we used to call just optical character recognition. We get, like, just a big text file of what’s on the form. With intelligent document processing, it can actually look at the form and figure out the fields, so you don’t have to write a lot of code to process the data. After you do the optical character recognition with OCR, you’re just going to get a bunch of Yes, no’s, and text, and you’re not going to really know what it means. With something like this, it can look at a form, look at a whole bunch of fields on a paper, where you just circle something, and it can say, OK, this is a field called symptoms. Here are the options for symptoms. It could be cough, fever, and diarrhea that are circled, and it knows that those are all the things that go with that field fever, and it just creates it all automatically for you. It’s amazing what intelligent document processing can do.

It takes about 10 or 15 minutes to set up a form as it comes in, but then you can just have a fact server taking all of your forms. It can recognize different forms, spit them out into different places, and again, export the data in the format that you need, which is ready to incorporate into your products. And what’s great about these is that it doesn’t really matter what your product is, right? You can be running legacy products that aren’t even cloud-based, but you can still be taking of these, taking advantage of these cloud-based technologies. Another one I was kind of surprised by, as we’ve gone to about six other public health conferences so far this year, was that our generative AI chatbots were very popular.

The first conference that we went to this year was WIC, and during the opening reception, we just asked people what their biggest pain points were. And everyone said, “All these people are calling and emailing us with simple questions that they could get answers to if they just looked at our website, so we’re like, ‘Oh, I bet we could build a chatbot for that.’” So we asked a couple of WIC programs for their documentation. They just gave us PDFs or websites, and the way our chatbots work, it’s integrated into a tool called Connect, which is a virtual call center where you can have a virtual agent be the first response that happens with a call center. So when a customer comes in with a question, whether it’s chat, they can actually call and ask the question via phone. They can do it via SMS. You can interface it in multiple ways. You can actually then have a virtual agent look at any of your documentation and answer those questions.

And we set it up so that, using knowledge bases or rags, it only looks at your documentation. It doesn’t look at anything else. If it can’t find the answer, it just says, I can’t find the answer. And then, because it’s part of our overall call center, it can then connect you with a person via chat, phone, or whatever the preferred methodology is, so that they can answer that question in person. But it can easily take away, you know, 50 to 60% of those questions that the public has when they’re trying to call with WIC, it’s questions about eligibility. The following week, we were at the immunization information system conference, and we heard that health departments are challenged with providers calling in with questions about how to use the immunization information system.

So Minnesota was there with their documentation. We loaded it up, and by the time the reception was going on that night, we actually had a chatbot, you know, showing how it can answer questions from, you know, something like, what do I do if twins come back as the same person? How do I duplicate them? Or even, what do I do if I accidentally give an immunization that has expired? All of that. Was in their documentation so the virtual agent could answer that for the providers. So it’s not necessarily just all external facing chat bots that public health could use. It’s also one for people who are using your systems that might not know how to use those systems. At CSDE, we are approached by another health department that was thinking about how their local health departments could do something like that for their disease surveillance system. Because it’s so complicated, we didn’t have all the right documentation to set it up, but that’s something we’re working on as a proof of concept as well.

So all of these can be incorporated directly onto public-facing websites if it’s something that you want for the public, or they can be behind a firewall. If it’s for a system that you only want certain people to do, but it’s just, you just drop it on that page. It’s very, very easy to do. The last one that I want to talk about, that I think is really interesting, is our generative business intelligence. We have a tool called Quick Site that’s our tool for data analytics. And the idea here is we don’t have as many data analysts as we need in public health. We’re always short of all of those types of resources. So when you get a call from the governor that wants to answer you know question or your your boss’s boss’s boss has, like, a question that they need to answer, you know, in five minutes, and you only have so many data analysts to go back to that can try to get the graphs set everything up, write the data story that goes along with all of those, those data points with AI assisted storytelling.

Instead, we’re using our data analyst to get the data set up within the tool, a quick site, so that it’s then available for people to just interface with it using natural language, so you can ask questions like, can you show me the relationship between diabetes and cardiovascular disease? And it becomes a good data analysis kind of set everything up with what we call synonyms for the data fields or what people might be asking questions about, the Gen AI tools can then go in and start suggesting and creating the graphs that you need to answer those questions. So a business user could go in, ask those questions using natural language processing, and build those questions, graphs, and stories on their own.

So all of those are things that we’ll have running as demos at our booth up on the fourth floor. If you want to come see them. My Solutions Architect, Venkata, who’s amazing, doesn’t get here till 10 o’clock tonight, so we won’t be at the reception tonight doing that. It’ll just be me, so it’ll be as fun, but he’ll be here the rest of the week. We do have some blogs about some of this work as well. We’ll have cards with these blogs on them tomorrow, and always feel free to contact me. That’s my contact information there. Thank you.

Truc Taylor:
Thank you. So now we are going to open it up for Q&A. If anyone in the audience has any questions, if not, I have some prepared, so don’t worry, but I want to give you guys a first shot at it. Okay, so, oh yeah, someone over here. I was so close.

Audience Member #1:
Thanks to the first speaker. Thanks for the so the seer model has been used in epidemiology for years, right? So, I guess two questions: one is how you’re using AI in the seer model a little differently from the standard used in a CIR, and the second is how you act on the information you’re getting here.

Dejan Jovanov:
So the seer model is more in the realm of machine learning, right? It’s predicting the predictions of what is the likeness or likelihood of the outbreak in a certain area or certain schools in this case, so how the people are using that, they can see how the vaccination like, when you change these model parameters, depending of how many of the kids in your school are vaccinated, then You can see how important is that part in all this calculation? So it is bringing awareness to the parents. I’m a parent also in some of those school districts. So I look at my school first, right see what the vaccination rate is? What is the likelihood of an outbreak there? So it’s informing the public. It’s helping those administrators to send the message that vaccination and all these efforts that we as a public health are doing are valuable and preventing outbreaks to stop them, even before. That happen.

Truc Taylor:
Other questions…

Audience Member #2:
Hi, thank you for your presentations. I’ve come from agencies that have centralized IT departments that are hesitant to adopt AI, and so they have policies in place. So I was wondering how you partnered with your IT departments? Any challenges you faced, and how you overcame those for adopting AI and machine learning? All that hesitancy.

Truc Taylor:
So if Stephanie Meyer is still on, Stephanie, are you there?

Stephanie Meyer:
Yeah, absolutely. So in Minnesota, we established the transparent artificial intelligence governance Alliance, or we called it TAIGA, and our IT services folks just recognized that there was a need for AI across multiple agencies, and to have some sort of governance structure in place really helped set us up for what we wanted to accomplish. So we had somebody on the IT side and embedded in our agency. So ours, our IT infrastructure is a separate agency, but with people who partner at all the different agencies and who are embedded with us. So one of our key people, Andrew Will Holmberg, who is he works for minutes, but he is at the Minnesota Department of Health.

He helped found this TAIGA group, and they really came up with, kind of, the governing principles, the transparency, the way that we would structure all of this for any AI proposal, and it helps it sort of go through a set of cases, so that there’s awareness across government of what we’re doing. The governor’s office is involved and knows what projects are happening, and we can kind of keep tabs on everything that’s being done across the state, to use AI. We sort of did this as a forward-thinking thing because we need to start using this technology. And from my perspective, in public health, we really need to do something about our infrastructure. We’ve been lagging for so long, and we need to get on board with some newer technology so that we can, you know, streamline our work and do a better job. So, TAIGA is our way of doing that. And if you just look up TAIGA, Minnesota, or the TAIGA government, you’ll find it. There’s a whole website and everything.

Dejan Jovanov:
Yeah, I can add Thank you, Stephanie. Some of the things that Stephanie already said are that, you know, working with the government, we always lack in technology, but the AI is here. We cannot ignore it. So we have our AI policy developed by our central IT agency. And during COVID, we learned very important lessons about the need to work together. So they cannot work without us as a program, as an informaticist, and we cannot work without them as a central agency that governs our systems and everything. We have developed a policy and are adhering to it. We are putting the human-in-the-loop in most of our solutions, experimenting with several of the air solutions Jim already talked about, and trying to do so as safely as we can. So we are not doing anything outside. There’s no right now in any of our projects are having personal information in them, so we are trying to experience and do these first steps with AI, adhering to the policy that we have, but also protecting the data and privacy.

Jim Daniel:
And I would say, from the AWS perspective, we often help the states respond to requests from central IT. And I think everyone is really interested in the same types of questions that they have around privacy and security. So we can certainly help you as you’re trying to answer those questions. But I would love to see a community of practice come together for states to put together the guidelines they’re using, so that states that don’t have policies can take that information back to their states and start working it out. I would say states with strict policies are easier to work with than those with no policies, because they’re the ones that are really tough. So it’d be great to start sharing some of that so that people can take it back.

Truc Taylor:
Thank you. Any other questions?

Audience Member #3:
Hi, Jim. So thanks for that. Your AI chatbots and some of the examples you gave, I guess I think a lot about validation and about truth. So are you using, like, AWS foundation models specific to AWS, and then also, are you doing any validation on, you know, like the IIS examples you gave, putting humans, you know, sort of providers, epidemiologists, and comparing it to what you get from the models.

Jim Daniel:
Yeah, so, great question. So, the tool that you use to access any large language model with us is called Bedrock, and with Bedrock, you can actually access now any of the large language models we didn’t have OpenAI, but as of two weeks ago, now, even OpenAI is available. So it’s your choice. There are Amazon models you can choose from, like Titan as well, but you can really choose any of them. Your cost to run AI really is largely dependent on the model that you choose. Some of them are more expensive than the others, but you can use any of them. You can also run multiple models at once so that you can set it up to compare answers. But we actually have tools to help you walk through that. All of these need validation processes like you, as you mentioned; I can’t guarantee you know what’s going to happen when you run your specific model. I don’t know what your prompts are going to be. Are you going to set it up? But there are tools available within our bedrock console to help you with that validation, and we can certainly help you as well on you know.

When we have set up the chatbots to answer questions about an immunization information system, we’re very careful about the prompts that we use to set that up to only look at the documentation that we’ve given it and not to look at any other documentation. If you don’t provide those sort of guardrails, AI will go off and find its own answers. When we were working with our customer to look at disease surveillance, and just reviewing some of the fact sheets around disease surveillance. We forgot to do that, and we’re asking questions about rabies, and got answers. Because I think the question was, what do you do if you wake up to a bat in your room and you’re concerned about rabies, and we had forgotten to give it those guard rails, to just stick to the fact sheets? And got responses like, look for a rash like what I don’t think that’s right. Am I remembering rabies incorrectly? But I think making sure that your prompts are right and that you’re limiting them are the ways to at least start, but it always needs a validation process. Okay?

Truc Taylor:
I think we have room for one more question. Anyone? Okay, so I will ask our speakers one more question: What has been the return on investment on your projects, and did it deliver the value that you were hoping for?

Dejan Jovanov:
So right now, it’s really early to talk about the return on investment, because right now we are investing a lot of time. We are exploring new technology. But what was our approach? And when I was presenting that, we even when we build a dashboard, we built that in mind, so we try to lower the cost, but bring them more value. So the value there is that it will prevent outbreaks. Like, there’s no price tag on human life, that’s what I think. But we are trying to build a solution that can be used and reused, so that will that was our model. Like we don’t build something that will be used once and then forgotten.

As I mentioned, the model we built for the dashboard can be used for many other diseases. We just need to change a few parameters and put the other project that we are doing, and we have seen it’s in the early stage, but we have calculated the return is digitalizing our vital records. So by using Document AI, we can read handwritten records, and its accuracy is at least 70% higher than that of a human. I’m not talking about the speed, which is like in 90%, and the saving of the people’s time and effort is almost like 98% saving in a month.

Truc Taylor:
Great. Thank you. Stephanie, are you still on the line? Could you answer from your perspective, from Minnesota?

Speaker 1:
Yeah, sure. I think I have a similar perspective in that it’s kind of early to say the immediate results of this, but that we also tried to build something that we could apply to so many other things, and we hope to continue to get five. Funding to look into it, because just starting a project like this, and I don’t know if Dejan, this was your experience as well, but we just didn’t know what types of questions we were going to have until we were into it and really understood even how to divide up the data, how to look at it, and how to build more of a hybrid approach than just try to insist that AI will handle everything, because there are other parts of our infrastructure that have been lagging, even just converting a medical chart into a machine readable format. We had not done that, and that, that alone helps us devise some plans to do some standard coding and other things that are going to help us. So, I would say, still to be determined, but there’s a lot of promise in what we’ve learned so far.

Truc Taylor:
Thanks, Stephanie, then Jim. Any last perspectives on that?

Jim Daniel:
Yeah, I mean, these guys are my customers. They’ve got the ones with the data on ROI, so I think I will trust their answers to that, but we love to help people figure that out for future projects.

Truc Taylor:
All right, so there are no lingering questions, obviously. Thank you, Stephanie, for joining us online. Certainly, come and bug and poke our speakers if you have more questions, all right. Appreciate you all joining us. Thank you.