Hi, everybody. My name is Stephanie Walker. I'm the head of product for the enterprise segments at CEB, and I will be your moderator for this webinar today on the responsible use of AI in legal research. I'm looking forward to a free flowing discussion on how tech the technology landscape has changed very rapidly in the legal profession and what CEB's own approach has been to incorporating AI and LLM technology enabled tools. First, before we get started, a little bit of housekeeping. I believe the q and a function is available for this webinar, so please feel free to drop any questions that you have into the q and a. We will try to answer as many of those we as we can at the end of the webinar, time permitting. Another reminder, just for clarity's sake, we aren't offering CLE credit for this webinar, but if you're in a state like California where self study credit is allowed, I believe you're allowed to track your own time for this webinar if you desire. And also, we will be offering some materials that we will email out to all attendees after the webinar has been concluded, just for everyone's convenience. So first, it is my pleasure to introduce our two panelists for today. Ben Pedrick is the Chief Technology Officer at Judicata and has spent over a decade building legal research technology, including engineering leadership roles at Fastcase, Vlext, and Clio. His work focuses on the data foundations that make AI assisted legal research accurate enough for attorneys to rely on. Thanks for joining us, Ben. I'm very excited to be here. And our second panelist is CEB's CEO, Kelly Lake, who has led CEB's digital transformation from print to digital to platform in response to an increasingly demanding marketplace. And prior to joining CEB, she was vice president of platforms for Thomson Reuters and was responsible for implementing a global content and technology transformation program, working globally with multiple business units in the scientific, financial, and legal information spaces. Welcome, Kelly, as well. Thank you. Great to be here. So before we dive into kind of the broader conversation, you both have very impressive resumes. Can you tell me a little bit about how you got here and what first interested you in the intersection between law and technology? Kelly, do you want to kick us off? Yeah, I thank you. Sure. So I guess I've always operated at this intersection, as you say, between legal technology and product is how I would that that's been my my core. And the thing that probably most prepared me for coming to California and leading CEB was my time in Asia, where I worked on acquiring legal information businesses primarily in India and China, Malaysia, but then kind of like figuring out how to build legal research and information platforms in those jurisdictions. And the big learning that I got out of those is that he technology transfers across domains and jurisdictions, but the law inherently particularly in large jurisdictions and complex jurisdictions, as I would argue that California is in its own right its own ecosystem and has its nuances and complexities is that the law is is very local and practices cut the norms around how law is practiced what attorneys are looking for is very local and I think that's that was a great lesson to come to CEB and to lead the organization. And I think it's even more pertinent today where really I think our advantage in for CEB is to really be hyper focused, really focused on building depth in California in terms of our coverage and what what we do. Yeah. That's great. And Ben, you're an engineer by training. How did you get kidnapped into this industry? Yeah. So yeah. So I was a talk I still am a software engineer. I was living in San Francisco after I graduated college and worked at a couple of startups and wanted to be working in a field where technology was meeting real people with real problems and helping them. And kind of stumbled into judicata back in the early judicata days over ten years ago now. And from there I got sucked in as you said, I've always had a few places since. But yeah. But I but I think the there's lots of ways to work in software that to me is either, like, frivolous or just kinda not that interesting. But law, I think, is super interesting. It like, it's very much with I can do a better job with the software and build better products. That means real people are getting better legal support in their issues, which is very meaningful for me. So Yeah. That's how I ended up here. Did anything when you first started familiarizing yourself with the legal industry, did did anything surprise you about just how detail oriented the language of lawyers can be? I don't know. I I feel like part of the part of what's interesting for law from, like, a engineering perspective is kind of the structure and the rigor and the rules and figuring, like, you know, it's a little bit of the the game of how all the pieces fit together and understanding it all. I think unlike games, the law gets very messy because real life is messy. And so that actually becomes interesting of how you think, okay, well law is gonna be all this like great structure and it's not really. Like there's a bit of structure, but then also you have to meet reality and law is extremely flexible in that way. So, yeah. So that's sort of how I have evolved to understand it. Was like, okay, it's actually a little bit of a mix. It's not just the structure, it's also the mess of reality and bringing those pieces together. And I mean, actually that's why we're having this conversation today, right? Because it not only is it messy, and not as structured and clear as you think it's going to be, there's a super high threshold for trust and accuracy that you have to meet, which is not the case in a lot of other fields. Yeah. Trust and accuracy. And I think what Ben was just saying also, even even with that, like, rigorous requirement, there's not always a one right answer because there is that flexibility and that nuance, which adds that complexity as well. Before we go any further, I'm sure like many people who work in the law, there's a wide range of kind of comfort levels with the topics of AIs or LLMs. Ben, could you maybe talk us through just what do we mean in plain English when we're talking about artificial intelligence and large language models? Yeah. Absolutely. So artificial intelligence is a somewhat old topic in computer science and in software. I think really the first big effort dates back to like the seventies. But there have been various waves of artificial intelligence that have been intelligent in various ways. And the current wave is powered by these things called large language models, which I imagine everyone here is familiar with in some form or another because they're fairly pervasive. But maybe we can talk a little bit about what they actually are and what they actually do, which I think will be interesting. Or I think it'll be useful for when we eventually talk about how the product works. The understanding at a high level, like how what these things actually are. Okay. So what is a large language model? So a model I'll take you to the large language model and kind of in turn. So a model in this sense, we're talking about like a numbers model. These are statistical models. It's sort of like a financial model where there's some kind of input and then the model makes a prediction and it creates some kind of output. Now this is a language model, which means it's predicting language. So text is the input and then it predicts text that's gonna come out of the output. For example, if you were to put in the text, what color is, you'd hope the model would predict like what color is the sky or the sun or an apple or something like that. Like those would be reasonable things that come out. Slightly less good would be like what color is apples? Because is apples isn't grammatically correct, but you're still kind of there. And then, like, really bad would be, well, like, either complete gibberish or, like, what color is you know, George Washington was the first president of the United States. Like, George Washington is the first president is a coherent statement, but it does not at all follow from the first part. So the idea of like, okay, if we can predict text, what should come after really accurately, you kinda get something sensible. That's sort of the difference in the quality of the models. And that's where the latest generation of models, these large English models have suddenly gotten actually very good at this. And especially recently, really like in the past probably nine months or so, there's been a bit of a turning point where they've gotten like good enough to actually really take on like substantial amounts of work. Where this whole text prediction thing, okay, what color is whatever, where it gets interesting is when there's a right answer. So if you were to put into the language model what color is the Golden Gate Bridge, you'd really want it to say red, or the Golden Gate Bridge is red or something like that. Technically it's international orange. Oh, I'm sorry. The Golden Gate Bridge is internet. Well, okay. So a better model would tell you the Stephanie Walker model will tell you that it's international orange. My model would not be as good. But but where does that knowledge come from? Think is the kind of like a raise a great point. Like how do you know? And inherently there's some amount of knowledge or something baked into the model based off all of this text. But then also you can kind of feed it information in the context. Basically, you instead of saying what color is the Golden Bridge, what if you wrote kinda like what if you wrote the Wikipedia article of the Golden Gate Bridge and then at the end said, now I have a question for you. What color is the Golden Gate Bridge? And it had all that information there. Hopefully, it would reference this is sort of like in grade school, you did like the reading comprehension test where it's like read a paragraph and ask them questions about it. If you're doing that, hopefully now it's like looking at the information that was applied instead of whatever's in its memory. Now you have the opportunity to provide information that wasn't there. Like maybe the model doesn't know that it's international orange, but if you gave it an article that said it and then asked the question, hopefully it would know that. And then you get kind of, like, better answers. And we can go back to that. I think that gets back to, like, the content and how kind of the Yeah. A lot more rag works, but that's the language model piece. And the reason these are large is the actual so these are mathematical models, statistical models, like I said. The mathematical operations are mostly pretty simple actually. It's like addition, multiplication, maybe like some absolute value occasionally. But they are large. They have lots and lots and lots of these operations. And the new models or the like leading models today actually have like trillions of operations of kind of parameters that control all the operations. So every time you pass text in, it's running trillions of operations to figure out the text that can come out. And that's that's kinda like where the power has come from actually. It's just kind of taking kind of these simple pieces and scaling them up bigger and bigger and bigger has been a huge part of what's gotten us to the accuracy of the LLNs, the leading LLNs today. So that's why it's large language models is that taking this kind of simple machinery, but making it really, really big, like trillion is an unimaginably huge number, right? Like the difference between a trillion and a billion is approximately one trillion, right? Like a billion is already a really big number. So it's unbelievably huge amount of parameters that suddenly you get something that looks like intelligence coming out of it, which is sort of remarkable in its own way. Yeah. Yeah, maybe pause there on the kind of large language model background. Well, think it's probably a pretty Oh, what were you gonna say Kelly? I was just gonna say, I think something that looks like intelligent, are like, I mean, it's because of the scale, right? It does feel really smart, intelligent. Yeah. Something that's Yeah. Yeah, I think but I think the Ben, what you just said about kind of, like, the content and where where are these large language models pulling pulling answers from is probably a good segue into talking about what CEB is doing and kind of our philosophy behind it. So we as you are both intimately aware, we recently launched CEB Insight, which is our first AI enabled search enhancement to the online pro platform. Kelly, as as CEO of CEB, what was your goal in launching Insight? What what were you hoping that that would do for us and for customers? Well, let me take one step back to kind of build on something that Ben because I think Ben made a great point there. And now I'm going to look up what golden gate orange is what was the models are incredibly, they're transformative. They're, they're incredibly, the scale is very difficult to even comprehend. But and they seem pretty fluent. And I think like, start like making that distinction mentally between like, let fluency is not authority. And for us, we knew that we have this responsibility in the market. And this is kind of like how we've approached all of our publishing and our transformation to digital information, which is, we got to figure out a way to transfer this using a technology transfer the same level of reliable reliability without diluting the authority and the quality of what we're doing. And so fundamentally, what insight is doing, it's an accelerator or catalyst on what is an incredibly fine tuned search already that we have of CB materials, it's really there to help aid the discovery of this great content set of content assets that we have sitting as part of the CB corpus that is constantly being curated updated, you know validated and verified. I so it's really, that's what it is and I think that's the first step. Early indicators suggest that we've been pretty successful in that and that I think will give us a little confidence to say okay how else can translate, use the technology now to deliver greater analysis and insight on the corpus or even extend the corpus? Yeah. So I guess building on that a little bit, Ben, you talked about, you know, in that Golden Gate Bridge example, it's like, is there an article that says that the name of the paint color is international orange? From a technical perspective, what role does the underlying content corpus or in CEB's case, the CEB content corpus, what role does that play in your development of Insight and other tools? Yeah. Absolutely. So the when you have something like Insight where you're asking a question expecting an answer, the question is where does the knowledge come from that's feeding that? And either it's baked into the model somehow where the model somewhere in its parameters knows. For something like, well, the golden gate bridge is no longer a good example now that I know that I'm wrong, but who's the first president of the United States? Hopefully that's like baked into the model. Like it's seen enough examples, somehow the parameters of it kinda captured that. But not all knowledge is captured in the model and lots of it is captured imperfectly, which is say wrongly for lawyers. So the other place is to feed it in in the context, kinda like what we were talking about before, which is basically you just feed a whole bunch of text plus whatever you want the model to do, and it reads the text and then it kind of uses that to form an answer. So when someone comes and asks a question, you have to go find the knowledge. And the way that we've built insight is we do not wanna rely on what's baked into the LLM, because as I'm sure everybody who's seen LMs get things wrong sometimes, and it's very hard to know when that's gonna be. And so much better would be that all the answers that it provides is really fundamentally based on the content that we have that we've vetted, that CV editors have written and contributors and everything. The kind of the core pieces that have powered, that have made research product good for years now are still very applicable. Like we need the search to be good. We need to be able to find, when a user asks a question, we need to be able to find the documents that are relevant to the question. If you don't find the right documents, then you're gonna give someone a wrong answer based on some other document, or you're gonna miss an answer based on it not being there. So the fact that I think you can go to CB and type in questions and generally get what you're looking for or reliably get what you're looking for is why Insight works well. And it works well because both the technical piece and the content piece that support each other to getting you to the right answer consistently. And, Kelly, I'd be curious to hear a little bit about kind of from your perspective, has has the development of technology changed how you look at either CEB's editorial process or our content roadmap? I think that the the difference between how we think about content and technology as a delivery mechanism, that distinction is completely blurred now. And it's really about, hey, what, you know, focusing on what is CB's core strength. And I think our core strength and advantage and value to attorneys in the market is, hey, our California focus, our editorial judgment that we have internally in terms of the content attorneys that sit in house and and build the system architecture around how the model what what the model should be looking for what questions it should ask what was the right answer? What is this is the wrong answer? Because I guess what Ben was actually kind of saying but he didn't say quite so directly is that the model is not so good at telling you whether something is true or not. That that's the piece that where I think that we really add and being able to do that. And then converting that into, you know, so that, in a way is kind of a layer that we're trying to figure out like how do we build insight into this layer over using a model that really tells us what what is authoritative, what you can rely on, what are the right answers. And that's so content now becomes not just the case law, it becomes the metadata, the structure, the analysis as well, all of that, that. And I think that is where even eighteen months ago, would have been outside of our ability from a cost and resource point of view to do that. And these tasks have become much more, much, much more doable from that point of view, an investment point of view. But it's really fundamentally the secret sauce is the people, people and the know how that we have in at CEB. Yeah, no, I think that's true. And especially given the especially given how complex and how nuanced and how deep California legal topics are, especially in certain areas, having that, like, subject matter expertise and human review baked into the editorial process is irreplaceable, really. I don't have to tell you that. I mean, Yvonne, every time we enter into a new area of law, go deeper, we have to go on. We have pretty experienced attorneys working on our content. We go out and we look for people who actually practicing the law to validate our assumptions to input. So there's, there's a very rigorous, human element behind everything that we're doing that I think is very difficult to replicate. Yeah, I think that's true. And it's so I don't want to dive completely into the questions here, but I am seeing kind of a little bit of a thread of just general concern from people on how do how do I verify that the information that I'm getting is true? How do I, like you know, how how can I have confidence in the tools that I'm using and the answers that I'm getting? So I think what I'm hearing from you guys is is, number one, if the tool is built on a really trustworthy database, that's a huge step in the right direction, right, because the underlying content is accurate and nuanced and talks about it at the level of detail you'd want. Is that a fair kind of assessment? Would say traceability. I think, you know, you you and I went through maybe four or five iterations of how to even tell users how to display the sources. Yeah. How to tell users what was coming from AI or not. And we still, I think, now realize that there's improvements that we need to make. And so traceability and tying it back to grounded in sources that can be verified by a reliable and authoritative source is really, it's really like that is the key. Particularly because what you know, it's different if you're dealing with just I don't know I use I use chat GPT to plan pretty much my meals, my holidays everything. That's okay you know if it gets it wrong that a museum is not open on one day or another. You're okay with that. But if you're, if you're practicing law delivering advice on the, you have a fiduciary responsibility, the threshold is much higher. Yeah, I certainly agree with all that. I think the, I mean, one of things that we've been careful with is to include citations to all the sources we come up with an answer so that you can go in and look and make sure for yourself. Kind of on the backend, we do a ton of testing with a whole bunch of sample queries and get more and more queries from, I mean, we like pass it around internally at CV and get people to use it. One of the things that I think is really difficult with LLMs is that historically like the quality of writing and like the quality of research would tend to track. So if you saw something that was badly written and had lots of typos and mistakes, you'd assume that whoever wrote it doesn't have a lot of attention to detail and therefore the quality of the content is also bad. And you kind of, you could flip that too of like, okay, well, if it's something that's very well written and everything looks really rigorous, you could assume that the quality of the research that backed it would also be good by the person who actually did this, because usually people, like all this stuff goes together. And LMs kinda get rid of that because they write with the same kind of fluent quality all the time, but you don't know whether the answer is right or not. I think the thing that I've, I guess a tip I would share for evaluating a system whether us or anything else that you're looking at is I think it helps to ask questions that you already know the answers to. So if you ask a question that you don't know the answer to, it's gonna look nice because LLMs write things that look nice and you're not gonna know. But if you ask questions that you do know the answer to, then you start to say, I can kind of see past that. There's a, I forget what the name of this is. There's some Dunning Kruger. It Dunning Kruger? No, not Dunning Kruger. There's something, there's a thing with newspaper articles where like, if you see a newspaper article on a topic that you know really well, it's usually easy to just like nitpick it and think that there are problems. But then you go read articles on things that you don't know and you assume that they're right. I feel like that's like this. Like if you go after a topic that you understand then you're gonna be able to evaluate and recognize quickly, well this I know and I know that it's right or I know that it's wrong. Whereas if you ask it kind of random questions that you may not know the answer to, it's all gonna look good. Yeah. It's tricky. I found it super difficult for hiring. I know that's not what we're talking about. But all resumes look good now because everyone passes them through GBT and it's like really, really hard to evaluate your resumes I think. I feel like that's a similar problem for research tools is everything kind of looks good until you you really have to kinda dig into it a little bit. Well and, I mean, from my perspective, I I I think it's also worth mentioning, like, the the most recent practical guidance that the California State Bar updated in twenty twenty six for the use of generative AI in the practice of law, because really what it's saying at the end of the day is like, any use of AI doesn't abdicate professional judgment that, you know, you're still a lawyer, you're still fully responsible for any outputs and any work product that you generate with the assistance of AI. So to me, you know, all the questions that we get about, well, how am I know that it's true? I think what you just said is a really great practical tip for kind of vetting the baseline behavior of how is this tool performing and what's the level of nuance and how is it doing. But at the end of the day, we put the sources upfront because it's still a lawyer's responsibility to check, right? You can't just cite a case without reading it. You have to check your sources and know that you're citing to information that you've ensured is is accurate and it has ensured that it's on point. I think that goes to the heart of kind of like the problem that we've been grappling with internally. As we build, we think about the roadmap for insight, which is these tools are incredibly great time is really like gifted at drafting and analyzing large document providing. But it's all it's doing is really not all but one of the things that we need to be aware of as is that it's moving judgment further up the workflow. That doesn't remove the need for judgment. There still needs to be and so as much as we can build in those verifications and checks, think it reduces the burden on the practitioner to kind of have to verify everything, or at least we can make it clear and simple and easy for you to verify. Here are the sources, go back to the source of truth. I know that if CEB says that, then I can see the sources, then maybe I, you know, I don't have to spend three hours instead of I can spend thirty minutes instead of three hours validating that the output, the work product is good. Yeah, I will say we do do a level of that. I haven't mentioned this earlier, but we do a level of that already, which is like we make sure all the citations are real and like the quotes actually came from those documents and things like that. We do like a post processing step that is not powered by an LLM, that is powered just by computer code that it always works the same way every time to catch some of those issues. Hopefully, they're already rare by the time we get to this point. But if they happen, we don't want those to keep going forward. So we do have some checks in place to to catch those problems. Because hallucinating citations would be a is a pretty big Yeah. Red flag. You know? Yeah. Yeah, I think I was reading recent if you look at the recent surveys that have come out about kind of what are the biggest fears around AI tools for lawyers and law firms, I think hallucinated citations is still the number one worry and that's still the number one concern. I think there's some kind of public database out there that lists somewhere in the vicinity of fifteen hundred examples of court opinions that have called out hallucinated sources globally, and quite a large number of them happen in the United States, but. Kind of what you were saying, Ben, about, you know, some of these tools are really good at making things look good. And how do you ensure that they actually are good? Is that now instead of just completely fabricating a case, which is in a lot of ways easier to catch, the worry is, is it characterizing the case? Do you from a technical perspective, how do you how do you look at accuracy on that level? Just That's a hard question. I think the the so the main the main tool we have to look at this is something I touched on before which is our test sets. So we as we go but we make kind of our own queries. We as an engineering we also work with the editorial team to collect kind of sample queries and sample questions. And then we also have like answer rubrics. And it turns out one of the things that you can do that is very common on the software side is called LLM as a judge where you can ask one LLM to come up with the answer and then separately you can ask an LLM like does the answer contain this information or not? And the LMs these days are good enough that they're like very good at judging, okay, did character, did it contain all of the elements of that claim where these are the four or five elements? Like the judge would do that like very reliably so that will let you catch the like, because that's a simpler task, you can get better reliability than what happens from the actual generation of an answer, which is a more complex task. And then we do that across, I don't know how many questions we have, certainly dozens, maybe a hundred, at least in our kind of standard question bank as we can keep doing that and making sure, okay. Every time we run this, are we saying all the things that shouldn't be there are there or all things that shouldn't be there are not there? And yeah. I mean, it's tough because things are they're sort of by their nature. There's this probability aspect. But you say, okay. Well, if off a hundred questions and we run it a few times, if it's always getting it right, that's a really good sign that we're getting it right consistently. Yeah. It could be wrong, though. Yeah. Yes. There's still the possibility that it gets it wrong or gets it wrong sometimes. Yeah. Yeah. And, I mean, I think it's I think it's important to just acknowledge that. And it also goes back to, again, that what is the lawyer's role? Right? And I think think the really interesting question that this conversation raises is how do lawyers and law firms vet the ROI on using these tools? Are they actually saving time overall? Or is there some, you know, the burden of verification, is it actually taking as much time as it would have to just do the research without an AI tool? So, I mean, I guess I guess the question is, are we there yet, or how how should lawyers think about think about vetting the tools that they're using from a kind of a cost benefit perspective? I'll advance that. Yeah. That's how we use it. So my sense is that the gains are there, but you kind of have to temper your expectations. And I think if you it's easy to go and ask GPT to write you something and it writes something in a minute which would have taken you all day or a few hours to pull up. And so all of a sudden you think, wow, instead of this taking me a few hours it takes me five minutes. But then you gotta go in and check and you've gotta read it. And so maybe instead of it taking you five hours it takes you twenty minutes or half an hour because you have to go check it. Or I don't know, maybe sometimes longer. So there is a real game there but maybe it's not as much as it looks on the surface of like it's magically done. I feel like that's roughly where we are right now where some things kind of happen magically quickly but you have to remember that there, you do need to go read it, you do need to go sanity check. And so, I don't know, sometimes that can feel like, ugh, but it was all here and it was all so fast and now I'm going slower. But I think if you think of that as instead as well, without the AI it would have taken me twice as long or five times as long as that's still great. Like that's still a big time game. What do you think Kelly? Yeah, you know ROI I think is going to vary depending on the legal task, where the legal the type of legal service, where the legal contact, there's a lot of context questions there. You know, in the US for I think about over one hundred and fifty years, it's been this relationship between vendors who provide legal research and content into providers, attorneys and firms, agents, government that deliver legal services. I think the challenge now is to understand what is the responsibility on the vendor and then like CEB of like how do we modernize and take advantage of the tools, so that we can provide real customer value. Right, and that that takes away some of that verification burden, some of that source tracking burden answers real questions but provides all the workings out. I think that's the balance here is you know the the challenge I think between using it comes down to a choice of like is a general model good enough for the profession. I think reasonable people can can argue around that. My view is, you know, if I was health and ethical and regulatory responsibility, I probably wouldn't require rely on a on a general model, provide rely on something that came with a background of authority grounded in sources, editorial and quality processes, because I think that's the difference between a you know, what has been termed a professional grade model, versus something that's really good and really smart, but doesn't have that context and the constraints built in and grounded in authority and, and just know how know how of how the legal system works and operates and structured. Yeah, I think it's a really good point, because I think a lot of times, you know, we do get asked the question of like, why why would I use Insight over another tool? You know, when when is it when should I use CEB versus versus another another vendor? And to me, the question has always been, well, if it's a question of California law, and especially if it's a complex one and especially if it's one where there's intersection of local law and policy and state law and policy, and it's complicated, and the procedures are complicated, and it's technical, and it changes all the time, you wanna be querying a database that has the right amount of depth for the topic that you're exploring because it's it's easy to get it's easy to get a general answer that doesn't actually get you where you need to go as a lawyer. Yeah. Which comes back to the CV content. Right? Like, having Yeah. Having an answer there is Yeah. What it's about. And it's I mean, as a product person, it's it's it's always a really interesting conversation for us is just kind of, you know, where can we provide the most value by deepening the content even more. Like, where where are the opportunities for us to kind of really give people more detail, more nuance, more depth, more local? How can we how can we expand coverage in ways that really make a difference that I think kind of feed the value up into a tool? And maybe that's a good kind of like I want to kind of like add on to that. Because I think that, you know, there's a lot of, it's sometimes it feels like we're living in a magical time where anything is possible. And we are we are, you know, our goal here is to continue doing what what we we've always been doing, which is providing kind of a a structured view of the knowledge of legal, the law and legal system in California for practicing attorneys, not not to not to cross that line. It's really understanding how the tools give us the ability to do even more of that at depth, but never crossing the line into trying to like take the judgment away. We're just trying to surface all of the considerations for the decision maker. I think that is probably a good note to kind of pivot towards maybe looking into the future a little bit. What excites you about the possibilities of what comes next? I mean, I feel like there's so many options of what you could attempt to do with these technologies. Like, Ben, what what are some of the use cases that you think are gonna be the most exciting to explore? So so I think that there's there's a lot of use cases that a lot of companies are getting into already, especially I think around, like, drafting and longer form answering. We think about where the evolution of insight in particular. My sense is in general for lawyers, the amount of time you spend writing documents goes down. Like as LMs are able to take on more and more of it with more reliability. That's not the part that I think is exciting. I I just think that that's kind of the part that's inevitable is that LMs start start taking over more and more of the drafting. I think things I think it's possible that there's kind of like more, I'm not trying to say this, but getting into like kind of totally different parts of law where software hasn't been able to help before. I think that there will be ways that use LMs to like prepare witnesses or prepare for depositions and things like that, where you have kind of like a, you can run like a fake kind of counterparty and like use that to practice. Or CLE can become a more of an interactive thing where you're working with somebody else. I don't know. I like that these kind of unformed thoughts are where I start to get excited into like, wow. What what really could you do? And I don't think the models are, like, quite there to do that in a really complete way yet. But if think about, like, a few years, I think that that gets interesting and exciting. I think in the meantime, starting to pick up more and more of the kind of written legal work is where where things go. I've heard you I've heard you mention tagging as an exciting kinda use case? Yeah. Yeah. So I think that there's a lot we can do on we talked before about, like, finding the right content and finding the information. And because you mentioned not just kind of the raw data itself, but also the metadata around it and how the various pieces of content we have associated with each other and with legal topics. And historically you'd probably ask a person to do it. I mean, at Judicata we did some automation of some tagging, like the on CV when you search cases, the like disposition and cause of action fields, those are a combination of manual and automated tagging from PLM technology. But there's definitely room to go much broader on that. And I think, I don't know how much of that actually like shows up for users, but it makes all the content connect to each other in the backend in a way that we're able to surface like kind of more smooth experiences of connecting pieces that we wouldn't have been able to connect before to go from the case on a topic to an online document that maybe doesn't cite that particular case, but is talking about the thing that you wanna talk about to some checklist or some step by step guide for something. Yeah, I think it would have been completely impractical to do that historically, but now it's like, okay, well maybe you can do that. And maybe for me as an engineer, I'm like, okay, how do I get an editor to do that? Like how can I give an editor the power to start identifying and connecting these documents so that we can feed it back and use that to build more seamless experiences? Yeah, there's a lot there. I could keep rambling. I thought maybe I'll stop rambling. You're very excited. Was genuine excitement. What about you, Kelly? I think the thing that I find most exciting and exhilarating to some extent is like the opportunity for us to really continue to grow CDs mission and impact in the market so you know you through this technology. I love the full circle nature of it, which is in nineteen forty seven the very first thing that CB did was to assemble a group of experts in a room in southern California to teach about changes to the tax codes. So we built up this muscle around capturing knowledge about the practice of law, the nature of the law in California, and structuring it in a way that can be disseminated and distributed right in. And what this does now which I know just as a kind of you know a reminder for for everyone here that CEB is a nonprofit. And so everything that we generate through our commercial activity goes back into supporting that core mission and building that. And so as we start to grow and build that impact these tools really unlock an incredible leverage for us to do this at a depth that would have been unimaginable even two years ago from a cost and time perspective. And so I think that's really exciting that we can grow with, with the profession with the complexity that's happening right now in California, especially. And it's just it's a great puzzle piece like you know this, the ability to ingest large volumes of content, target at scale, surface insights, it's just a kind of a geek's dream, honestly. It's like an informatics dream. Yeah, I mean, we launched SequaHub a couple of months ago, and one of the reasons that was so exciting was because it was really a great foray into a really tangled area of local policy and ordinances and state and some federal and a lot of administrative activity. And when I think about areas like that in California, like land use is such a good example. There's so many areas where there's a level of complexity that that like, what if you layer the right tool on top of the right content set, it's really exciting to think about what could be possible for people. So I'm I'm certainly interested in seeing what we can play with over the next couple of years. Me too. And one thing I want to kind of be explicit about is we have been relatively late to market with Incyte. We've been on our drawing board and in discussion over a year now, I think, Ben and Stephanie, you've been very close to it. And part of it was really making sure that we can ground it in the guardrails that we wanted to put in place. Part of it is understanding that we're coming from a position of already established trust in the market, and that trust doesn't automatically transfer to this new paradigm. And that we want to be really thoughtful about not breaking that trust. So anything that we put out into the market, we can be really transparent about how we built it, what's gone into the, to the machine. And you can verify and trace trace the workings out, right, I think that's really important. And that's not that sounds a lot simpler when I just say it in a sentence. But if you just if you listen to Ben's analogy about having the LLM create answers and check and validate and it's, it takes a I think there's a lot more design, right, that has to go into how we construct the system. Well, I think I should probably dive into some of the questions that we've been getting to see if there's anything else we can address. I've got, I'm not sure we'll be able to get to all of them because there's quite a few. Thank you, everybody, for submitting. I've got one gentleman who's asking that as a judge serving on an arraignment calendar, should I be concerned whether a discovery motion or request for continuance has been generated by AI? If so, why? I I mean, I I kinda feel like we partially address that in the course of the conversation, which is that the worry is whether any of the information in the motion is inaccurate, right? That's really the underlying concern. And I think we touched on it before, which is none of these tools obviate the need for an attorney to exercise their good judgment and to verify the information that they're putting forward. I think I don't know, I I'm friends with a lot of teachers, and so I've have been having a lot of conversations about tools that kind of catch purport to catch whether or not content has been AI generated. But that can be problematic for people who either use more complex grammar or use an em dash. Apparently, em dashes are now considered a sign of AI usage. But but I don't know if we have anything to add to that. But I I think generally, the concern is, is the information that's being submitted accurate and the responsibility on that accuracy falls on the on the lawyer. Honestly, I think as the judge is just really hard right now. Like, I don't think there's really, like a great answer to like, oh yeah, you just put the motion here and it tells you whether the lawyers made it up or made it up by accident or something. Like, think you should assume that you're getting AI written documents. Like, imagine you already are and you'll continue to get more and more. That's not very helpful. Like, don't know. I think there's a good answer to like, okay, how do you know whether it's bad or not? I think a lot of that does go to like, you'd hope that the counter, the lawyer on the other side would catch some when they respond or something like that. But yeah, no, I think it's really hard judges. My guess is it continues to be hard for judges. So sorry, that's bleak. We're a transition moment across all professions right now, how to how to adapt professional standards and usage around something that is just so incredibly powerful. And so I don't I don't doubt and I think you referenced the state law regulations and guidance. And so you know, I think that there's an ethical and professional responsibility to verify and apply a weed. I think I think it's also going to be an interesting year to keep an eye on the California legislature, because I don't have a complete list in front of me. But I'm I think there are several pending bills that deal with kind of AI and other technology in various capacities. And so policy and governance hasn't quite caught up with adoption, so we're in that weird little middle zone right now. We've got a question and a compliment for you, Ben. We've got someone saying, I was super impressed by the early NLP work at Judicata back in the day, which I think relied on legal linguistic structures rather than just machine learning. How does the current LLM architecture continue that sort of approach, and how much is it totally different? In short, it's totally different. Like that's the I I think that there's a lot they're not super compatible. You can kind of get an LLM to generate the linguistic structures depending on your task, but also you can ask LMs directly for a lot of that information. And if you do that right, you can get very good results. So we were kind of talking, to me this feeds into the discussion earlier of like tagging content at CEB. Part of how I mentioned like I wanna get the editors to be able to do this because I think the real way you get kind of powerful information here is by having subject matter experts able to write like basically prompts for LLMs that do the tagging, that do the identification and give them the tools that also supports the accuracy and checks the work and make sure that things are kind of right. And as we make changes, things improve. So I think that there's a lot of room for us to explore that, that we are starting to do and starting to talk about how to bring that together. But I think the really interesting thing is taking the engineers actually build the tools and the legal and the, like, knowledge really comes more directly from editors. Whereas in the past, we kind of had to, like, have engineers understand enough of the law to be able to encode it. I think we will no longer have to do that, which unlocks a lot more power, I think, for us in terms of what we can we can put in the product. We've got a question. Is Insight closed universe, or does it look outside of CEB? It is closed universe too. Closed universe. Yeah. I think and I think that really goes back to a lot of this conversation about how cautious we're being about trust and accuracy, where we have our own very stringent editorial processes. We want to take full advantage of those to make and maintaining a closed universe helps us do that by only relying on highly vetted source material. The sources are really critical to us. And so we're very, very judicious in our use of sources and what we reference. Yeah. I think it hasn't been asked here directly, but we get the question a lot. Also, does CEB train Incyte on user inputs? No. Easy easy answer. No. We we do a lot of internal testing testing with our team I mentioned earlier to, like, generate queries and generate expected inputs and outputs. But that is all with CB. We we do not do that with any user information. Yeah. Always good to emphasize that. So sorry. Let me just scan through these for a second. I guess there's some questions in here which I could kind of look at the general theme as to, you know, where is the line between analyzing the law and actually practicing law, or then do do some of these AI tools approach or cross that line? I don't know if either of you have have thoughts on on that. Kelly, do you wanna Well, sorry, Stephanie. I was looking at the questions. Oh, yeah. So I think it's I think the question is, you know, I'm I'm paraphrasing here. So as kind of insight or any kind of AI enabled tool gets more into analyzing the law, is that crossing line over to a vendor like CEB actually practicing the law itself or providing legal advice? I can't speak on behalf of other vendors clearly because I don't know what everyone's strategic plan and road map is. I can definitively say for CB it's this is about deepening our own with staying in our lane and deepening and adding more value to practitioners and the system, the attorneys in the market. There will be no there's no delivery of legal services here. I think inevitably because of this transition period and the power of these tools what you may find is that you can deliver, you can create a work product more effectively, but there is still an attorney that is part of the credentials, is part of the bar subject to ethical and professional and sometimes fiduciary obligations that is at the other end of that product that's delivering the legal service. That's absolutely not our lane. I think we have a lot of strengths and this just amplifies those strengths but that's not one of them. Ben, were you gonna add anything? Yeah, I was gonna say I feel like that kind of is if it's going through an attorney then the software is like, to me, that's not practicing law. If you're providing legal instruction to someone who's not an attorney, then that's where you start to get into, like, is this practicing law? So yeah. So to Kelly's point, we certainly are on the side of giving this information to lawyers to then use in their professional judgment. Yeah, but it certainly is interesting where you can totally ask GPT or Gemini or whatever a question that starts to get very borderline no matter what kind of disclaimer they put on it. Yeah, and I think, I mean, looking at InSite as a practical example, in layperson's terms, really what it's doing is it's summarizing and synthesizing information that exists in practice guides and statutes and cases. It's an extra layer of legal research, really. That's exactly it's supercharge retrieval and discovery engine that we've added on top of our corpus. And actually, it's probably worth mentioning, I don't know how many people on the call are familiar with AB three sixteen, which I think was already effective as of January one where basically, what that law says is that it it prohibits anyone in the AI supply chain, including end users, from escaping liability by just claiming, well, the AI made an independent decision. It wasn't me, which is, again, goes back to the lawyer as end user saying you have to vet and verify and take responsibility for the for the materials that you're submitting. Well, I think we are nearly out of time. I know there's still a lot of questions. We will review all of those and see if those can either lead to a future webinar or if we can answer them in some other forum, but I'm really appreciative, about everybody's, inputs. Any closing thoughts or messages before we log off from either of you? No, thank you very much. This was Yeah. Hope that found this all interesting and, yeah, I'll see if Stephanie sends more questions my way. Yeah, will send them your way for sure Ben. Thanks everybody for joining us on your lunch hour. We really appreciate your time and hope you have a good rest of your day. Thanks very much.