Vals AI has invited legal technology vendors to participate in its benchmarking studies on legal AI, including an upcoming assessment of U.S. legal research tools.
Readers might remember that in February, Vals AI released the first benchmark study focusing on legal AI tools.
The Vals Legal AI Report (VLAIR) represented the initial systematic effort to independently evaluate legal AI tools alongside a control group of lawyers, using practical tasks sourced from Am Law 100 firms.
This report did not cover legal research products; however, Vals mentioned that it would publish a following report dedicated to benchmarking legal research.
Following the report’s release, some vendors indicated they were unaware of the study and had not had the chance to participate.
Vals is now encouraging maximal involvement from companies offering generative AI legal research tools for this and future legal AI benchmarking studies.
“We decided to differentiate legal research from the rest of the study because we aimed to guarantee representative vendor participation and a sufficiently varied set of questions,” Tara Waters, the project’s lead, stated in an email.
“We are gathering 200 Q&AS from law firms. As before, we will include a human baseline.”
How to Express Your Interest
Firms interested in having their products evaluated—whether in the legal research study or in future studies—should fill out this Vendor Interest Form.
Vals also welcomes law firms, solo practitioners, and in-house attorneys to join its benchmarking consortium.
Law firms keen on participating should complete this Law Firm Interest Form.
Waters mentioned that the legal research benchmarking will differ from the previous VLAIR tasks in that human assessors will evaluate both the attorney group’s responses and those from the AI products.
“We will have a combination of law librarians and independent legal reviewers,” she stated.
“The evaluation will consider accuracy, authoritativeness, and suitability of the responses, with applied weighting.”
Waters is also one of the key figures behind the AI Adoption Index, which, as I recently reported, seeks to create an accurate, comprehensive, and transparent index of AI use among law firms, in-house legal teams, and individual legal practitioners.
The Vals legal AI study was created in collaboration with Legaltech Hub and law firms, including Reed Smith, Fisher Phillips, McDermott Will & Emery, and Ogletree Deakins.
Source: Law Sites
Leave a Reply