Page 1
State Bar Admits Failure to Ask Supreme Court Before Allowing Firm to Draft AI Bar Questions
Despite Petitioning Court to Approve Kaplan as Sole Drafter of Questions in October, Agency Did Not Disclose That Testing Consultant Had Been Asked to Prepare Questions in September
By a MetNews Staff Writer
The State Bar has responded to the California Supreme Court’s demand for information about how artificial intelligence was used to develop some of the multiple-choice questions on the problem-riddled February examination, admitting that 23 of the 171 questions ultimately recommended for inclusion in the scored portion of the exam were generated with the assistance of AI.
The regulatory agency also said that some of these inquiries were generated by a testing consulting firm that was not disclosed to the Supreme Court in an October filing, identifying Kaplan Exam Services LLC as the only preparer of multiple-choice questions.
An admission was made by the State Bar last week that some portions of the test were drafted with the assistance of AI.
In a petition filed late Tuesday, the State Bar requested that the court approve scoring adjustments for certain test-takers and a reduction of the passing threshold for those who sat for the examination.
As part of an effort to revamp the bar examination with an eye toward cost savings and the option of remote testing, the agency moved away from using multiple-choice questions prepared by the National Conference of Bar Examiners (“NCBE”) for the February test. In a petition filed last October, the State Bar informed the court that it had entered into a contract with Kaplan Exam Services LLC to draft this portion of the exam.
Attached to the October request was the agreement with Kaplan, which indicated that the State Bar would provide First-Year Law Students’ Examination (“FYLSX”) questions to Kaplan as source materials.
However, the filing did not contemplate the use of multiple-choice questions drafted by any party other than Kaplan.
In the explanation provided on Tuesday, the agency said:
“The State Bar expected that…the multiple-choice questions for future bar examinations, including the February 2025 bar examination, would consist of Kaplan-drafted questions, as well as FYLSX questions that were provided to Kaplan as source materials. However, in late October 2024, Office of Admissions’ staff determined that there were not enough multiple-choice questions for each of the subtopics of the seven subject areas tested. As such, staff requested that ACS Ventures, LLC (ACS)—the psychometric and test development consulting company with which the State Bar contracts to assist with examination analysis, grading, and related services—draft additional questions for the February 2025 bar examination.”
Admissions staff identified the topic areas and the number of items needed. ACS then drafted prompts to yield multiple-choice questions that fit the provided criteria and ran them through OpenAI ChatGPT.
The multiple-choice portion of the February exam included 117 Kaplan-drafted questions, 54 devised by FYLSX, and 29 developed by ACS with the assistance of AI.
In a footnote, the State Bar admitted that ACS had been asked, in September, to develop 14 questions for a bar examination study to be conducted in November. All of these questions were drafted using OpenAI ChatGPT.
Eleven of the 29 questions drafted by ACS and included in February’s exam were developed as part of the November study.
The State Bar said that the AI-aided inquiries that were included in the February test “were among the top-performing questions” from the November study. The footnote continues:
“The decisions by Admissions staff to request that ACS develop questions for the November bar examination study and for use on the February 2025 bar examination were not clearly communicated to State Bar leadership. Structural changes within Admissions have been made to address this issue.”
Content Validation
Once the questions were received from all sources, the State Bar had them reviewed by content-validation panels, consisting of recently licensed attorneys, supervisors of newly admitted lawyers, and law school faculty. Batches of questions were reviewed for content accuracy, bias, and minimum content alignment, as well as whether the inquiries aligned with the State Bar content maps.
ACS psychometricians compiled data based on feedback provided on rating forms. Panelists were not informed of the source of the questions.
After compiling the feedback, an ACS specialist produced a report for the State Bar, which identified whether each question was fine as drafted, should be revised, or needed to be replaced.
All multiple-choice questions were also reviewed by at least one subject-matter expert for legal accuracy, and the State Bar said that it ensured that the inquiries met the subject-matter allocations outlined in the agency’s content maps.
Scored Questions
When the exam included the questions developed by the NCBE, 200 questions appeared on the tests, with only 175 of those to be scored. The remaining 25 questions were considered “experimental.”
The State Bar anticipated using the same metrics for the February exam, identifying which questions should be scored after analysis by ACS founder and psychometrician Chad Buckendahl.
As part of the analysis, questions were screened for “negative item discrimination,” which means that those who performed poorly on the overall test were more likely to answer that inquiry correctly than those who were likely to achieve high scores. A validation panel then assessed whether a minimally competent lawyer would be expected to answer the question correctly.
Of the 200 tested questions, 17 had a negative item discrimination and were removed from the scored items. Six of the 17 removed questions were drafted by ACS.
The remaining 183 questions were evaluated by ACS to ensure that 25 scored questions were included in each selected subject matter. Eight additional questions, none of which were drafted by ACS, were removed.
After ACS made these selections, the State Bar identified an additional four questions, all drafted by Kaplan, which were flagged by a subject-matter expert as potentially inaccurate. These questions were also removed from the pool, resulting in a total of 171 questions to be scored.
Of these, 100 questions were drafted by Kaplan, 48 were adopted from the State Bar’s FYSLX bank, and 23 were prepared by ACS. The pool included 24 civil procedure inquiries, 23 criminal law and procedure questions, 24 tort interrogatories, and 25 questions each for the remaining subject matter areas.
Saying that “[r]emoving these 29 questions improved examination reliability,” the agency noted that “by scoring 171 multiple-choice questions instead of 200, the reliability of the multiple-choice section of the examination increased from 0.87 to 0.89 on a scale of 0.00 to 1.00,” exceeding the “desired range of 0.80 or above.”
Request for Results
The State Bar said on Tuesday:
“This petition requests that this Court adopt the CBE’s recommendations on psychometric imputation of scores and on setting the raw score of the February 2025 bar examination, and issue an order as soon as possible, so that the State Bar can release the results of the bar examination on or around May 2, 2025.”
A raw passing score of 534 was recommended, representing a 26-point reduction from the 560-threshold set by ACS as part of its psychometric analysis of the test. The agency also recommends that scores be “imputed” for missing multiple-choice answers if the test taker has answered at least 114 out of the 171 scored questions.
For missing essays or performance tests, the requested score imputation would apply if the applicant answered at least four out of the six written sections.
Copyright 2025, Metropolitan News Company