High-flying law students perform worse in exams when using AI

Avatar photo

By Rhys Duncan on


Lower performers saw significant benefit

The grades of high achieving law students suffer when they are given access to artificial intelligence tools, new research has found. But low performers saw a significant increase in performance.

Forty-eight law students at the University of Minnesota were given a final paper without AI, and then, after prompt training, were given a second paper and permitted to use GPT-4, OpenAI’s latest and most advanced AI model.

On average, AI-powered students scored 29% better than when in mere human form. For lower performing students this improvement was 45%, although for those at the top of their class scores dropped by a whopping 20%. Whilst the AI could boost multiple choice scores, its essay writing wasn’t quite up to scratch.

The 2023 Legal Cheek Law Schools Most List

Commenting on the results in their paper, professors Choi and Schwarcz said “GPT-4’s impact depended heavily on the student’s starting skill level…This suggests that AI may have an equalizing effect on the legal profession, mitigating inequalities between elite and nonelite lawyers”.

Earlier this month, Legal Cheek reported on another study suggesting that AI will lead to more legal work being done by those without traditional qualifications, with the profession opening up to experts in the computing and coding fields.

Many firms have already sought technological solutions to expedite more administrative processes like document review, freeing up time, and allowing the human machines to work on more complex issues.



It’s particularly interesting that the use of AI by top performing students was to their detriment. I’m curious as to why there was this discrepancy in positive outcome compared to lower performing students.

The use of AI in legal education needs to be such that there are benefits across the board, maybe not equal benefits realistically thinking, but certainly not regression.

The computing sector taking a share of legal work is no surprise. How law firms intend to react to competition from a sector they’ll coexist with long term is yet to be seen. A similar situation existed and still does with the consulting sector.

Why would computing sector want to enter law?

The computing sector won’t take a piece of the pie from lawyers. Legal advice is still reserved in most significant areas – so you still need a legally qualified person to dispense that advice. I.e. a machine by itself can’t.


AI currently performs to a B- level in tertiary essays. That means it will be a big help for those in the new unis form whom B- is the standard expected for a first.

Doubt it

If you had an assistant during an exam you’d ask it to check for errors quickly or provide comments on structure, not just write it. GPT-4 can produce peer-reviewed publishable work with the right prompting. I doubt it’s doing that detrimental.

Deed U No

“Open up the pod bay doors-HAL “

Join the conversation

Related Stories

Solicitor aims to become first ‘AI-powered MP’

Andrew Gray hopes tech-focused approach will lead to election success

Jul 19 2023 9:17am

Will AI replace lawyers?

Technology associate Phillippa Stubbs considers whether the rapid evolution of large language models means the end of legal careers

Jul 17 2023 10:55am