Westminster Uni academic uses ChatGPT to pass contract law exam

Avatar photo

By Emily Hinkley on


But only just — bot performs as an average law student, Dr Ioannis Glinavos finds

A university academic has pitted ChatGPT against a first-year law degree exam paper in contract law, with underwhelming results.

Senior law lecturer at the University of Westminster, Dr Ioannis Glinavos, asked the bot to answer a selection of sample exam questions taken from a Pearson law textbook available online.

One scenario considered a contract to build a motor yacht for a computer tycoon and required ChatGPT to provide advice in response to supply chain issues and changes to a payment plan. Unfortunately, the bot’s response was marked as “tactical advice, not legal advice”, and unlikely to gain a pass mark in a law examination.

The bot’s responses continued to be hit and miss in all of the questions put to it, leading Glinavos to conclude its performance was equivalent to that of an average first-year law student. “The bot could sit in my class and I wouldn’t know the difference,” he said in the video (below).

Like most students the bot had areas of strength and weakness. Glinavos explained, “it can identify correctly the relevant legal issues but it struggles with analysis and authority.”

The 2023 Legal Cheek Firms Most List

“The bot is very close to the average student performance, thus not easily detectable as an AI-produced answer,” said Glinavos. Ruling that ultimately the bot’s attempt at the exam paper scored around 45% but was still good enough to merit a pass.

He finishes with a warning that current AI-detection tools for exams are not yet advanced enough to competently detect students using these types of tools to cheat. “It will eventually lead to the abandonment of online timed exams that are not proctored,” he predicted.

For all the latest commercial awareness info, news and careers advice:

Sign up to the Legal Cheek Newsletter

Related Stories

GPT-4 claims it could ‘potentially replace’ paralegals

New AI tool reckons it can replicate their research and organisational skills

Mar 23 2023 11:11am

‘Robot lawyer’ sued because it ‘does not have a law degree’

DoNotPay CEO Joshua Browder says claims have ‘no merit’

Mar 13 2023 11:31am