Exclusive: AI chatbot successfully answers Watson Glaser test

Avatar photo

By Legal Cheek on

24

Could ChatGPT spell the end of law firm fav?

Legal Cheek can reveal ChatGPT — the AI chatbot everyone’s talking about, including us — can successfully answer questions on the Watson Glaser test.

For the uninitiated, Watson Glaser is the critical thinking assessment favoured by City law firms as a way weaning down training contract hopefuls during the highly-competitive recruitment process.

But this may soon become a thing of the past after one student reached out to us claiming they had scored an impressive 70% on a mock version of the test, within the required time limit, using only the bot’s responses. Pass rates for the assessment normally sit at around 75%.

“The tests I ran were from one continuous script,” the student explained, “so individualising the script for each section of the test may improve accuracy.”

Keen to find out for ourselves, we ran some Watson Glaser-style questions through the bot and the results where pretty impressive to say the least.

The 2023 Legal Cheek Firms Most List

The AI chatbot enjoyed full marks for four types of question (evaluation of arguments, interpretation, deduction, and recognition of assumptions). However, it came up short when it was challenged with an inference-style question with the added complexity of deciphering the difference between the ‘probably true’ and ‘probably false’ options.

Check out the questions, AI inputs and answers below…

Question 1 — deduction

Possible answers: conclusion follows or conclusion does not follow

AI input + response = conclusion follows ✅

Question 2 – assumption

Possible answers: assumption made or assumption not made

AI input + response = assumption made ✅

Question 3 – interpretation

Possible answers: conclusion follows or conclusion does not follow

AI input + response = conclusion does not follow ✅

Question 4 – evaluation of arguments

Possible answers: strong argument or weak argument

AI input + response = weak argument ✅

Question 5 – inference

Possible answers: true; probably true; insufficient data; probably false; false

AI input + response = false ❌

Could this be the start of the end of the Watson Glaser test?

For all the latest commercial awareness info, news and careers advice:

Sign up to the Legal Cheek Newsletter

Related Stories

AI creativity machine to have its day in Supreme Court

Decision likely to come next year

Sep 14 2022 10:08am
3

We quizzed the AI robot everyone is talking about

Hearsay, TC applications, Donoghue v Stevenson and more

Dec 6 2022 9:59am
4

Future lawyers could bill ‘units of attention’ via computers hooked up to their brains, report claims

Law Society commissioned report reveals possible uses of neurotech in legal profession 🧠 💡

Aug 10 2022 9:12am
12