Comment

The wretched Watson Glaser test and why I think it should be scrapped

By on
58

Why do grad recruiters still use the 85-year-old test?

lead12

The Watson Glaser is an aptitude test designed to assesses job candidates’ understanding and analytical abilities. It is used by a number of law firms in their training contract recruitment process. Today, we hear from one law student aspiring solicitor who thinks law firms should ditch the test altogether.

The Watson Glaser test, which claims to measure critical thinking and be an accurate indicator of performance as a lawyer, is an unnecessary hurdle for future lawyers and an inaccurate indicator of a candidate’s potential career success.

Students who have applied for training contracts and vacation schemes are all too aware that the 85-year-old test is an essential form of assessment for a number of magic circle and silver circle firms. However, whether this test is effective is another issue all together.

Having applied for numerous City-based training contracts and vacation schemes myself, this is an honest account of why I feel the test is ineffective. I believe — despite what it might claim — that the Watson Glaser is surplus to requirements.

A bit about the test. Essentially, it is used to measure cognitive abilities in professionals. This is tested through a series of statement-based questions and accounts via a written exam. A candidate is told not to use their knowledge but rather to read the given scenario and answer the questions without applying any current knowledge. This therefore allows an individual to apply their reasoning.

The test covers four key areas of ability that wannabe lawyers should have: to recognise assumptions, to make deductions, to come to conclusions and to interpret and evaluate arguments.

However most if not all of these abilities are already tested in various other assessment day activities, like group exercises, presentations and interviews. Focusing more heavily on these methods would allow individuals to have confidence in the recruitment process because decisions are not based on the outcome of a test but rather an individual’s ability to impress with their personality and interpersonal skills.

Conversely, there are essential lawyer skills which are not tested in the Watson Glaser. Sure, graduate recruiters could argue that’s why there are other assessment day activities in place to do that — but that argument does not work where a student has performed well overall but is not offered a training contract or vacation scheme because they failed the test.

I have personal experience of this. Having done several of these tests myself, I had on a number of occasions passed the online test but failed it when having a second go at the assessment days. This shows the test is somewhat unreliable because at one point my potential success is rated highly while at other times it isn’t. I was even rejected from a vacation scheme at a top international firm solely based on the outcome of the Watson Glaser test. It felt like a massive blow, and ultimately led to my disbelief in the assessment method.

When we spoke to a graduate recruiter about this law student discontent, she told us:

The thing to remember is that this sort of test should never be used as a sole measurement of success. Watson Glaser tests a person’s ability for analytical thinking. Clearly analytical thinking is an important part of any lawyer’s role. It tests how quickly and competently a person can analyse statements and deduce facts or make accurate assumptions. Clearly a lawyer has to be competent in a range of other skills including effective communication, for commercial law a keen understanding and interest in business and also to be good in team situations so they can work efficiently and effectively with colleagues and clients.

When asked specifically about how the firm would handle a situation where a candidate has performed really well in other forms of assessment but does not pass the test, she added:

When looking for candidates who most accurately match what is required for the role you are recruiting for a candidate has to be strong in all aspects of the assessment process.

While law firms continue to use this test as a good indicator of who would make it as a top lawyer, I really hope that they become more reliant on other mechanisms of assessments. Graduate recruiters should make decisions on the entire assessment day and not just one element of it, especially not the Watson Glaser, which seems to return inconsistent results. Surely, this cannot be the test that decides a person’s success in the law profession; a lawyer’s skills are not limited to how they respond to a critical thinking test.

For all the latest commercial awareness info, and advance notification of Legal Cheek’s careers events, sign up to the Legal Cheek Hub here.

58 Comments

Anonymous

Not really seeing any strong argument for scrapping it.

Just sounds like you’re complaining because you aren’t very good at it…

(100)(13)

Anonymous

Performance in these tests usually improves with practice. I am an in-house attorney and we were required to do the test [online] recently. Some colleagues who do have good analytical skills scored poorly and I “fluked” a 94% score. Doesn’t mean my reasoning is better, just that I got a good score when I did the test.
Some 3 years ago, I applied for a part-time judicial post in the UK and I had to do an online IQ tests identifying sequences of shapes, patterns etc (which I found very difficult) with some simple reasoning questions. I didn’t finish it.
I suspect that many partners in the MC firms, and sitting judges who have been in post for years, would fail the tests but can do their jobs perfectly well. Conclusion? They are not a good measure to determine who will make a better employee.

(13)(1)

Anonymous

Hmm, try winning a case on the basis of such anecdotal evidence.

(5)(2)

Anonymous

“I was even rejected from a vacation scheme at a top international firm solely based on the outcome of the Watson Glaser test.”

Really? Who? I doubt that any firm would reject just based on this test alone. I suspect you aren’t up to it for other reasons.

(18)(18)

Anonymous

Given that firms have so many applicants, I’d imagine that they would indeed reject someone based on poor performance of a single test. If nothing else, it’s a quick and easy way of thinning out the herd.

(48)(3)

Anonymous

You clearly don’t know what you’re talking about. Some firms, like Clifford Chance, won’t even read your application unless you pass that test first.

(27)(0)

Anonymous

You mean the verbal reasoning test, then comes the Watson Glaser once an assessment day have been gained.

(1)(0)

Anonymous

Not necessarily, I know of at least one firm for which after your application has been approved, you must also pass a shorter online version of the Watson Glaser before being offered an interview/assessment day.

(2)(0)

Vinny

Keep these arelcits coming as they’ve opened many new doors for me.

(0)(0)

Egypt

Apaetpnrly this is what the esteemed Willis was talkin’ ’bout.

(0)(0)

Anonymous

Could not agree more. Along with video interviews, it seems like just a way to cut down numbers due to lack of capacity to interview the sheer number of applicants.

It’s not coincidence that firms with fewer applications do not have these sorts of tests.

Nobody can say, hand on heart, that there is this magical test that can determine with accuracy whether or not somebody would be a good lawyer.

(23)(1)

Anonymous

The firms face vast numbers of applications for a limited number of TCs. They have to apply a range of filters to get them down to the final number.
While the WG is not perfect, I would suggest that there is at least a relatively strong correlation between WG scores and lawyering ability, and as such it is a useful tool for the firms.
Unfortunately for you, the grad rec system exists to serve the firms, not graduates.
If you are good enough, you should eventually succeed somewhere.

(14)(2)

Deon

Gee wilelkirs, that’s such a great post!

(0)(0)

Anonymous

“Surely, this cannot be the test that decides a person’s success in the law profession; a lawyer’s skills are not limited to how they respond to a critical thinking test”

The LEGAL PROFESSION also requires writing skills.

(18)(6)

Anonymous

I think the principles behind the test are sound- lots of people don’t have the ability to concentrate and think carefully enough to be good at law. However, a big problem is that you can practice the tests, and someone who has done a it of practice will have a big advantage over someone who hasn’t- so it’s a bit of a preparation game.

(13)(0)

Anonymous

Unfortunately this article is poorly reasoned and demonstrates a lack of critical analysis, which tends to undermine its premise.

(55)(2)

Anonymous

#wrekt

(3)(3)

Anonymous

* #rekt

(3)(2)

stuckonaboatinbermuda

“Not really seeing any strong argument for scrapping it.”
“I think the principles behind the test are sound”

heh, looks like HR found the comment section.

It’s just one of those things they do to justify their existence rather than effectively screening applications and interviewing. And no, HR should not be interviewing even graduates. Slaughter and May, for example, has a good recruitment process – it screens applications and has partner interviews only. There’s a reading and writing test which are pretty easy. Other good places do this too. Barristers pretty much exclusively do this. They’re full-time working professionals, they have effectively zero HR – they form panels and screen and interview themselves entirely. A law firm has a lot greater resource.

The problem with a Watson Glasner or any other online test is that it boils down to luck and chance and doing well at this particular kind of test which has absolutely nothing – absolutely nothing – to do with the job at hand. Your exams are university should demonstrate your intellectual ability and the interviewers can take it from there.

(10)(10)

Anonymous

Agree with you re HR v partner interviews.
Partner interviews are normal conversations with human beings.
HR interviews are wooden and mindless hoop-jumping exercises.

(10)(1)

Jones Day Partner

Interviews with me involve putting wood into hoops

(6)(3)

Louise

Your posting really staneghterid me out. Thanks!

(0)(0)

Anonymous

Slaughters outsource their application screening stage to an external recruitment firm

(4)(0)

Anonymous

This.

(0)(0)

Anonymous

Slaughters has a poor recruitment process. I’ve worked there and heard even trainees say this. They focus solely on grades without looking for any previous law related work experience. Doesn’t sound like a good way to hire future trainees at all.

(7)(0)

Adam "I have small hands" Deen

No – it’s a great process – YOUR GRADES DETERMINE YOUR FUTURE AND HOW CAPABLE OF A SOLICITOR YOU WILL BE

(1)(6)

Anonymous

Lots of people are vying for a coveted TC. Most of them are qualified for that position. Why not get someone who is overqualified for it?

(1)(2)

Anonymous

Perfectly said, how can firms reject solely on a crap outdated test. They say they’re innovative but use such a platform to test out applicants.

You should have mentioned how firms just use this platform to filter down their applicants. Alot easier to reject on a test than actually reading the full application. I personally think that’s why alot of firms use such tests.

(1)(4)

Anonymous

Do you actually think it’s “alot”?
Once is a typo. Twice explains the failures that I assume led to the resentment evident in your comment.

(5)(5)

Anonymous

Having been involved in some analysis on WG and other assessments, it is one of the strongest predictors of performance in the LPC and on the job as a trainee. Much stronger than A-level results, and also it also makes interviews seem like the most inefficient way to recruit where their predictability is so low.

If you really want to put an argument forward for scrapping types of assessments or recruitment criteria, unfortunately evidence would suggest you would rely on the WG more and remove partner interviews and UCAS points as a minimum criteria (and I can’t see firms doing that anytime soon).

(12)(1)

hrhrgoawaycomebackanotherday

Your comment speaks for itself really. Talking about “analysis” as if anything you’re talking about could be held up to the slightest scrutiny. When has anyone in the history of humanity ever given a slightest toss about the LPC? And “on the job as a trainee” speaks volumes about whatever micromanaging non-shop you work for. The most important things you need by a country mile as a trainee are people skills. And then an element of organisation. These things simply cannot be tested in an online test.

(3)(9)

Anonymous

Wow. Someone’s chippy!

Firms do care about the LPC if they pay students to pass the course, give them a maintenance grant and then they fail it. Firms are risk adverse and will find ways to minimise that risk. If that is using a test that predicts you are less likely to fail the LPC, then that is just one way of doing that.

On the job performance is also about the people skills you talk about. If you take performance data, through appraisals ratings and evidence, that will include details of how individuals work with others . From my experience in reviewing recruitment processes, that data shows that higher performing individuals are more likely to have a higher performing WG result. And that correlation is much stronger than other assessments.

As has been highlighted by others, I agree these tests shouldn’t be used in isolation and that the type of skills you talk about are equally important to assess through other means. However such assessments are far more likely to have conscious or unconscious bias to them. Some partner interview analysis has actually shown you would have better success rates by flipping a coin than trusting an interviewer’s judgement.

(3)(1)

hrhrgoawaycomebackanotherday

I’m not chippy, I’m a solicitor and calling it as it is; no one cares about the LPC and everyone passes it. I’m sure there is the odd person every other hundred strong MC intake fails it. It’s not an issue firms actively consider. Academics are all the screening needed.

WG is NOT designed to test people skills. It doesn’t test people skills. It’s supposed to test reading comprehension and the ability to make inferences. There is no necessary correlation between better reading comprehension and people skills – anything you find is anecdotal coincidence.

And stop talking about “analysis” to cover up the fact you’re talking bullsh1t. Partner interviews are conducted for all levels besides graduate. And all companies use them for recruiting staff – not just law firms – at all levels. You’re the outlier maverick suggesting everyone else is wrong because of some “analysis” you did on the back of an envelope under the sheets in your mother’s basement.

(3)(2)

Anonymous

If academics are all the screening needed, why do firms bother with applications forms when they could just ask for transcripts?

And if people skills were the only important part of performance on the job why would academic performance be important?

(2)(0)

hrhrgoawaycomebackanotherday

Who said people skills were the only important part of the job? You did.

You can judge motivation, intellect (of a wide variety) and people skills from an application form and interview.

In fact, though, that’s basically how many barristers and Slaughter and May recruit. With Slaughter and May, all you have is the academics and a short covering letter of a few hundred words. If you have a first, you have a fair shot at interview. Barristers sometimes put in a legal problem question into the application form which they may further discuss at interview. Usually the best places will have you debate a topic at interview which helps test a variety of metrics.

(1)(5)

Anonymous

Your words – “The most important things you need by a country mile as a trainee are people skills.”

(0)(0)

Hrhrgoawayguy

OK. OK. I night have been wrong. You’ve demonstrated the need for a Watson Flasher to filter out gimps like you without basic reading comp.

Anonymous

Barristers don’t need to undertake something like the WG test as part of their recruitment process as they can rely on the Bar Aptitude test as part of the process of getting on to the BPTC. That Bar aptitude test has a lot of similarities to the WG test.

(0)(0)

Anonymous

Slaughters outsource their application screening to the type of “non-shop” organisation that provide this type of “analysis” on recruitment processes to their other clients.

(0)(0)

Anonymous

Any analysis, even if it is on a back of an envelope, would be better than your thinking which lacks any supporting evidence and is just a unsubstantiated opinion.

Inefficient recruitment assessments like interviews are kept in place to keep internal people happy and comfortable with hiring decisions. It is the same reason firms continue to use A-level filters for absolute fear that they don’t want to be seen publically as dropping standards, despite there being significant evidence out there that A-levels mean sweet FA, mainly due to the inequality in standards of teaching.

(1)(1)

Huh?

I don’t understand this.

“However most if not all of these abilities are already tested in various other assessment day activities, like group exercises, presentations and interviews. Focusing more heavily on these methods would allow individuals to have confidence in the recruitment process because decisions are not based on the outcome of a test but rather an individual’s ability to impress with their personality and interpersonal skills.”

Why would individuals have no confidence in the recruitment process unless decisions were based on their personality rather than the outcome of a test (bearing in mind that academic results, which are crucial, are effectively outcomes of tests)?

“Conversely, there are essential lawyer skills which are not tested in the Watson Glaser. Sure, graduate recruiters could argue that’s why there are other assessment day activities in place to do that — but that argument does not work where a student has performed well overall but is not offered a training contract or vacation scheme because they failed the test.”

Assuming (as you have assumed for the purposes of this particular argument, and as you have failed in any meaningful way to dispute in your article) that the WG does legitimately test something relevant to lawyers’ competence, why shouldn’t candidates who have otherwise performed well be rejected because they failed the test? The equivalent logic would say that there should be no interviews, because they do not test everything, and it is not right for a candidate to be rejected where he has performed well otherwise but flunked his interview.

(7)(0)

Opinionated but ignorant

I took the free Watson Glazer test on the website. It’s all about what inferences and assumptions you can or cannot draw.

There’s only one test of inferences that I need:

If someone voted brexit or didn’t vote for Hillary Clinton, they’re a racist!

PS the answer is “definitely true”

(3)(0)

Anonymous

Here’s another question:

There’s only one test of inferences that I need:

Opinionated but ignorant’s momma gave birth to a fool

PS the answer is “definitely true”

(1)(2)

Anonymous

I would also add that the Watson Glaser test is much friendlier than, say, the symbolic puzzles used by a firm like Simmons and Simmons. Those can phase even really smart people if they haven’t come across them before.

(2)(0)

Jenny

An answer from an expert! Thanks for coitbnruting.

(0)(0)

Anonymous

the other methods you have cited are equally flawed, in different ways.

The test isn’t foolproof, has lots of flaws, but the economics are simple. It allows HR to cut out 80%+ of the applications without having to look at them. Of course you’ll lose some decent ones, but among the applications you have left on the pile after that cull will be some diamonds.

(2)(0)

Former HR Linklaters

Linklaters rejects applications based on the Watson Glaser test. If your score is not in the 80th percentile then you will get an automatic rejection after submitting. Even if you fed the whole of Africa or interned at the UN, Linklaters will reject you!

(11)(0)

Anonymous

Haha, ‘fed the whole of Africa’.

(3)(1)

Anonymous

Mirrick is that you

(6)(0)

LegalRec

This “outdated” test has been completed by almost all of the “top” lawyers in our country. I assume that the vast majority of those done fairly well in the test. London is recognised as a global hub for law, if not THE global hub, so clearly it has assisted in some of the best lawyers in the world being chosen for training contracts at the beginning of their careers. It is clearly a good thing and those that resent it are only ever going to be those who have not done well and as a result have inferior legal careers.

(1)(5)

Anonymous

Well that’s obviously not true. There are plenty of elite law firms who do not use the Watson Glazer yet still produce “top” lawyers; Jones Day and Slaughter and May to name a few. As someone mentioned earlier, the test is used to cut out applications so that grad rec have a managable number to deal with as opposed to a measure of how good a lawyer you’re going to be. If the test was such a good measure I am sure that ALL “elite” firms would be using it.

(5)(1)

Anonymous

At least LC have learnt to make the authors anonymous so not to commit career suicide.

(13)(1)

TheAcresOfFour

Yeah, criticizing the legal profession is legal suicide.

Quod erat demonstrandum.

(0)(0)

Anonymous

Having spent a week relentlessly practicing the WG tests on assessmentday, I can confidently say that there are some elements of the test that are valuable to the process of critical thinking, and some that are totally arbitrary, such as deductions. Like anything, if you set your mind to it then it is possible to score highly, unless you’re a moron, in which case you’re f**ked either way. The intensive training did pay off, at least to the extent that I passed the tests and now await some other barrier to access. The inductive reasoning tests should be eradicated, wtf have revolving shape patterns got to do with anything? Unless you’re filing patents for a new Kaleidoscope of course…

(1)(0)

Great Uncle Bulgaria

It’s clearly not suitable for Magic Triangle or Silver Square firms.

(0)(0)

Anonymous

Looks like someone is salty about being unable to pass the test. enjoy your TC at Shoosmiths or Irwin Mitchell!

(2)(1)

Anonymous

I totally agree. Have been practising the damn things for days now, and I simply cannot see how they have more than the vaguest correlation to ability. Particularly the ‘true/probably True/Not enough Info’ etc. ones, where a valid argument can usually be constructed for more than one answer, which means it’s pure luck whether you choose the one the test-setter is thinking of. They test much the same abilities as academic exams, so it would seem to me they are superfluous and just a way of randomly weeding out a few candidates. You’d have the same effect by just throwing the applications down the stairs and eliminating the ones that landed on the third step.

(2)(0)

Keli

Just got a stick of 2GB RAM from him. My Macbook is now as fast as Be18;ict&#e2n7ds service! Best deal yet and at an unbeatable price.Thanks, and looking forward to more purchases!

(0)(0)

Comments are closed.