10 January 2020

Robot Lawyers Are Not the Future

We also decided to offer individual lawyers the opportunity to assess their own expertise for free. This decision was inspired in large part by my experience as a PhD student studying how legal experts think. At that time, I needed the voluntary participation of a large number of legal specialists who would agree to give me at least an hour of their time, let me test them with very challenging legal problems, record their real-time responses using an audio recorder, and to do all this for free! Needless to say, few people thought any lawyer, let alone my target sample of lawyers from the most prestigious law firms, would agree to these terms.


But they did. And I’ve always wanted to give back to the legal community in ecognition of their generosity. So we created the Meisterline Index Self-Assessment service, which, if you’re a lawyer and your jurisdiction and area of legal specialisation are covered by the Index, you can take here. A side benefit of offering this free service is that lawyers who are skeptical about our assessment process can test it, confidentially and with zero cost. They can then determine for themselves whether or not we know what we’re talking about. Lawyers are known for their critical thinking skills, so giving them the chance to test us like this has been a great ice-breaker. What we didn’t anticipate was just how attuned and self-aware many of them seem to be in relation to their own levels of cognitive development.

Let me give two examples to illustrate what I mean.
One of the first lawyers to complete her self-assessment was Jane (not her real name). She’s a technology lawyer in the UK and we assessed her to be an expert with a Meisterline Index expertise rating of 7.7. I should point out that as with all our Meisterline Index Self-Assessments, Jane was only told in her assessment document that she was a Band 5 expert (a designation that covers expertise ratings from 7.0 to 8.9). A lawyer’s specific rating number is only available in their paid
Meisterline Index Development Reports and Rating Certificates.
When we asked Jane for her feedback on how accurate she thought our assessment of her expertise was, she responded: “It looks about right to me.” This might seem like faint praise, but it was actually a vote of confidence in our approach and Jane was won over. This was good feedback, of course, but then we started to notice just how self-aware Jane and many of other lawyers who tested our services appeared to be. We even began to wonder whether our expertise ratings were simply the objective confirmation of what these lawyers already knew about themselves.

We asked another lawyer, whom I’ll call Charles, which aspect of his development as a legal specialist he believed had the greatest scope for improvement. Charles had received a Meisterline Index expertise rating of 8.6, which meant he was more expert than Jane, a comparison we were able to confirm insofar as they both practised in the same area of law albeit in different jurisdictions. What we wanted to find out was Charles’ level of self-knowledge and specifically whether he’d respond as we thought he would given his level of expertise.

In broad terms, we asked Charles to tell us whether he thought his immediate developmental focus should be his knowledge of his chosen specialist area of law, his ability to understand key issues and what a client really wanted in a given instance, his judgment in consistently choosing the best possible options and making the right calls on difficult issues, or his intuitive ability to transcend the details of a difficult file or case and to quickly develop an accurate diagnosis and an effective legal strategy.
Any of these options would have been valid choices for a lawyer of Charle’s abilities, but if he chose the last one (developing his intuition) it would be an indicator that he knew with some accuracy the level of his own expertise. And that’s exactly what happened. Charles’ unambiguously said his intuition was where he had greatest potential to grow. I won’t go into the details of what further information we used to analyse Charles’ assessment of his own abilities, but that information also confirmed that both he and we were on the same page when it came to measuring how expert he actually is.

As for Jane, she identified her judgment as the area of her greatest future potential development, which again fitted with our assessment of her as a lower-level expert than Charles. Focusing on developing her judgment is indeed the area where Jane is most likely to experience the biggest dividends in terms of her growth as a legal specialist. Had she identified here intuition as her immediate focus, this would have been getting ahead of herself in a cognitive development sense, though it should also be noted that she cannot ignore the development of her intuitive abilities even now.

Obviously I’ve simplified these examples and haven’t told you about the other lawyers we’ve tested and how they’ve performed in their self-assessment tests (neither have I explained how our findings mesh with previous studies like those concerning the Dunning-Kruger effect). Much of this information remains the subject of further research in our lab. What I can say is that there is a growing body of empirical evidence that legal experts are indeed self-aware as to their levels of specialist expertise and that their perceptions of how good they are align consistently with their assessed Meisterline Index expertise ratings.

In writing this brief article, I wanted to provide some feedback to those legal experts who’ve already used our rating services. I also wanted to report more generally on one area in which we’re learning more about how legal experts think. If you have any comments or questions, please don’t hesitate to leave your thoughts below or contact me if you’d like to discuss these issues further.

Share on:

More News

In Defence of the Legal Curmudgeon

January 10

In Defence of the Legal Curmudgeon

Read More
Lawyer Know Thyself: Emerging Evidence of Self-Awareness Among Legal Experts

January 7

Lawyer Know Thyself: Emerging Evidence of Self-Awareness Among Legal Experts

Read More

See All