Separating the wheat from the chaff
Not all legal rankings and awards are trustworthy – some are scams leveraged by firms, others are pay to play. Even trustworthy publications have different methodologies that are not as thorough as others. Understanding the process is crucial to assessing a ranking or award’s legitimacy.
Legal rankings and awards are not published with the sole betterment of the legal industry in mind (shock!) and that’s the cold hard truth. The legal accreditation industry is a business and like all businesses they need money to sustain their operations.
Scam rankings and awards are a pitfall. Rankings you have never heard of inform you have won an award in a practice area you don’t even cover, usually asking for money to receive the award.
It is apparent that some firms do not care, or are oblivious, about the legitimacy of a ranking or award. Any ranking/award is a good award! Firms will market these rankings and awards, take a profile and buy trophies. These scam publications will continue to exist and proliferate for as long as firms continue to give them a financial lifeline.
There are firms that wish to be ranked in legitimate, independent publications where the rankings and awards are based on facts and merit so they can be used for business development. True research led rankings and awards do exist.
However, even for legitimate rankings, there are always doubts as to whether they are pay to play. These doubts ultimately stem from the lack of/mis-understanding of how the research process works and the business model the publication uses.
This report is designed to inform and create mutual understanding of the different research and business models adopted by internationally recognised publications, however the information should still apply to other players in the market.
Rankings
There are two main research models:
Model 1 – Research led by submissions and peer/client feedback
These ranking publications typically employ a dedicated research team to conduct a thorough review to develop the rankings. The researchers base their rankings on peer reviews, submissions and client feedback.
They consider quantitative and qualitative analysis of law firm case and deal submissions, as well as surveys and interviews with individual private practitioners and in-house counsel.
All these different elements act as a verification process of one another and the information is vetted by a researcher and then validated by an editor before publication.
How the process works
Peer review is primarily used for tiering firms and lawyers. In peer reviews, partners are shown the previous year’s rankings and asked if it still reflects the current market and to provide reasons for any suggested changes.
The researcher will check the submissions and/or independently verify the information received from the peer review.
Model 1 research is multi-faceted, yet firms can be ranked purely based on peer and client review only.
Submissions are used to inform the researcher of the significant deals/cases a firm has worked on, and they will be able to get a sense of how important the cases are by comparing them with the submissions from other firms.
Bear in mind:
- No submissions + negative peer review can = poor rankings
- No submissions + positive peer review can also = good rankings.
If there are no submissions from a firm, the researcher often conduct their own research to fact check.
Client feedback is not used to tier firms and lawyers, contrary to popular belief and what the directories might tell you. Most clients will only have worked with a few firms/lawyers at best so they can only provide a comparison of the firms/lawyers they have worked with. Client feedback is primarily used for general feedback, to verify work and client quotes only.
Model 2 – Research led only by peer review
Research led by peer review normally takes the form of a survey where lawyers are invited to nominate peers they feel are “leading lawyers”. The nominations are then verified by a researcher through their own independent research. A high number of nominations equals a greater chance for a lawyer to be included in the list.
While peer review is important, the results can be manipulated by lawyers eager to be recognised. As a result, peer reviewed ranking/leading lawyer lists can look skewed in comparison to lists found in Model 1 publications.
Though Model 1 research is not perfect, the research process is definitely more robust than Model 2.
But Model 1 is more expensive, so not all publications can afford to do it. A single researcher is able to cover the research for an entire region (or even the world) with Model 2.
The researchers
Critical thinking and writing skills are the fundamental prerequisites for the role for researchers.
Therefore, you may be shocked to learn that many researchers are non-qualified lawyers, have little knowledge of the legal landscape about your jurisdiction and normally move on after one to two years.
With this in mind, submission forms need to be written so the average lay person can understand the deal/case. The harder it is for the researcher to make sense of your submission the more likely they will miss details of how fantastic the deal/case is.
Researchers can be dealing with submissions for over 100 firms and each firm will typically cover many areas of practice which is a lot of volume to get through.
Make it as easy for them as possible. It’s advisable to follow their processes so you can maximise the opportunity. Submitting documents in your format may be easier for you but if it means the researcher might miss details then it’s a wasted opportunity.
Your choice.
Why ranking results differ between publications
A comparison between different publications can reveal great variance in the results even though they employ a comparable methodology.
This variance is due to several factors, including the quality and detail of the law firm submissions, client and law firm participation rates, differences in terminology and categorisation of practice areas, depth of market coverage, as well as the individual(s) responsible for the research process.
The researcher can have a huge impact on the rankings and unfortunately the competence of the researcher isn’t something you can control.
If you wish to know your firm’s “actual/real” standing in the market, it is best to refer to several sources rather than just the one.
Understanding the business model
Publications will reach out to firms and lawyers and offer trophies, firm/lawyer profiles, banners, articles etc. It is hard to know if sponsorship will actually influence your ranking or award unless you try it.
For research led publications, the research and publication of the rankings is free. Offering commercial opportunities is one way for them to generate revenue and your firm will benefit from added exposure through this option.
An alternative model would be to charge firms to submit, but how many would be willing to pay and what charge would be acceptable? Would this be any less pay to play than the advertising model?
A lot of the internationally recognised publications have been long established and have built up a high level of trust within the legal community for their independent research. These trusted publications would not bother hiring large research teams if their rankings could simply be bought.
Just because X law firm places a profile, is highly ranked and you disagree with it isn’t indicative their sponsorship resulted in a favourable ranking. Fact: Less Tier 1 firms/lawyers take profiles than the lower tiers.
Are paid listings a worthwhile investment?
This depends on what your expectations are.
Directories are a validation tool designed to help buyers of legal services with their decision making. The publication will typically publish the rankings and a short editorial which will cover your firm’s significant deals and cases.
A paid listing will:
- allow firms and lawyers to host their own content and build narratives beyond what the publications provide;
- increase their visibility and brand awareness on the publication’s platform; and
- improve SEO.
Lastly, if you feel your firm benefits from the rankings and the publication(s) do a good job then support their business and create a win-win situation.
Awards
Awards are arguably more important than rankings because being shortlisted means a firm/lawyer is one of a very select number of parties considered to be “the best”. Also, awards reflect how relevant law firms and lawyers have been in the last 12 months.
The research process
The research process for awards is normally a simple two-step process: 1) Firms need to submit to be considered; 2) The submissions are vetted, and a winner is chosen by an internal committee.
Are awards pay to play?
Winners selected by the internal committee will note the reasons why a firm/individual has won an award.
When the winners are published, publications should be transparent and provide the reasons for the wins and the reasons should give a good indication of whether the awards are credible. If publications only provide a winners list the validity of the awards should be questioned.
The publications will of course ask the winners for sponsorship, to buy trophies, use their logos etc. If they want to charge you for marketing your win on your own platform it is their prerogative. However, they should be publishing the full nomination and winners list on their own platform at no cost.
As with the rankings, just because X law firm sponsors, wins an award and you disagree, it isn’t indicative their sponsorship resulted in their win(s). Sometimes results can look skewed simply due to the nature of the research process and only a handful of firms participated in the submissions.
The importance of rankings and awards
Rankings and awards are important for strategic business development as they represent the market perception of a firm/lawyers standing in the market. It reflects a firm/lawyer’s overall business reputation, client satisfaction rates, and public relations ability.
It is unfortunate that differences in standards means it is very difficult to know a firm/lawyer’s actual position in the market and making sense of the variances across multiple sources is time consuming.
Now there is a way to see a consolidated overview of rankings and awards from different publications.
By using data analytics and industry expertise, Legal Media 360 has developed a brand sentiment index which standardizes, aggregates and weights rankings from across market leading legal publications to provide the legal industry with a simple benchmarking tool which more accurately reflects a law firm or lawyers standing in the market.
This will allow law firm management teams to track their performance accurately and easily. This also means buyers of legal services can make informed choices before instructing a law firm or selecting legal counsel by using our key information.
More information can be found on our Methodology and About Us pages.
By adding our own proprietary data, Legal Media 360 will be producing specialist reports, the first of which will be on gender diversity within law firms. You can sign up for a special preview of the report here.