Investors and firms using ESG data are faced with conflicting metrics and widespread confusion about what’s being measured, and must rely on their own groundwork to make that information useful, according to one of the leading scholars in sustainability reporting.
Without standardization, user involvement, and a better understanding of methodologies and scopes, ESG scores and ratings will fail to both enhance responsibility and deliver financial returns, Massachusetts Institute of Technology (MIT) Professor Roberto Rigobon said at the recent Qontigo Investment Intelligence Summit.
“We do have to measure our impact better, and this is true for everyone,” the professor of applied economics at the MIT Sloan School of Management told the audience in New York on May 19. “For that, we need to measure better. Measurement is a first-order issue.”
Aggregate confusion
Prof. Rigobon is one of the researchers behind the Aggregate Confusion Project, which is trying to reduce ESG data noise, develop more rigorous methods for ESG integration and produce smarter ways to aggregate ESG factors into composite indices. In a paper1 published in 2019 and updated last April, the Project’s team found that the average correlation between ratings on 924 companies from six rating agencies was 0.54. In comparison, major credit ratings from Moody’s and S&P Ratings have a correlation of near 1, meaning they almost always coincide.
Rather than showing a uniform picture, “the data looks like a confetti,” Prof. Rigobon, who is also the Society of Sloan Fellows Professor of Management, said in a presentation titled ‘The ESG Zoo and What to Do.’ “You will have a lot of companies that are rated very highly by one agency and very low by another. That is very common.”
Qontigo is one of five founding members of the Aggregate Confusion Project.
A large body of data
ESG rating providers have faced criticism because their methodologies and verdicts often clash at a time when they can determine the fate of multi-billion-dollar investment flows. Investments in ESG strategies reached USD 35.3 trillion at the end of 2019, according to the most recent data from the Global Sustainable Investment Alliance (GSIA).
The vast discrepancies in ESG ratings stem from the large and intricate range of activities that agencies analyze, monitor and grade. Each rating firm will have its distinct methodology in terms of what should be measured — as well as how and where.
In their April 2022 study, Prof. Rigobon and colleagues from MIT and the University of Zurich identified 163 ESG categories scrutinized by six major rating providers – from Employee Development to Green Products and Business Ethics. Only a few categories are covered by all six raters.
The authors have estimated that measurement discrepancies (for example, how is employee satisfaction measured?) explain over 50% of the divergence between ratings. Scope (the choice of categories that are considered ESG) accounts for about 40% of the difference, while dissimilarities in weight (the relevance that is given to each ESG category) explain up to 10%.
Different views, dissimilar outcomes
It is very common that two indicators seemingly measuring the same thing can look very different. Prof. Rigobon gave one example within the category of Employee Satisfaction. American women tend to walk away from companies when feeling unappreciated, but rarely pursue legal action. Dissatisfied female workers in France, on the other hand, stick with their employers but tend to file lawsuits. A rating provider that considers turnover to measure a company’s employee satisfaction is likely to get very different outcomes than one that tracks the same company’s court complaints.
Unlike credit scores, ESG ratings are plagued by infrequent measurements, insufficient statistics, a tendency to measure extreme events, and perception-driven biases, Prof. Rigobon explained. What is measured will also dictate the final score: should one look at a company’s actions, their outcomes, its procedures, or its statements? While all may be valid, the results may be disparate at best and contradictory at worst.
The plus side of diversity
To be sure, while Prof. Rigobon is critical of the ESG ratings maze, he acknowledges the key role that rating agencies play (ratings providers actively cooperate with the Aggregate Confusion Project). Moreover, a variety of opinions can be a good thing as it gives a heterogeneous audience the option to select the methodology closest to their preferences. Measurement divergence becomes problematic when clients fail to grasp that ESG rating is a subjective discipline.
“There is no solution without ESG rating agencies,” the author said at the Qontigo summit. “These are very complicated issues to measure. We are trying to measure ourselves.”
Recommendations
So, what is an investor or firm to do with this ‘ESG zoo’? Prof. Rigobon, who is also a research associate at the National Bureau of Economic Research, gave the audience three broad recommendations.
Firstly, ESG data users should be guided by their individual goals and priority issues. These are likely to be different from other firms. With that in mind, they should try to work with granular data that covers and is aligned with those objectives.
“Go back a bit,” Rigobon said. “Reset. Request the raw data from different rating agencies. Understand what is being computed so you can improve the measurements.”
The second recommendation is not to aggregate data, as this can lead to counterproductive and misleading trade-offs (i.e., ‘do I prioritize water management or labor practices?’). Each ESG category should be treated independently from others, as much as possible.
Finally, Prof. Rigobon recommended that the audience never stop challenging their ESG goals and achievements.
“Do not be satisfied; move the goal posts consistently,” he said. “When you think that you’ve reached success, re-evaluate. Because you may actually be missing the point. Do not develop a false sense of accomplishment.”
Further reading
For more coverage of panels and presentations at the Qontigo Investment Intelligence Summit, please visit the following:
Ever more sophisticated data fueling rise in thematic index strategies, panel tells Qontigo summit
BlackRock’s Ang on sustainability alpha signals: ESG and factors as ‘best friends’
1 Berg, F., Koelbel, J. and Rigobon, R., ‘Aggregate Confusion: The Divergence of ESG Ratings,’ August 15, 2019.
