World Association of Newspapers and News Publishers


New Tow Center study: the role of metrics in journalism

World News Publishing Focus

World News Publishing Focus
Your Guide to the Changing Media Landscape

New Tow Center study: the role of metrics in journalism

A new report, “The Traffic Factories”, launched on May 7th by the Tow Center for Digital Journalism looks at how analytics affect newsroom culture and the journalists’ daily work. Research fellow Caitlin Petre conducted an ethnographic study examining three case studies: the analytics company Chartbeat, Gawker Media, where metrics are highly available to the editorial staff, and the New York Times, where the use of metrics is more marginal. She found that metrics “exert a powerful influence over journalists’ emotions and morale”, inspiring a range of strong feelings, which are too often overlooked. Newsroom managers should consider the potential effects of traffic data not only on editorial content but also on editorial workers, the study says.

The field of news analytics companies that produce dashboards has become more and more crowded and these tools have become ubiquitous in newsrooms, Petre emphasised when presenting the report (the presentation was live-streamed). But while there is a lot of debate about what are the right metrics and what newsrooms should or should not be doing with them, there hasn’t been a lot of rigorous research on what newsrooms are actually doing with data in the daily work, she continued. “I wanted these case studies to be representative of a broader dynamics.”

Here are some of the main findings:

At Chartbeat
“Chartbeat has an emotional understanding of the powerful feelings of analytics”, said Petre. She quoted a Charbeat employee who said: “It’s not the identity of the number, it’s the feeling that the number produces. That’s the thing that’s important about it”.
Petre underlined that the success of Chartbeat - which works with 80 percent of the highest-trafficked publishers in the US and newsrooms in 35 other countries - is partly due to communicating deference to journalistic judgement and the fact that they don’t actively tell the editors where to place articles in the page or how to react to the data.
As the study underlines, one way to communicate deference and build trust is in the choice of the metrics themselves: a metric like engaged time, which is one of the main metrics Chartbeat focuses on, is designed to reward high-quality content, “meaning the kinds of rigorous, thoughtful reporting and writing that are central to journalists’ professional identity”.
When we talk about analytics we need to take into account the emotional power of these tools, said Petre.
“While we tend to think of analytics dashboards as a rationalizing force in newsrooms […] it turns out that Chartbeat’s dashboard is designed to play a social and emotional role in newsrooms just as much a rational one”, the study reads.

At Gawker
According to the study metrics at Gawker Media are:
- prominent, a big screen constantly updated, populated by Chartbeat data, reigns in the  newsroom
- powerful, over the year editors and writers have been rewarded with a range of traffic-based pay-for-performance schemes
- public, many of the company’s metrics are publicly available online.

Given the company’s metric-driven approach Petre’s goal conducting the research was to analyze how this affected employees and the content they produce. The study found that “traffic data was a central feature of the organizational culture” and a one saturated with emotions for employees who are constantly bombarded with data. Metrics affects the shape of competition, fueling internal competition rather than external.

Metrics don't just reflect reality but also can shape it, Petre said. Interestingly, despite the anxiety of the “metric surveillance”, many Gawker’s journalists effectively prefer it to traditional editorial oversight, Petre said.

At the New York Times
Contrary to Gawker, reporters at the New York Times didn't have much access to data at the time the study was conducted. "In the past years, representatives of The Times have publicly taken a dismissive posture towards metrics", the study says, defining the use of metrics as "restricted, discretionary and rare".
Editors had lots of discretion on how to use data and they used it to back up or corroborate decisions they already made, Petre said.

"Metrics provided an alternative yardstick - aside from editors' evaluations - by which reporters could judge the worthiness of their stories and their job performances more broadly. The data, therefore, threatened to undermine not only news judgment, but also the traditional hierarchical structure of the Times newsroom, in which editors were the final arbiters of the nebulous quality that is 'newsworthiness'. If editors alone had access to metrics, they alone could control the way in which the data was interpreted and mobilized", the study says.

A few conclusions and more questions to debate
As different as Gawker and the Times have been, there is evidence they might be moving in the same direction - Petre said - converging to a more similar approach as Gawker is trying to de-emphasize metrics and the New York Times is trying to give them more prominence.

While one of the main recommendations of the Times' Innovation report was, in fact, to urge a greater role of data in editorial decision-making, Gawker at very beginning of year dodged the bonus/traffic system and despite still having the big board in the newsroom, it is not as powerful as it was used to be.

Petre's report leaves some questions open for debate and future research: existing studies have focused on the effects of analytics on journalists but what about news consumers' relationship to metrics? Are audiences aware that their behavior on news sites is being tracked to the extent that it is", the study asks? "Are metrics actually shaping content, and if so, how?" and lastly, "what is the impact of impact metrics?"

You can read the full report here.

Image credit: Flickr, some rights reserved.

Author

Federica Cherubini's picture

Federica Cherubini

Date

2015-05-09 19:24

Author information

The World Editors Forum is the organisation within the World Association of Newspapers devoted to newspaper editors worldwide. The Editors Weblog (www.editorsweblog.org), launched in January 2004, is a WEF initiative designed to facilitate the diffusion of information relevant to newspapers and their editors.


© 2019 WAN-IFRA - World Association of News Publishers

Footer Navigation