User commentary on news sites is intensively discussed and disputed: Harmful effects are feared by news media and the public alike, and the chasm seems to be wide between normative conceptions of ideal usages and users’ actual practices. Moderation appears to be a promising setscrew to mend this dilemma: Studies indicate, for instance, that active moderation—especially with an encouraging mindset—increases participation and decreases the number of uncivil comments (e.g., Stroud et al., 2015; Ziegele & Jost, 2016). However, this form of moderation binds resources, leading most news outlets to focus on policing and banning negative commenting rather than engaging in the encouragement of “good” contributions, although such engagements could “act as cues for a positive feedback loop with a community” (Park et al. 2016, p. 1115). In addition, many outlets have implemented profanity filters to assist such policing strategies. Interestingly, some resource-rich outlets such as the New York Times, the Washington Post, or the Austrian news outlet Der Standard have started to explore ways to also automatically detect “good” contributions through machine learning. However, these are exceptions in the news industry, and only very few academic projects head down the same road as of yet (e.g., Diakopoulos, 2015; Park et al., 2016). The aim of this presentation is to discuss existing criteria and to come up with additional indicators for both low-quality and high-quality comments to be best identified through machine learning. While the cited studies focused on implementations and fieldwork, we take a step back and derive criteria from deliberation theory and incivility research—the result being a systematically developed and theoretically grounded classification. By doing so, we bring the strength of Communication Theory to the field of Computer and Data Science. Additionally, we contribute to the stream of research on machine learning for comment moderation by focusing on German, a thus far underrepresented language in this still relatively small body of research. As such, our grid for automatically identifying low-quality and high-quality comments also aims to serve resource-poor outlets in advancing their efforts toward more sophisticated automated approaches. In the remaining time before the ECREA conference, we schedule interviews with experts in comment moderation and editorial management of user commentary to validate our category system. Implications and drawbacks as well as potential implementation scenarios are discussed.

Springer, N. & Haim, M. (11/2018). Machines in moderation: A theoretical framework for the application of machine learning in the management of user commentary. Presented at the 7th Annual ECREA Conference, Lugano. (content_copy)