SS26: Disagreement in NLP

Seminar Description

Traditional NLP approaches resolve label disagreements into a single “gold standard,” since disagreements are treated as noise in the data, resulting from the lack of attention or mistakes of the annotators, subjective bias, or insufficient annotation guidelines.
However, recent research highlights that a single gold label may not capture the ambiguity and diversity in language. For subjective tasks such as abuse detection and quality estimation, there is an even greater need for multi-perspective modelling in order to include different viewpoints and improve the robustness and fairness of NLP models.

This seminar explores disagreement in linguistic annotation and perspectivist approaches in NLP, focusing on learning from non-aggregated datasets and multi-perspective evaluation. We will explore the causes of annotation disagreements and strategies to address them. We will discuss current research on modelling diverse viewpoints and the broader implications for AI fairness and inclusion. We will then also discuss how these disagreement-aware models can interact with users in practice, including connections to human-centered AI, adaptive interfaces, and cognitive models of interpretation and attention.