SS26: Pragmatic Processing in Large Language Models

Seminar Description

Pragmatics – the study of how language is used in context and how nonliteral meaning is conveyed – is central to human communication. Phenomena such as irony, metaphor, sarcasm, and indirect requests all fall under the umbrella of pragmatics. Given that people use language in nonliteral ways all the time, pragmatic competence is essential for large language models (LLMs) to achieve nuanced language understanding and natural interaction. While LLMs have demonstrated impressive performance across a wide range of tasks, their performance on different pragmatic phenomena, such as recognizing irony and interpreting indirect requests, is more mixed.

In this seminar, we will examine recent research on the pragmatic competence of large language models across different linguistic phenomena. We will also explore whether pragmatic abilities require specific model architectures or fine-tuning strategies, or whether they can be acquired end-to-end given appropriate data and training regimes. In addition, the seminar will survey existing benchmarks and evaluation methodologies used to assess pragmatic abilities of LLMs. At the end of the seminar, participants will do a final project in which they evaluate the performance of LLMs on a pragmatic phenomenon of their choice.