Descriptive adequacy and Large Language Models

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Generative language models produce unconstrained text with few grammatical errors. But full linguistic competence also requires the ability to associate grammatical sentences with the correct syntactic structures, resulting in the correct semantic interpretations. We performed experiments to establish whether five commercially available LLMs are able to distinguish structurally ambiguous sentences from unambiguous ones in Dutch, and to assign the correct interpretations to the unambiguous variants. In order to minimize the effects of pragmatics, an additional experiment was performed with stimuli containing pseudowords. The results are compared with results obtained from human subjects. It is found that the LLMs tested indeed show an ability to recognize structural ambiguity and to choose the correct semantic interpretations, although they perform worse than humans, and show different biases.