Elucidating Human Language Processing with Large Language Models

Large language models (LLMs) exhibit extraordinary linguistic abilities. However, we still do not know to what extent the mechanisms that underlie these abilities are similar in humans and LLMs.

This project aims to take a step towards answering this question, by comparing the performance of humans and LLMs on sentences that necessitate complex syntactic processing (for example Garden Path sentences such as "While the man hunted the deer ran into the woods", and center-embedding sentences such as "The teacher who the student who the principal liked praised left the school").

We specifically focus on comprehension metrics, which have been neglected in past psycholinguistic research, compared to reading speed or well-formedness judgments, despite constituting the most important processing outcome of successful communication.

Correlation between average human and LLM sentence plausibility ratings, across four plausibility ratings datasets. We plot the LLM with the highest correlation (GPT-4 in all cases, except for the bottom right where GPT-3.5 is shown)