Generalization

We aim to understand how language models generalize, including out-of-distribution generalization in formal reasoning and the role of memorization. Examples include Symbolic Brittleness, Limits of Transformers, Easy-to-Hard Generalization, and Llemma.

Related publications

2024

  1. Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
    Zhiqing Sun, Longhui Yu, Yikang Shen, and 4 more authors
    arXiv preprint arXiv:2403.09472, Nov 2024

2023

  1. Llemma: An Open Language Model For Mathematics
    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, and 6 more authors
    arXiv preprint arXiv:2310.06786, Nov 2023
  2. Faith and Fate: Limits of Transformers on Compositionality
    Nouha Dziri, Ximing Lu, Melanie Sclar, and 13 more authors
    In Thirty-seventh Conference on Neural Information Processing Systems, Nov 2023

2021

  1. Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics
    Sean Welleck, Peter West, Jize Cao, and 1 more author
    In AAAI Conference on Artificial Intelligence, Nov 2021