About the workshop
In-context learning refers to the ability of a neural network to learn from information presented in its context. While traditional learning in neural networks requires adjusting network weights for every new task, in-context learning operates purely by updating internal activations without needing any updates to network weights. The emergence of this ability in large language models has led to a paradigm shift in machine learning and has forced researchers to reconceptualize how they think about learning in neural networks. Looking beyond language models, we can find in-context learning in many computational models relevant to cognitive science, including those that emerge from meta-learning.
This workshop presented at CogSci 2024 aims to delineate and discuss the implications of this phenomenon for the cognitive sciences. For this, we have invited a diverse group of researchers to map out the following questions:
- How well can human learning be modeled using in-context learning?
- Which neural architectures support in-context learning?
- When and why do natural and artificial systems rely on in-context versus in-weights learning?
- How does in-context learning relate to classical concepts from cognitive science?
Workshop proposal available here.
Program
| Time (CEST) | Speaker | Title |
|---|---|---|
| TBD | Marcel Binz | Introduction to in-context learning |
| TBD | Stephanie Chan | What do you need for in-context learning in transformers? |
| TBD | Roma Patel | Towards understanding the (conceptual) structure of language models |
| TBD | Break | |
| TBD | Ishita Dasgupta | Concepts and categories within context |
| TBD | Akshay K. Jagadish | Ecologically rational meta-learned inference explains human category learning |
| TBD | Jacques Pesnot Lerousseau | Training data distribution drives in-context learning in humans and transformers |
| TBD | Break | |
| TBD | James Whittington | Different algorithms for in-context learning in prefrontal cortex and the hippocampal formation |
| TBD | Greta Tuckute | LLMs as strong learners: applications to language neuroscience |
| TBD | Thomas McCoy | Understanding and controlling neural networks through the problem they are trained to solve |
| TBD | Break | |
| TBD | Christopher Summerfield Morgan Barense Micha Heilbron Thomas L. Griffiths Brenden M. Lake |
Panel discussion |
Speakers
Stephanie Chan
Stephanie Chan is a senior research scientist at Google Deepmind. Having a background in both cognitive and computer science, she studies how data distributional properties drive emergent in-context learning.
Roma Patel
Roma Patel is a fourth-year PhD student at Brown University. Her research uses language to structure reinforcement learning, aiming towards building more intelligent and interpretable agents that can learn to use language to communicate and coordinate with each other.
James Whittington
James Whittington is a Sir Henry Wellcome postdoctoral fellow at Stanford University & the University of Oxford. He works on building models and theories for understanding structured neural representations in brains and machines.
Greta Tuckute
Greta Tuckute is a PhD candidate at the Department of Brain and Cognitive Sciences at MIT. She studies how language is processed in the biological brain, and how the representations and processes in artificial neural networks models compare to those in humans.
Thomas McCoy
Thomas McCoy is an assistant professor in the Department of Linguistics at Yale University. He studies the computational principles that underlie human language using techniques from cognitive science, machine learning, and natural language processing.
Organisers
Marcel Binz
Marcel Binz is a research scientist at Helmholtz Munich. He works on modeling human behavior using ideas from meta-learning, resource rationality, and language models.
Ishita Dasgupta
Ishita Dasgupta is a research scientist at Google Deepmind. She uses advances in machine learning to build models of human reasoning, applies cognitive science approaches toward understanding black-box AI systems, and combines these insights to build better, more human-like artificial intelligence.
Akshay K. Jagadish
Akshay K. Jagadish is a PhD student at the Max Planck Institute for Biological Cybernetics, Tübingen. His current research is dedicated towards understanding the ingredients essential for explaining human adaptive behavior across multiple task domains.
Jacques Pesnot Lerousseau
Jacques Pesnot Lerousseau is a postdoc at the Institute for Language, Communication, and the Brain, Marseille. His current research addresses the question of in-context learning in human brains and artificial neural networks, aiming to uncover the mechanisms behind rule generalization in the brain and algorithms.
Christopher Summerfield
Christopher Summerfield is a Professor of Cognitive Neuroscience at the University of Oxford. His work focusses on the neural and computational mechanisms by which humans make decisions.