To be able to handle dynamic, changing conditions, we want to move from deep statistical models which are able to perform system 1 tasks to deep structural models also able to perform system 2 tasks by taking advantage of the computational workhorse of system 1 abilities.

Today's deep networks may benefit from additional structure and inductive biases to do substantially better on system 2 tasks, natural language understanding, out-of-distribution systematic generalization and efficient transfer learning. We have tried to clarify what some of these inductive biases may be, but much work needs to be done to improve that understanding and find appropriate ways to incorporate these priors in neural architectures and training frameworks.

We have motivated these inductive biases in terms of expected (and observed in recent work) gains in terms of out-of-distribution generalization and fast adaptation in transfer settings rather than the standard test set from the same distribution as the training set.

The general insight here is that the proposed inductive biases should help organize knowledge into the stable reusable parts that are likely to be useful in new settings and tasks (such as causal mechanisms), separating them from the more volatile pieces of information (the values of variables) that can be changed by agents (through causal intervention) or those that are affected by these changes and may vary across environments or tasks.

Some of the more salient inductive biases we propose deserve especially more attention in deep learning research include (a) the fairly direct connection between high-level variables and natural language or more generally how humans communicate knowledge among them, i.e., we can verbalize our thoughts to a large extent and this can provide rich insights about underlying inductive biases such as these: (b) the modular decomposition of knowledge into independent reusable pieces that can be composed on the fly to address new contexts, (c) the causal interpretation of actions by agents and of changes in distribution, with agents generally intending to affect a single or very few (generally latent) variables, and (d) the sparsity of dependencies between high-level variables (and thus the small number of variables that are linked by causal mechanisms imagined by humans to explain their environment).

Finally, we would also like to mention that inductive biases are not the only way to bridge the gap to high-level human cognition: we may gain by improving our optimization algorithms, by scaling up neural networks and by moving to other frameworks that better capture uncertainty about the world (e.g., by learning a Bayesian posterior over neural network models as compared to learning a point estimates).

It would also be intriguing to think of ways to combine all these different elements together.

The authors are grateful to Alex Lamb, Rosemary Nan Ke and Olexa Bilaniuk for leading many projects some of which are also discussed here. The authors are grateful to Mike Mozer and Bernhard Schölkopf for many brainstorming discussions. The authors would also like to thank Stefan Bauer, Aniket Didolkar, Nasim Rahaman, Kanika Madan, Philippe Beaudoin, Charles Blundell, Dianbo Liu, Moksh Jain for useful feedback. The authors would also like to acknowledge comments by the reviewers which helped in improving the manuscript.