The past decade has witnessed a surge in the development and adoption of machine learning algorithms to solve day-a-day computational tasks. Yet, a solid theoretical understanding of even the most basic tools used in practice is still lacking. Surprisingly, many of the “exotic” behaviours of deep neural networks have shown to hold in models as simple as high-dimensional linear regression. In this talk, I will introduce two simple toy models describing learning with correlated features. Next, I will motivate how these models encompass many learning problems of interest, e.g. ridge and logistic regression, kernel methods, random features, scattering transforms and transfer learning. Finally, I will discuss how these models can be used to approximate real data learning curves in some scenarios.