Ilya Sutskever's reading list

A poster titled "learn these and you'll know 90% of what matters" lists 27 papers, posts and chapters from 1993 to 2020. Here is what each one teaches, what still holds up in 2026, and how much of the story behind the list can be checked.

Where the list comes from

Things on the poster that are not in any source: the seven themed groups (they are the poster's own), the "1990–2020" span (the oldest item is from 1993), and calling all 27 "papers" (seven are blog posts, a course, a thesis or a book chapter). The "Ilya 30u30" name some copies use is a fan label.

The 27, by idea

Grouped the way the poster groups them. Citation counts are OpenAlex's largest single record for the work on 20 October 2026; OpenAlex splits famous papers across several records, so read them as lower bounds and for order of magnitude only. Numbers in brackets are the poster's.

Information, complexity and compression

The largest group, and the one most people skip. Its thread: learning is compression. A model that explains data well is a short description of it.

1. The First Law of Complexodynamics [2], Scott Aaronson, 2011 (blog). Entropy only rises, but how "interesting" a system looks rises and then falls, like cream stirred into coffee. Aaronson asks how to measure that and sketches an answer with resource-bounded Kolmogorov complexity.
2. The Coffee Automaton [20], Aaronson, Carroll and Ouellette, 2014 (arXiv, 16 citations). The experiment behind that post: simulate the coffee cup, estimate complexity by how well a coarse-grained picture compresses, and watch it rise and fall. The least cited item on the list by far.
3. Keeping Neural Networks Simple by Minimizing the Description Length of the Weights [6], Hinton and van Camp, 1993 (COLT, 957 citations). Add noise to weights and charge for the bits needed to describe them; the result is an early form of variational learning for neural networks. The oldest item.
4. A Tutorial Introduction to the Minimum Description Length Principle [24], Peter Grünwald, 2004 (arXiv, 251 citations). The clearest statement of the principle the group rests on: prefer the model that gives the shortest total description of model plus data.
5. Kolmogorov Complexity and Algorithmic Randomness [26], Shen, Uspensky and Vereshchagin, 2017 (AMS). Circulating copies point to chapter 14 (from page 434), on algorithmic statistics: what it means for a model to be a good explanation of one particular string.
6. Machine Super Intelligence [25], Shane Legg's PhD thesis, 2008 (PDF). A formal definition of intelligence as performance across all computable environments, weighted by simplicity. Legg went on to co-found DeepMind.
7. Variational Lossy Autoencoder [18], Chen, Kingma, Salimans and others, 2016 (arXiv, 439 citations). Pairs a VAE with an autoregressive decoder so you can choose what the latent code keeps (global structure) and what it throws away (local texture). Sutskever is a co-author.

Convolutional vision

1. ImageNet Classification with Deep Convolutional Neural Networks (AlexNet) [8], Krizhevsky, Sutskever and Hinton, 2012 (112,073 citations, Communications of the ACM reprint). A large CNN trained on two GPUs with ReLUs and dropout won ImageNet 2012 by a wide margin and started the deep learning decade.
2. Deep Residual Learning for Image Recognition (ResNet) [11], He, Zhang, Ren and Sun, 2015 (arXiv, 234,403 citations, the most cited item). Let each block learn a change to its input instead of a replacement, and 150-layer networks train. Every transformer's residual stream descends from this.
3. Identity Mappings in Deep Residual Networks [16], same authors, 2016 (arXiv, 10,648 citations). Keep the skip path a clean identity and move normalization before the layer; it trains over 1,000 layers. Pre-norm transformers use the same idea.
4. Multi-Scale Context Aggregation by Dilated Convolutions [12], Yu and Koltun, 2015 (arXiv, 7,736 citations). Space out a convolution's taps to see wider context without shrinking the image, for dense prediction such as segmentation.
5. CS231n: Convolutional Neural Networks for Visual Recognition [27], Stanford, Li, Karpathy and Johnson (course notes). The standard from-scratch course: backpropagation, optimization and CNNs, with assignments that make you write them.

Recurrent networks

1. The Unreasonable Effectiveness of Recurrent Neural Networks [3], Andrej Karpathy, 2015 (blog). A character-level model trained only to predict the next character writes passable Shakespeare, LaTeX and C. The intuition behind GPT, years early.
2. Understanding LSTM Networks [4], Christopher Olah, 2015 (blog). The diagrams nearly everyone learned LSTM gates from.
3. Recurrent Neural Network Regularization [5], Zaremba, Sutskever and Vinyals, 2014 (arXiv, 2,259 citations). Apply dropout only between layers, never on the recurrent connection, and large LSTMs stop overfitting.

Memory and pointers

1. Neural Turing Machines [21], Graves, Wayne and Danihelka, 2014 (arXiv, 1,057 citations). A network with a differentiable external memory, read and written by attention, learns to copy and sort.
2. Pointer Networks [7], Vinyals, Fortunato and Jaitly, 2015 (arXiv, 2,025 citations). Use attention to point at input positions as outputs, so the output vocabulary grows with the input: convex hulls, travelling salesman tours.
3. Order Matters: Sequence to Sequence for Sets [9], Vinyals, Bengio and Kudlur, 2015 (arXiv, 696 citations). The order you feed or emit a set in changes what a sequence model learns; proposes an attention-based read, process, write model for sets.

Attention and transformers

1. Neural Machine Translation by Jointly Learning to Align and Translate [15], Bahdanau, Cho and Bengio, 2014 (arXiv, 16,219 citations). Attention is born: the decoder looks back over every source word instead of squeezing the sentence into one vector.
2. Attention Is All You Need [14], Vaswani and others, 2017 (arXiv, 78,007 citations on its largest record). Drop recurrence, keep attention. The architecture under nearly every model in use today.
3. The Annotated Transformer [1], Sasha Rush, 2018, updated by Austin Huang and others in 2022 (Harvard NLP). The paper above as runnable PyTorch, line by line. Read the two together.

Relational reasoning

1. A Simple Neural Network Module for Relational Reasoning [17], Santoro and others, 2017 (arXiv, 1,282 citations). Apply one small network to every pair of objects and sum; that alone beat human accuracy on the CLEVR visual question benchmark.
2. Relational Recurrent Neural Networks [19], Santoro and others, 2018 (arXiv, 153 citations). A recurrent memory whose slots attend to each other with multi-head attention.
3. Neural Message Passing for Quantum Chemistry [13], Gilmer and others, 2017 (arXiv, 5,064 citations). One framework for graph neural networks, nodes passing messages along edges, used to predict molecular properties.

Scale

1. GPipe [10], Huang and others, 2018 (arXiv, 1,129 citations). Split a model across accelerators and feed it micro-batches in a pipeline, so models too big for one chip still train efficiently.
2. Deep Speech 2 [22], Amodei and others at Baidu, 2015 (arXiv, 2,110 citations). One end-to-end network for English and Mandarin speech, and as much a paper about the engineering of training at scale as about the model.
3. Scaling Laws for Neural Language Models [23], Kaplan and others at OpenAI, 2020 (arXiv, 1,743 citations). Loss falls as a smooth power law in parameters, data and compute, so you can predict a big model from small ones. The newest item.

Reading it in 2026

This section is our judgment, not something the sources measure.

• Still central: residual connections, attention and the transformer, and scaling laws. These are in the models people use every day.
• Underrated: the compression group. It explains why next-token prediction learns so much, which is the question that matters most now, and it is the part a quick reader skips.
• History, mostly: the LSTM and memory-network items read as the road to attention. One honest caveat: recurrence came back in 2023–2025 as state-space and linear-attention models (Mamba, RWKV), so "historical" is too strong for the ideas, if not the specific papers.
• Superseded in detail: Kaplan's advice on how to split a compute budget between model size and data was revised by DeepMind's Chinchilla work in 2022. Read the 2020 paper for the method, not the ratios.
• Missing, because it is a 2020 list: instruction tuning and RLHF, in-context learning at GPT-3 scale, diffusion models, mixture of experts, and reinforcement learning for reasoning. And whatever the lost meta-learning section held.

An order that works: CS231n for the basics, then Karpathy and Olah, Bahdanau, the Transformer with its annotated version, ResNet and its follow-up, Kaplan; then the compression group, starting with Aaronson's post and Grünwald's tutorial. The rest as interest takes you.

How we checked

We used our own research toolkit, apps/intel, with honest sources only: official APIs, published feeds and pages whose robots.txt allows reading. No data from our trellis mirror was used, and Reddit and X were not read; where an X post matters above, it is quoted through a page that quotes it, and labelled so.

• Items: 525 OpenAlex records searched by title, matched to the 20 scholarly items (all 20 found, citation counts above), arXiv's API for preprint ids and dates, and the 7 blogs, course and thesis fetched from their own sites.
• Provenance: 268 Hacker News and web search records plus 13 pages fetched in full. bin/grade put every one at D or E on the Admiralty scale: nothing primary, and nothing from Sutskever or OpenAI. The only first-hand record is Carmack's interview.
• Corroboration: bin/trace found 165 story clusters, 13 with more than one source. We discount that figure: the tool currently counts separate commenters on one thread as separate sources, and almost every page traces back to the same 2024 post anyway. Many copies of one claim are not corroboration.
• Ranking: bin/rank found the right pages but cannot tell first from loudest; the timeline above is a reading of dates, by hand.
• Challenge: bin/challenge ran on both main conclusions with a small local model (qwen3:4b), since hosted credit is used up. It invented statistics and a Sutskever blog post, so most of its output was discarded. Two points survived and are on this page: that the 27 may not come from Carmack's list at all, only from a list someone else called a partial version; and that recurrence is not as dead as "historical" implies.

What we did not check: Carr's original post and images, Carmack's posts on X directly, and the arc.net folder. If Sutskever ever publishes the list, this page should be corrected against it.