Browse All Papers¶

A visual grid of every one of the 156 summaries, grouped by category and ordered by paper number, with an editorial relevance badge and a three-line pitch each. Use it to scan the whole collection quickly; for the generated, always-current listings see the Paper Index or browse by topic tag. Looking for a learning path? See docs/ROADMAP.md or the README for numbered reading orders. Wondering what is not here yet? See Coverage & Gaps.

The badges and pitches on this page are editorial judgements, not claims from the papers.

🏗️ Foundational Architectures¶

The backbone papers - what the models are actually built out of, from the 2014 attention mechanism through the Transformer, ViT, sparse MoE and the state-space challengers. 14 papers.

[Attention Is All You Need](./papers/architectures/01-attention-is-all-you-need/summary.md) (2017) - 🔥 CRITICAL - Foundation of everything - Introduced Transformer architecture - Self-attention mechanism - [Paper](https://arxiv.org/abs/1706.03762)	[Vision Transformer (ViT)](./papers/architectures/11-vision-transformer/summary.md) (2020) - ⭐ HIGH - Transformers for computer vision - Images as patch sequences - Enables multimodal models - [Paper](https://arxiv.org/abs/2010.11929)	[Mamba](./papers/architectures/20-mamba/summary.md) (2023) - 🔥 CRITICAL - First viable Transformer alternative - Linear-time sequence modeling (O(n) vs O(n²)) - Selective state spaces - [Paper](https://arxiv.org/abs/2312.00752)
[Mixtral & Mixture-of-Experts](./papers/architectures/37-mixture-of-experts/summary.md) (2024) - 🔥 CRITICAL - 47B parameters, 13B active - Matches LLaMA 2 70B at 6x faster inference - The architecture DeepSeek-V3, Qwen3 and Llama 4 adopted - [Paper](https://arxiv.org/abs/2401.04088)	[Seq2Seq](./papers/architectures/55-seq2seq/summary.md) (2014) - 📚 HISTORICAL - Variable-length in, variable-length out - Stacked LSTM encoder-decoder for translation - Its fixed-size bottleneck is what attention fixed - [Paper](https://arxiv.org/abs/1409.3215)	[Bahdanau Attention](./papers/architectures/66-bahdanau-attention/summary.md) (2014) - 🔥 CRITICAL - Where attention was invented - Soft alignment over every encoder state - The conceptual seed of self-attention - [Paper](https://arxiv.org/abs/1409.0473)
[Switch Transformer](./papers/architectures/67-switch-transformer/summary.md) (2021) - ⭐ HIGH - Trillion parameters at near-constant compute - Top-1 routing: one expert per token - The sparsity recipe Mixtral productionized - [Paper](https://arxiv.org/abs/2101.03961)	[ResNet](./papers/architectures/73-resnet/summary.md) (2015) - 🔥 CRITICAL - Residual connections - `x + f(x)` is in every Transformer - Most-cited deep learning paper - [Paper](https://arxiv.org/abs/1512.03385)	[U-Net](./papers/architectures/74-unet/summary.md) (2015) - ⭐ HIGH - Encoder-decoder + skips - The denoiser inside diffusion models - Trained on 30 images, won its challenge - [Paper](https://arxiv.org/abs/1505.04597)
[Grouped-Query Attention](./papers/architectures/75-grouped-query-attention/summary.md) (2023) - 🔥 CRITICAL - 8x smaller KV cache - In Llama 2/3/4, Mistral, Qwen, Gemma - Why long context is affordable - [Paper](https://arxiv.org/abs/2305.13245)	[Masked Autoencoders (MAE)](./papers/architectures/88-mae/summary.md) (2021) - ⭐ HIGH - BERT-style pretraining for vision - Hide 75% of patches, reconstruct the rest - Asymmetric encoder-decoder, cheap to scale - [Paper](https://arxiv.org/abs/2111.06377)

[Longformer](./papers/architectures/131-longformer/summary.md) (2020) - ⭐ HIGH - Linear-cost attention for long documents - Sliding window plus global tokens - Continued pretraining from RoBERTa - [Paper](https://arxiv.org/abs/2004.05150)	[Multi-head Latent Attention (DeepSeek-V2)](./papers/architectures/141-multi-head-latent-attention/summary.md) (2024) - 🔥 CRITICAL - GQA's successor - Cache a small latent, not keys and values - 93% smaller KV cache, better quality - [Paper](https://arxiv.org/abs/2405.04434)
[Mamba-3](./papers/architectures/156-mamba-3/summary.md) (2026) - ⭐ HIGH - The state-space line, three generations in - Improved sequence modelling from state space principles - From the Mamba authors, ICLR 2026 - [Paper](https://arxiv.org/abs/2603.15569)

🤖 Language Models¶

The model releases themselves, in lineage order: GPT-1 through GPT-5, the BERT and T5 branch, the Llama and Mistral open-weight line, and the frontier reasoning models. 27 papers.

[BERT](./papers/language-models/03-bert/summary.md) (2018) - 📚 HISTORICAL - Pre-training revolution - Bidirectional pre-training - Masked language modeling - [Paper](https://arxiv.org/abs/1810.04805)	[GPT-3](./papers/language-models/04-gpt3-few-shot-learners/summary.md) (2020) - ⭐ HIGH - Few-shot learning paradigm - 175B parameters - Foundation for ChatGPT - [Paper](https://arxiv.org/abs/2005.14165)	[InstructGPT (RLHF)](./papers/language-models/05-instructgpt-rlhf/summary.md) (2022) - 🔥 CRITICAL - Human preference learning - Enabled ChatGPT - RLHF methodology - [Paper](https://arxiv.org/abs/2203.02155)
[Constitutional AI](./papers/language-models/14-constitutional-ai/summary.md) (2022) - ⭐ HIGH - Alternative to RLHF - AI self-critique via principles - Powers Claude - [Paper](https://arxiv.org/abs/2212.08073)	[LLaMA](./papers/language-models/15-llama/summary.md) (2023) - 🔥 CRITICAL - Compute-optimal training - 13B matches GPT-3 175B - Open weights - [Paper](https://arxiv.org/abs/2302.13971)	[LLaMA 2](./papers/language-models/17-llama2/summary.md) (2023) - 🔥 CRITICAL - Production-ready open model - Commercial license - RLHF alignment - [Paper](https://arxiv.org/abs/2307.09288)
[DPO](./papers/language-models/19-dpo/summary.md) (2023) - 🔥 CRITICAL - Simpler than RLHF - Direct preference optimization - No reward model needed - [Paper](https://arxiv.org/abs/2305.18290)	[DeepSeek-R1](./papers/language-models/26-deepseek-r1/summary.md) (2025) - 🔥 CRITICAL - Pure RL reasoning - Matches OpenAI o1 - Fully open source - [Paper](https://arxiv.org/abs/2501.12948)	[DeepSeek-V3](./papers/language-models/27-deepseek-v3/summary.md) (2024) - 🔥 CRITICAL - $5.76M training cost - 671B MoE architecture - Matches GPT-4 efficiency - [Paper](https://arxiv.org/abs/2412.19437)
[Qwen3](./papers/language-models/28-qwen3/summary.md) (2025) - 🔥 CRITICAL - Unified thinking/non-thinking - Adaptive reasoning modes - Best of both worlds - [Paper](https://arxiv.org/abs/2505.09388)	[Claude 3.5 Sonnet](./papers/language-models/30-claude-3.5-sonnet/summary.md) (2024) - 🔥 CRITICAL - Computer use capability - Best coding model (49% SWE-Bench) - AI controls computers - [Announcement](https://www.anthropic.com/news/3-5-models-and-computer-use) - [Paper](https://www.anthropic.com/news/3-5-models-and-computer-use)	[OpenAI o1](./papers/language-models/31-openai-o1/summary.md) (2024) - 🔥 CRITICAL - Started reasoning model era - PhD-level performance - RL for reasoning - [Announcement](https://openai.com/index/learning-to-reason-with-llms/) - [Paper](https://openai.com/index/learning-to-reason-with-llms/)
[LLaMA 3.3](./papers/language-models/33-llama3.3/summary.md) (2024) - ⭐ HIGH - Distillation breakthrough - 70B matches 405B performance - Knowledge transfer success - [Paper](https://www.meta.ai/blog/meta-llama-3-3/)	[GPT-4](./papers/language-models/36-gpt4/summary.md) (2023) - 🔥 CRITICAL - The capability jump over GPT-3.5 - Bar exam top 10%; first multimodal GPT - Loss predicted in advance from smaller runs - [Paper](https://arxiv.org/abs/2303.08774)	[GPT-4o](./papers/language-models/40-gpt4o/summary.md) (2024) - ⭐ HIGH - Text, audio and vision in one model - 232ms average voice response - Half the price and 2x the speed of GPT-4 Turbo - [Paper](https://cdn.openai.com/gpt-4o-system-card.pdf)
[Llama 4](./papers/language-models/41-llama4/summary.md) (2025) - ⭐ HIGH - Meta's first MoE models - 10M-token context window on Scout - Natively multimodal, open weights - [Paper](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)	[GPT-5](./papers/language-models/42-gpt5/summary.md) (2025) - 🔥 CRITICAL - One model routing fast vs deep reasoning - 94.6% AIME 2025, 74.9% SWE-bench Verified - Ended the GPT-4 / o1 / o3 product split - [Paper](https://cdn.openai.com/gpt-5-system-card.pdf)	[Claude 4 Family](./papers/language-models/43-claude4/summary.md) (2025) - 🔥 CRITICAL - State of the art at agentic work - 80.9% SWE-bench Verified (Opus 4.5) - Extended thinking with controllable depth - [Paper](https://www.anthropic.com/news/claude-4)
[Codex](./papers/language-models/56-codex/summary.md) (2021) - ⭐ HIGH - The model behind GitHub Copilot - Introduced HumanEval and the pass@k metric - Ancestor of every code model since - [Paper](https://arxiv.org/abs/2107.03374)	[GPT-2](./papers/language-models/64-gpt2/summary.md) (2019) - 📚 HISTORICAL - GPT-1 scaled to 1.5B parameters - Zero-shot transfer with no fine-tuning - The staged-release safety debate started here - [Paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)	[T5](./papers/language-models/65-t5/summary.md) (2019) - ⭐ HIGH - Every NLP task as text-in, text-out - Systematic ablation of the whole pretraining recipe - Gave the field the C4 corpus - [Paper](https://arxiv.org/abs/1910.10683)
[GPT-1](./papers/language-models/93-gpt1/summary.md) (2018) - 📚 HISTORICAL - Where the GPT recipe begins - Generative pretraining, then task fine-tuning - 117M parameters, decoder-only Transformer - [Paper](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)	[PaLM](./papers/language-models/94-palm/summary.md) (2022) - ⭐ HIGH - 540B, largest dense Transformer of its day - Trained across two TPU pods via Pathways - Where chain-of-thought results broke out - [Paper](https://arxiv.org/abs/2204.02311)	[Mistral 7B](./papers/language-models/95-mistral-7b/summary.md) (2023) - 🔥 CRITICAL - Made open weights genuinely competitive - Beat LLaMA 2 13B on every benchmark tested - Apache 2.0, with GQA and sliding-window attention - [Paper](https://arxiv.org/abs/2310.06825)
[Llama Guard](./papers/language-models/96-llama-guard/summary.md) (2023) - ⭐ HIGH - Safety classification as an LLM task - Moderates both the prompt and the response - The open-source production reference - [Paper](https://arxiv.org/abs/2312.06674)

[phi-1 (Textbooks Are All You Need)](./papers/language-models/135-phi-1-textbooks/summary.md) (2023) - ⭐ HIGH - Data quality over quantity - 1.3B model, 50.6% HumanEval - Filtered and synthetic textbook data - [Paper](https://arxiv.org/abs/2306.11644)
[Tulu 3](./papers/language-models/148-tulu-3/summary.md) (2024) - ⭐ HIGH - The open post-training recipe - Fully open SFT, then DPO, then RLVR on Llama 3.1 - Names and popularises RLVR - [Paper](https://arxiv.org/abs/2411.15124)

🎨 Image, Video & 3D Generation¶

From VAEs and GANs through the diffusion pipeline that replaced them, the tokenizer line that made images modellable, the video models built on top, and neural 3D scene representation. 18 papers.

[GANs](./papers/image-generation/02-generative-adversarial-networks/summary.md) (2014) - 📚 HISTORICAL - Generative modeling origins - Adversarial training - Generator vs discriminator - [Paper](https://arxiv.org/abs/1406.2661)	[Diffusion Models (DDPM)](./papers/image-generation/06-diffusion-models/summary.md) (2020) - 📖 THEORY - Diffusion foundations - Iterative denoising - Better than GANs - [Paper](https://arxiv.org/abs/2006.11239)	[Stable Diffusion](./papers/image-generation/07-stable-diffusion/summary.md) (2022) - ⭐ HIGH - Practical implementation - Latent space diffusion (10-100× faster) - Open-source, democratized AI art - [Paper](https://arxiv.org/abs/2112.10752)
[Sora & Diffusion Transformers](./papers/image-generation/44-sora-dit/summary.md) (2022-2024) - 🔥 CRITICAL - Diffusion on a Transformer backbone - Video as spacetime patches, any resolution - Launched the AI video generation industry - [Paper](https://arxiv.org/abs/2212.09748)	[DALL-E 3](./papers/image-generation/48-dalle3/summary.md) (2023) - ⭐ HIGH - Follows the prompt you actually wrote - Recaptioned training data was the whole trick - First model to render readable text - [Paper](https://cdn.openai.com/papers/dall-e-3.pdf)	[Variational Autoencoder (VAE)](./papers/image-generation/57-vae/summary.md) (2013) - 🔥 CRITICAL - The reparameterization trick - A smooth, interpolatable latent space - The latent half of latent diffusion - [Paper](https://arxiv.org/abs/1312.6114)
[Classifier-Free Guidance](./papers/image-generation/69-classifier-free-guidance/summary.md) (2021) - 🔥 CRITICAL - The "CFG scale" slider - Two predictions, one difference vector - Also how negative prompts work - [Paper](https://arxiv.org/abs/2207.12598)	[DDIM](./papers/image-generation/70-ddim/summary.md) (2020) - 🔥 CRITICAL - 1,000 steps down to 20 - Deterministic sampling, no retraining - Basis of diffusion image editing - [Paper](https://arxiv.org/abs/2010.02502)	[ControlNet](./papers/image-generation/71-controlnet/summary.md) (2023) - 🔥 CRITICAL - Edges, depth, pose control - Zero convolutions, base model frozen - ICCV 2023 best paper - [Paper](https://arxiv.org/abs/2302.05543)
[Flow Matching / SD3](./papers/image-generation/72-flow-matching-sd3/summary.md) (2022-2024) - 🔥 CRITICAL - What replaced diffusion - Straight noise-to-image paths - Powers SD3, SD3.5 and Flux - [Paper](https://arxiv.org/abs/2403.03206)	[VQ-VAE](./papers/image-generation/89-vq-vae/summary.md) (2017) - ⭐ HIGH - Discrete codebook latents - Straight-through estimator for the argmin - Made images tokenizable like text - [Paper](https://arxiv.org/abs/1711.00937)	[VQ-GAN](./papers/image-generation/90-vq-gan/summary.md) (2020) - ⭐ HIGH - VQ-VAE latents plus an adversarial loss - Transformers autoregress over image tokens - The compress-then-model recipe - [Paper](https://arxiv.org/abs/2012.09841)
[Imagen](./papers/image-generation/91-imagen/summary.md) (2022) - ⭐ HIGH - A frozen T5 encoder beat bigger image models - Text understanding was the real bottleneck - Cascaded super-resolution diffusion - [Paper](https://arxiv.org/abs/2205.11487)	[DreamBooth](./papers/image-generation/92-dreambooth/summary.md) (2022) - 🔥 CRITICAL - Your subject, any scene, from 3-5 images - Rare-token binding plus prior-preservation loss - The blueprint every personalization method copies - [Paper](https://arxiv.org/abs/2208.12242)

[NeRF](./papers/image-generation/117-nerf/summary.md) (2020) - 🔥 CRITICAL - A 3D scene stored in a neural network - Photoreal novel views from photos - Volume rendering through an MLP - [Paper](https://arxiv.org/abs/2003.08934)	[3D Gaussian Splatting](./papers/image-generation/118-3d-gaussian-splatting/summary.md) (2023) - 🔥 CRITICAL - Real-time photoreal 3D - Millions of 3D Gaussians, rasterised fast - NeRF quality at real-time speed - [Paper](https://arxiv.org/abs/2308.04079)	[DALL-E 2 (unCLIP)](./papers/image-generation/119-dalle2-unclip/summary.md) (2022) - ⭐ HIGH - CLIP latents to images - Prior maps text to CLIP image embedding - Diffusion decoder paints the image - [Paper](https://arxiv.org/abs/2204.06125)
[Consistency Models](./papers/image-generation/120-consistency-models/summary.md) (2023) - ⭐ HIGH - Diffusion quality in one step - Map any noisy point straight to the output - Distilled or trained from scratch - [Paper](https://arxiv.org/abs/2303.01469)

🔗 Multimodal & Audio¶

Models that cross modalities - vision-language, speech recognition and generation, video, and the omni-models that fold them into a single network. 9 papers.

[CLIP](./papers/multimodal/08-clip/summary.md) (2021) - ⭐ HIGH - Vision-language bridge - Vision-language contrastive learning - Zero-shot image classification - Powers text-to-image models - [Paper](https://arxiv.org/abs/2103.00020)	[GPT-4V(ision)](./papers/multimodal/23-gpt4v/summary.md) (2023) - 🔥 CRITICAL - Multimodal frontier model - GPT-4 with vision capabilities - State-of-the-art VQA and OCR - Real-world applications - [Paper](https://cdn.openai.com/papers/GPTV_System_Card.pdf)	[Gemini 2.5](./papers/multimodal/29-gemini-2.5/summary.md) (2025) - 🔥 CRITICAL - Most advanced multimodal AI - Native multimodal (text, image, audio, video) - 1M context, 3-hour video understanding - Integrated thinking mode - [Paper](https://arxiv.org/abs/2507.06261)
[SAM 2](./papers/multimodal/32-sam2/summary.md) (2024) - ⭐ HIGH - Universal video segmentation - 44 FPS real-time performance - Zero-shot generalization - Segment anything in video - [Paper](https://arxiv.org/abs/2408.00714)	[LLaVA](./papers/multimodal/46-llava/summary.md) (2023) - 🔥 CRITICAL - Vision encoder + projection + LLM - Instruction data generated by text-only GPT-4 - Democratized open vision-language models - [Paper](https://arxiv.org/abs/2304.08485)	[Gemini 3](./papers/multimodal/47-gemini3/summary.md) (2025) - ⭐ HIGH - First model past 1500 LMArena Elo - 95% AIME 2025, 91.8% MMLU - Deep Think mode for the hardest reasoning - [Paper](https://blog.google/products-and-platforms/products/gemini/gemini-3/)
[Whisper](./papers/multimodal/49-whisper/summary.md) (2022) - 🔥 CRITICAL - 680,000 hours of weakly supervised audio - 99 languages, zero-shot, no fine-tuning - The default open speech recognition model - [Paper](https://arxiv.org/abs/2212.04356)

**[AudioLM](./papers/multimodal/121-audiolm/summary.md)** (2022) - ⭐ **HIGH** - Language modelling for sound - Semantic plus acoustic tokens - Coherent speech and piano continuations - [Paper](https://arxiv.org/abs/2209.03143)

**[VALL-E](./papers/multimodal/122-vall-e/summary.md)** (2023) - ⭐ **HIGH** - Voice cloning from 3 seconds - Text-to-speech as codec language modelling - Zero-shot speaker imitation - [Paper](https://arxiv.org/abs/2301.02111)

⚡ Techniques & Methods¶

The methods layer: how models are trained, aligned, made to reason, served fast, evaluated, interpreted, and pointed at problems outside language. 75 papers.

[Chain-of-Thought](./papers/techniques/09-chain-of-thought/summary.md) (2022) - 🔥 CRITICAL - Reasoning breakthrough - Step-by-step reasoning prompts - "Let's think step by step" - Improves complex problem-solving - [Paper](https://arxiv.org/abs/2201.11903)	[LoRA](./papers/techniques/10-lora/summary.md) (2021) - 🔥 CRITICAL - Efficient fine-tuning - Low-rank adaptation - 10,000× fewer trainable parameters - Enables custom models - [Paper](https://arxiv.org/abs/2106.09685)	[Scaling Laws](./papers/techniques/12-scaling-laws/summary.md) (2020) - 🔥 CRITICAL - Predictive theory - Predictable power laws - Guides compute allocation - Justified massive investments - [Paper](https://arxiv.org/abs/2001.08361)
[RAG](./papers/techniques/13-rag/summary.md) (2020) - 🔥 CRITICAL - Production standard - Retrieval-augmented generation - Reduces hallucinations - Production LLM standard - [Paper](https://arxiv.org/abs/2005.11401)	[FlashAttention](./papers/techniques/16-flash-attention/summary.md) (2022) - 🔥 CRITICAL - IO-aware attention - 10-20× faster than standard attention - Enables 64k+ context lengths - Powers all modern long-context LLMs - [Paper](https://arxiv.org/abs/2205.14135)	[Chinchilla](./papers/techniques/18-chinchilla/summary.md) (2022) - 🔥 CRITICAL - Rewrote scaling laws - Equal scaling of params and tokens - Proved GPT-3 was undertrained 4× - Validated by LLaMA - [Paper](https://arxiv.org/abs/2203.15556)
[ReAct](./papers/techniques/21-react/summary.md) (2023) - 🔥 CRITICAL - AI agents foundation - Synergizing reasoning and acting - Interleaves thought and action - Powers ChatGPT plugins, LangChain - [Paper](https://arxiv.org/abs/2210.03629)	[QLoRA](./papers/techniques/22-qlora/summary.md) (2023) - 🔥 CRITICAL - Efficient fine-tuning at scale - 4-bit quantization + LoRA - Fine-tune 65B on consumer GPU - 16× memory reduction - [Paper](https://arxiv.org/abs/2305.14314)	[Toolformer](./papers/techniques/24-toolformer/summary.md) (2023) - ⭐ HIGH - Self-taught tool use - LLMs learn to use tools automatically - No manual annotations needed - Inspired ChatGPT function calling - [Paper](https://arxiv.org/abs/2302.04761)
[Tree of Thoughts](./papers/techniques/25-tree-of-thoughts/summary.md) (2023) - ⭐ HIGH - Advanced reasoning - Tree search over reasoning paths - Deliberate problem solving - 18× better than CoT on hard problems - [Paper](https://arxiv.org/abs/2305.10601)	[Meta-CoT](./papers/techniques/34-meta-cot/summary.md) (2025) - ⭐ HIGH - System 2 reasoning - Metacognitive strategies - Deliberate problem-solving - Next-gen reasoning approach - [Paper](https://arxiv.org/abs/2501.xxxxx)	[rStar-Math](./papers/techniques/35-rstar-math/summary.md) (2025) - ⭐ HIGH - Small models rival large ones - MCTS for math reasoning - 7B model beats 70B+ competitors - Efficient reasoning breakthrough - [Paper](https://arxiv.org/abs/2501.04519)
[GRPO](./papers/techniques/38-grpo/summary.md) (2024) - 🔥 CRITICAL - The algorithm behind DeepSeek-R1 - Drops PPO's critic model entirely - Group-relative advantage, ~50% less memory - [Paper](https://arxiv.org/abs/2402.03300)	[RLVR](./papers/techniques/39-rlvr/summary.md) (2024-2025) - 🔥 CRITICAL - Reward correctness, not human preference - No reward model to train or to game - Where reasoning models get their training signal - [Paper](https://arxiv.org/abs/2501.12948)	[Speculative Decoding](./papers/techniques/45-speculative-decoding/summary.md) (2022) - 🔥 CRITICAL - 2-3x faster, outputs provably unchanged - Small draft model proposes, big model verifies - Deployed in every serious serving stack - [Paper](https://arxiv.org/abs/2211.17192)
[Scaling Test-Time Compute](./papers/techniques/50-test-time-compute/summary.md) (2024) - 🔥 CRITICAL - Think longer instead of training bigger - A small model that searches can match a 14x larger one - The theory underneath o1 and R1 - [Paper](https://arxiv.org/abs/2408.03314)	[Process Reward Models](./papers/techniques/51-process-reward-models/summary.md) (2023) - 🔥 CRITICAL - Grade every step, not just the answer - PRM800K: 800,000 step-level human labels - How reasoning models verify themselves - [Paper](https://arxiv.org/abs/2305.20050)	[PagedAttention & vLLM](./papers/techniques/52-pagedattention-vllm/summary.md) (2023) - 🔥 CRITICAL - OS paging applied to the KV cache - KV memory waste from 60-80% down to under 4% - 24x throughput; vLLM became the standard - [Paper](https://arxiv.org/abs/2309.06180)
[Word2Vec](./papers/techniques/53-word2vec/summary.md) (2013) - 📚 HISTORICAL - Dense embeddings replace one-hot vectors - king - man + woman ≈ queen - Started representation learning for text - [Paper](https://arxiv.org/abs/1301.3781)	[RoPE](./papers/techniques/54-rope-rotary-position-embedding/summary.md) (2021) - 🔥 CRITICAL - Relative position by rotating the query and key - Zero learned parameters, a few dozen lines - In LLaMA, PaLM, Gemini, Mistral, DeepSeek, Qwen - [Paper](https://arxiv.org/abs/2104.09864)	[Generative Agents](./papers/techniques/58-generative-agents/summary.md) (2023) - ⭐ HIGH - 25 agents living in a sandbox town - The memory -> reflection -> planning loop - Unscripted emergent social behavior - [Paper](https://arxiv.org/abs/2304.03442)
[Model Context Protocol (MCP)](./papers/techniques/59-model-context-protocol/summary.md) (2024) - 🔥 CRITICAL - One protocol for model-to-tool wiring - Turns M x N bespoke integrations into M + N - Industry-wide adoption within months - [Paper](https://www.anthropic.com/news/model-context-protocol)	[GraphRAG](./papers/techniques/60-graph-rag/summary.md) (2024) - ⭐ HIGH - Answers global "what are the themes?" queries - LLM-extracted knowledge graph over the corpus - Hierarchical community summaries, map-reduce answering - [Paper](https://arxiv.org/abs/2404.16130)	[AlphaGeometry](./papers/techniques/61-alphageometry/summary.md) (2024) - ⭐ HIGH - 25/30 IMO geometry problems solved - Neuro-symbolic: LM proposes, solver deduces - Trained on 100M synthetic theorems, zero human proofs - [Paper](https://www.nature.com/articles/s41586-023-06747-5)
[AlphaEvolve](./papers/techniques/62-alphaevolve/summary.md) (2025) - ⭐ HIGH - Broke Strassen's 1969 matrix-multiply record - Gemini proposes, evolutionary search selects - Already running in Google production - [Paper](https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/)	[PPO](./papers/techniques/63-ppo/summary.md) (2017) - 🔥 CRITICAL - The RL algorithm underneath RLHF - Clipped objective: TRPO's stability without the pain - What InstructGPT and ChatGPT were trained with - [Paper](https://arxiv.org/abs/1707.06347)	[AlphaFold 2](./papers/techniques/68-alphafold/summary.md) (2021) - 🔥 CRITICAL - Solved 50-year protein folding problem - Attention generalizing far beyond language - 200M+ predicted structures released free - [Paper](https://www.nature.com/articles/s41586-021-03819-2)
[ZeRO & Megatron-LM](./papers/techniques/76-zero-megatron/summary.md) (2019) - 🔥 CRITICAL - 3D parallelism - The systems layer under every big run - DeepSpeed, FSDP, tensor parallelism - [Paper](https://arxiv.org/abs/1910.02054)	[Self-Consistency](./papers/techniques/77-self-consistency/summary.md) (2022) - 🔥 CRITICAL - Sample N, majority vote - +17.9 points on GSM8K over greedy CoT - Inference compute buys accuracy - [Paper](https://arxiv.org/abs/2203.11171)	[Reflexion](./papers/techniques/78-reflexion/summary.md) (2023) - 🔥 CRITICAL - Agents learn from failure - Verbal reflection, no weight updates - The retry loop every coding agent runs - [Paper](https://arxiv.org/abs/2303.11366)
[Self-Instruct](./papers/techniques/79-self-instruct/summary.md) (2022) - 🔥 CRITICAL - 52K instructions from 175 - Made instruction data free; led to Alpaca - Why post-training data is synthetic now - [Paper](https://arxiv.org/abs/2212.10560)	[FLAN](./papers/techniques/80-flan/summary.md) (2021) - 🔥 CRITICAL - Invented instruction tuning - Beat GPT-3 175B on 20 of 25 benchmarks - The middle stage of the modern pipeline - [Paper](https://arxiv.org/abs/2109.01652)	[Emergent Abilities](./papers/techniques/81-emergent-abilities/summary.md) (2022-2023) - ⭐ HIGH - Do capabilities jump at scale? - Paired with the "Mirage" rebuttal - Why exact-match alone misleads - [Paper](https://arxiv.org/abs/2206.07682)
[Sparse Autoencoders](./papers/techniques/82-sparse-autoencoders/summary.md) (2022-2024) - 🔥 CRITICAL - Reading a model's concepts - Millions of features from Claude 3 Sonnet - Steering works: "Golden Gate Claude" - [Paper](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)	[Sleeper Agents](./papers/techniques/83-sleeper-agents/summary.md) (2024) - 🔥 CRITICAL - Backdoors survive safety training - SFT, RLHF and adversarial training all failed - Adversarial training hid the trigger better - [Paper](https://arxiv.org/abs/2401.05566)	[SWE-bench](./papers/techniques/84-swe-bench/summary.md) (2023) - 🔥 CRITICAL - Real GitHub issues, real tests - ~2% resolved in 2023, ~70-80% in 2025 - The coding-agent headline metric - [Paper](https://arxiv.org/abs/2310.06770)
[LLM-as-a-Judge](./papers/techniques/85-llm-as-judge/summary.md) (2023) - 🔥 CRITICAL - Model-graded evaluation - ~80% agreement with humans, plus the biases - Created Chatbot Arena Elo - [Paper](https://arxiv.org/abs/2306.05685)	[GPTQ & AWQ](./papers/techniques/86-gptq-awq-quantization/summary.md) (2022-2023) - 🔥 CRITICAL - 4-bit post-training quantization - 70B from ~140 GB to ~35 GB - Why local inference exists - [Paper](https://arxiv.org/abs/2306.00978)	[Dense Retrieval](./papers/techniques/87-dense-retrieval/summary.md) (2019-2020) - 🔥 CRITICAL - DPR, ColBERT, Sentence-BERT - The retriever every RAG system runs on - Where RAG actually fails, and the fixes - [Paper](https://arxiv.org/abs/2004.04906)
[STaR](./papers/techniques/97-star/summary.md) (2022) - ⭐ HIGH - Generate rationales, keep the ones that work - Rationalize backwards from known answers - The quiet ancestor of o1 and R1 - [Paper](https://arxiv.org/abs/2203.14465)	[Quiet-STaR](./papers/techniques/98-quiet-star/summary.md) (2024) - ⭐ HIGH - Learn to think before every token - Generalizes STaR past math to arbitrary text - Predates o1's hidden chain-of-thought - [Paper](https://arxiv.org/abs/2403.09629)	[Self-Refine](./papers/techniques/99-self-refine/summary.md) (2023) - ⭐ HIGH - One model critiques and rewrites itself - No extra training, no second model - Codified the LLM-as-critic pattern - [Paper](https://arxiv.org/abs/2303.17651)
[Voyager](./papers/techniques/100-voyager/summary.md) (2023) - ⭐ HIGH - Lifelong open-ended agent in Minecraft - Grows a reusable skill library written as code - GPT-4 planning over multi-hour horizons - [Paper](https://arxiv.org/abs/2305.16291)	[AlphaFold 3](./papers/techniques/101-alphafold3/summary.md) (2024) - ⭐ HIGH - Proteins plus DNA, RNA, ligands and ions - A diffusion module replaces the structure head - Predicts interactions, not just single shapes - [Paper](https://www.nature.com/articles/s41586-024-07487-w)	[AlphaZero](./papers/techniques/102-alphazero/summary.md) (2017) - 🔥 CRITICAL - Superhuman from self-play and the rules alone - MCTS guided by a single policy-value network - Ancestor of every self-improvement loop in AI - [Paper](https://arxiv.org/abs/1712.01815)
[KTO](./papers/techniques/103-kto/summary.md) (2024) - ⭐ HIGH - Alignment without paired preference data - Prospect theory turned into a loss function - Thumbs-up / thumbs-down signals are enough - [Paper](https://arxiv.org/abs/2402.01306)	[Genie](./papers/techniques/104-genie/summary.md) (2024) - ⭐ HIGH - A playable world from a single image - Latent actions learned unsupervised from video - The first foundation world model - [Paper](https://arxiv.org/abs/2402.15391)	[DreamerV3](./papers/techniques/105-dreamerv3/summary.md) (2023) - ⭐ HIGH - Minecraft diamonds from scratch, no demos - One hyperparameter set across 150+ tasks - Learn a world model, then plan in imagination - [Paper](https://arxiv.org/abs/2301.04104)
[ESM-2 / ESMFold](./papers/techniques/106-esm/summary.md) (2023) - ⭐ HIGH - A 15B language model trained on protein sequences - Structure with no MSA search, up to 60x faster - 617M predicted metagenomic structures released - [Paper](https://doi.org/10.1126/science.ade2574)	[CICERO](./papers/techniques/107-cicero/summary.md) (2022) - ⭐ HIGH - Human-level Diplomacy, negotiation included - Language generation fused with strategic planning - Top 10% across 40 games against humans - [Paper](https://doi.org/10.1126/science.ade9097)

[Induction Heads](./papers/techniques/126-induction-heads/summary.md) (2022) - ⭐ HIGH - The circuit behind in-context learning - [A][B]...[A] -> [B] pattern completion - Forms in a sudden training phase change - [Paper](https://arxiv.org/abs/2209.11895)	[GCG Adversarial Attacks](./papers/techniques/127-gcg-adversarial-attacks/summary.md) (2023) - ⭐ HIGH - Automated, transferable jailbreaks - Optimised suffixes bypass safety training - Found on open models, transfer to closed ones - [Paper](https://arxiv.org/abs/2307.15043)	[Weak-to-Strong Generalization](./papers/techniques/128-weak-to-strong/summary.md) (2023) - ⭐ HIGH - Can weak supervisors align strong models? - Weak labels still elicit strong ability - Performance gap recovered (PGR) - [Paper](https://arxiv.org/abs/2312.09390)
[Red Teaming LMs](./papers/techniques/129-red-teaming-lms/summary.md) (2022) - ⭐ HIGH - One model attacks another - Generated test cases at scale - Found offensive replies and data leaks - [Paper](https://arxiv.org/abs/2202.03286)	[YaRN](./papers/techniques/130-yarn-context-extension/summary.md) (2023) - 🔥 CRITICAL - Cheap context extension for RoPE models - 4K models extended to 128K - Per-frequency interpolation plus temperature - [Paper](https://arxiv.org/abs/2309.00071)	[RULER](./papers/techniques/132-ruler/summary.md) (2024) - ⭐ HIGH - Claimed vs effective context length - 13 synthetic tasks beyond needle tests - Most models fall short of their window - [Paper](https://arxiv.org/abs/2404.06654)
[FineWeb](./papers/techniques/133-fineweb/summary.md) (2024) - ⭐ HIGH - Open 15T-token web data, ablated - Every curation choice tested - FineWeb-Edu lifts knowledge benchmarks - [Paper](https://arxiv.org/abs/2406.17557)	[Knowledge Distillation](./papers/techniques/134-knowledge-distillation/summary.md) (2015) - 🔥 CRITICAL - Teach a small model with a big one - Soft targets carry 'dark knowledge' - Temperature-softened softmax - [Paper](https://arxiv.org/abs/1503.02531)	[BPE Subword Units](./papers/techniques/136-bpe-subword-units/summary.md) (2015) - 🔥 CRITICAL - Where tokens come from - Merge frequent pairs into subwords - Ended the unknown-word problem - [Paper](https://arxiv.org/abs/1508.07909)
[MMLU](./papers/techniques/137-mmlu/summary.md) (2020) - 🔥 CRITICAL - The benchmark every launch quoted - 57 subjects, multiple choice - From 44% to saturation above 90% - [Paper](https://arxiv.org/abs/2009.03300)	[ARC-AGI](./papers/techniques/138-arc-agi/summary.md) (2019) - ⭐ HIGH - Intelligence as skill-acquisition efficiency - Abstract grid puzzles from few examples - Hard for models, easy for people - [Paper](https://arxiv.org/abs/1911.01547)	[OSWorld](./papers/techniques/139-osworld/summary.md) (2024) - 🔥 CRITICAL - Real computers, real tasks - 369 tasks in a live Ubuntu VM - Humans 72%, best model 12% at launch - [Paper](https://arxiv.org/abs/2404.07972)
[Model Collapse](./papers/techniques/140-model-collapse/summary.md) (2024) - ⭐ HIGH - Risk of training on AI output - Recursive training erases the tails - Keeping real data largely avoids it - [Paper](https://www.nature.com/articles/s41586-024-07566-y)	[Adam](./papers/techniques/142-adam/summary.md) (2014) - 🔥 CRITICAL - The default optimiser - Per-parameter adaptive step sizes - AdamW trains nearly every LLM - [Paper](https://arxiv.org/abs/1412.6980)	[Mixed Precision Training](./papers/techniques/143-mixed-precision-training/summary.md) (2017) - ⭐ HIGH - Half the bits, same model - FP16 maths with FP32 master weights - Loss scaling rescues small gradients - [Paper](https://arxiv.org/abs/1710.03740)
[Muon](./papers/techniques/144-muon/summary.md) (2024) - ⭐ HIGH - First serious challenger to AdamW - Orthogonalised momentum updates - About 2x compute efficiency at scale - [Paper](https://arxiv.org/abs/2502.16982)
[Measuring AI Ability to Complete Long Tasks](./papers/techniques/145-metr-time-horizons/summary.md) (2025) - ⭐ HIGH - How long a task can an AI finish? - Measures capability in human-hours, not benchmark scores - Time horizon doubled roughly every seven months - [Paper](https://arxiv.org/abs/2503.14499)	[Deep RL from Human Preferences](./papers/techniques/146-deep-rl-from-human-preferences/summary.md) (2017) - 🔥 CRITICAL - Where RLHF began - Learns a reward model from human comparisons of short clips - Ordinary deep RL then optimises that learned reward - [Paper](https://arxiv.org/abs/1706.03741)	[Learning to Summarize from Human Feedback](./papers/techniques/147-learning-to-summarize-from-human-feedback/summary.md) (2020) - ⭐ HIGH - The bridge from 2017 RLHF to InstructGPT - Preferences, then a reward model, then PPO with a KL penalty - Beat human-written reference summaries on TL;DR - [Paper](https://arxiv.org/abs/2009.01325)
[Why Language Models Hallucinate](./papers/techniques/149-why-language-models-hallucinate/summary.md) (2025) - ⭐ HIGH - Hallucination as a statistical and grading problem - Traces hallucination to classification errors in pretraining - Argues benchmark grading rewards confident guessing - [Paper](https://arxiv.org/abs/2509.04664)
[Circuit Tracing and the Biology of an LLM](./papers/techniques/151-circuit-tracing/summary.md) (2025) - ⭐ HIGH - Watching a model think, step by step - Attribution graphs trace how features feed each other - Shows multi-step reasoning and poem planning inside Claude 3.5 Haiku - [Post](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)	[Reasoning Models Don't Always Say What They Think](./papers/techniques/152-reasoning-models-dont-say-what-they-think/summary.md) (2025) - ⭐ HIGH - Is the chain of thought honest? - Hint experiments test whether reasoning traces admit what they used - Claude 3.7 Sonnet 25%, DeepSeek R1 39% mentioned the hint - [Paper](https://arxiv.org/abs/2505.05410)	[Emergent Misalignment](./papers/techniques/153-emergent-misalignment/summary.md) (2025) - ⭐ HIGH - Narrow training, broad misbehaviour - Fine-tuning on insecure code makes models broadly misaligned - Controls show it is not a plain jailbreak - [Paper](https://arxiv.org/abs/2502.17424)
[s1: Simple Test-Time Scaling](./papers/techniques/154-s1-simple-test-time-scaling/summary.md) (2025) - ⭐ HIGH - Reasoning from 1,000 examples - s1K: a tiny curated reasoning dataset - Budget forcing controls how long the model thinks - [Paper](https://arxiv.org/abs/2501.19393)	[DAPO](./papers/techniques/155-dapo/summary.md) (2025) - ⭐ HIGH - An open RLVR recipe at scale - Clip-higher, dynamic sampling, token-level loss - Overlong reward shaping stabilises long reasoning - [Paper](https://arxiv.org/abs/2503.14476)

🦾 Robotics & Embodied AI¶

Vision-language-action models and the shared datasets behind them - how web-scale pretraining started to reach physical robots. 3 papers.

**[RT-2](./papers/robotics/123-rt2/summary.md)** (2023) - 🔥 **CRITICAL** - Defined vision-language-action models - Robot actions emitted as text tokens - Web knowledge transfers to control - [Paper](https://arxiv.org/abs/2307.15818)

**[pi0](./papers/robotics/124-pi0/summary.md)** (2024) - ⭐ **HIGH** - Flow-matching VLA with open weights - VLM backbone plus action expert - Dexterous tasks at up to 50 Hz - [Paper](https://arxiv.org/abs/2410.24164)

**[Open X-Embodiment](./papers/robotics/125-open-x-embodiment/summary.md)** (2023) - ⭐ **HIGH** - Robotics' shared dataset - 1M+ trajectories, 22 robot types - Cross-robot training helps - [Paper](https://arxiv.org/abs/2310.08864)

✍️ Essays & Landmark Posts¶

Not papers, but writing that set the field's direction - from Turing's imitation game to the arguments about scaling, agents and where it all leads. Pitches summarise the essays' claims, not this collection's endorsement. 10 papers.

[Computing Machinery and Intelligence](./papers/essays/108-computing-machinery-and-intelligence/summary.md) (1950) - 🔥 CRITICAL - The question that started the field - Proposes the imitation game (Turing test) - Answers nine objections still raised today - [Essay](https://academic.oup.com/mind/article/LIX/236/433/986238)	[The Unreasonable Effectiveness of RNNs](./papers/essays/109-unreasonable-effectiveness-of-rnns/summary.md) (2015) - 📚 HISTORICAL - The post that made people try it - Character-level RNNs generate Shakespeare, code, LaTeX - Previewed next-token prediction's power - [Essay](https://karpathy.github.io/2015/05/21/rnn-effectiveness/)	[Software 2.0](./papers/essays/110-software-2/summary.md) (2017) - ⭐ HIGH - Neural nets as a new way to write software - Code written by optimisation, not by hand - Framed the ML engineering shift - [Essay](https://karpathy.medium.com/software-2-0-a64152b37c35)
[The Bitter Lesson](./papers/essays/111-bitter-lesson/summary.md) (2019) - 🔥 CRITICAL - General methods plus compute win - Search and learning beat hand-built knowledge - The one-page thesis of the scaling era - [Essay](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)	[The Scaling Hypothesis](./papers/essays/112-scaling-hypothesis/summary.md) (2020) - ⭐ HIGH - GPT-3 read as a turning point - Bigger models unlock new abilities - Made the scaling bet explicit in 2020 - [Essay](https://gwern.net/scaling-hypothesis)	[Situational Awareness](./papers/essays/113-situational-awareness/summary.md) (2024) - ⭐ HIGH - A dated AGI forecast, and a policy pitch - 'Count the OOMs' to AGI around 2027 - Trillion-dollar clusters and national security - [Essay](https://situational-awareness.ai/)
[Machines of Loving Grace](./papers/essays/114-machines-of-loving-grace/summary.md) (2024) - ⭐ HIGH - The optimistic case, made concretely - What powerful AI could do for biology, health, economies - 'Compressed 21st century' thesis - [Essay](https://www.darioamodei.com/essay/machines-of-loving-grace)	[Building Effective Agents](./papers/essays/115-building-effective-agents/summary.md) (2024) - 🔥 CRITICAL - Workflows before agents - Five composable patterns for LLM systems - Start simple; add autonomy only when needed - [Essay](https://www.anthropic.com/engineering/building-effective-agents)	[Welcome to the Era of Experience](./papers/essays/116-era-of-experience/summary.md) (2025) - ⭐ HIGH - Beyond human data, to experience - Agents learning from streams of interaction - Grounded rewards, not human judgement - [Essay](https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf)
[On the Dangers of Stochastic Parrots](./papers/essays/150-stochastic-parrots/summary.md) (2021) - ⭐ HIGH - The critique side of the scaling debate - Argues bigger language models carry cost and data risks - Says fluent text is form without grounded meaning - [Essay](https://dl.acm.org/doi/10.1145/3442188.3445922)

📊 Quick Stats¶

Every paper in the collection has a card on this page.

Category	Papers
Foundational Architectures	14
Language Models	27
Image, Video & 3D Generation	18
Multimodal & Audio	9
Robotics & Embodied AI	3
Techniques & Methods	75
Essays & Landmark Posts	10
Total	156

The authoritative counts live in papers.json and are regenerated by scripts/build_manifest.py; scripts/check_counts.py fails CI if the table above drifts from it.

🔍 Filter by Badge¶

Relevance ratings are editorial, and describe how much the paper matters to a reader today:

🔥 CRITICAL (74 papers) - Essential - read these to understand the field
⭐ HIGH (74 papers) - Important, with significant downstream impact
📚 HISTORICAL (7 papers) - Formative context; superseded in practice
📖 THEORY (1 paper) - Theoretical foundations

Want a structured learning path? → docs/ROADMAP.md · README.md

Want the generated listings? → INDEX.md · TAGS.md

Wondering what is missing? → docs/GAPS.md

[Attention Is All You Need](./papers/architectures/01-attention-is-all-you-need/summary.md) (2017) - 🔥 CRITICAL - Foundation of everything - Introduced Transformer architecture - Self-attention mechanism - [Paper](https://arxiv.org/abs/1706.03762)	[Vision Transformer (ViT)](./papers/architectures/11-vision-transformer/summary.md) (2020) - ⭐ HIGH - Transformers for computer vision - Images as patch sequences - Enables multimodal models - [Paper](https://arxiv.org/abs/2010.11929)	[Mamba](./papers/architectures/20-mamba/summary.md) (2023) - 🔥 CRITICAL - First viable Transformer alternative - Linear-time sequence modeling (O(n) vs O(n²)) - Selective state spaces - [Paper](https://arxiv.org/abs/2312.00752)
[Mixtral & Mixture-of-Experts](./papers/architectures/37-mixture-of-experts/summary.md) (2024) - 🔥 CRITICAL - 47B parameters, 13B active - Matches LLaMA 2 70B at 6x faster inference - The architecture DeepSeek-V3, Qwen3 and Llama 4 adopted - [Paper](https://arxiv.org/abs/2401.04088)	[Seq2Seq](./papers/architectures/55-seq2seq/summary.md) (2014) - 📚 HISTORICAL - Variable-length in, variable-length out - Stacked LSTM encoder-decoder for translation - Its fixed-size bottleneck is what attention fixed - [Paper](https://arxiv.org/abs/1409.3215)	[Bahdanau Attention](./papers/architectures/66-bahdanau-attention/summary.md) (2014) - 🔥 CRITICAL - Where attention was invented - Soft alignment over every encoder state - The conceptual seed of self-attention - [Paper](https://arxiv.org/abs/1409.0473)
[Switch Transformer](./papers/architectures/67-switch-transformer/summary.md) (2021) - ⭐ HIGH - Trillion parameters at near-constant compute - Top-1 routing: one expert per token - The sparsity recipe Mixtral productionized - [Paper](https://arxiv.org/abs/2101.03961)	[ResNet](./papers/architectures/73-resnet/summary.md) (2015) - 🔥 CRITICAL - Residual connections - `x + f(x)` is in every Transformer - Most-cited deep learning paper - [Paper](https://arxiv.org/abs/1512.03385)	[U-Net](./papers/architectures/74-unet/summary.md) (2015) - ⭐ HIGH - Encoder-decoder + skips - The denoiser inside diffusion models - Trained on 30 images, won its challenge - [Paper](https://arxiv.org/abs/1505.04597)
[Grouped-Query Attention](./papers/architectures/75-grouped-query-attention/summary.md) (2023) - 🔥 CRITICAL - 8x smaller KV cache - In Llama 2/3/4, Mistral, Qwen, Gemma - Why long context is affordable - [Paper](https://arxiv.org/abs/2305.13245)	[Masked Autoencoders (MAE)](./papers/architectures/88-mae/summary.md) (2021) - ⭐ HIGH - BERT-style pretraining for vision - Hide 75% of patches, reconstruct the rest - Asymmetric encoder-decoder, cheap to scale - [Paper](https://arxiv.org/abs/2111.06377)

[BERT](./papers/language-models/03-bert/summary.md) (2018) - 📚 HISTORICAL - Pre-training revolution - Bidirectional pre-training - Masked language modeling - [Paper](https://arxiv.org/abs/1810.04805)	[GPT-3](./papers/language-models/04-gpt3-few-shot-learners/summary.md) (2020) - ⭐ HIGH - Few-shot learning paradigm - 175B parameters - Foundation for ChatGPT - [Paper](https://arxiv.org/abs/2005.14165)	[InstructGPT (RLHF)](./papers/language-models/05-instructgpt-rlhf/summary.md) (2022) - 🔥 CRITICAL - Human preference learning - Enabled ChatGPT - RLHF methodology - [Paper](https://arxiv.org/abs/2203.02155)
[Constitutional AI](./papers/language-models/14-constitutional-ai/summary.md) (2022) - ⭐ HIGH - Alternative to RLHF - AI self-critique via principles - Powers Claude - [Paper](https://arxiv.org/abs/2212.08073)	[LLaMA](./papers/language-models/15-llama/summary.md) (2023) - 🔥 CRITICAL - Compute-optimal training - 13B matches GPT-3 175B - Open weights - [Paper](https://arxiv.org/abs/2302.13971)	[LLaMA 2](./papers/language-models/17-llama2/summary.md) (2023) - 🔥 CRITICAL - Production-ready open model - Commercial license - RLHF alignment - [Paper](https://arxiv.org/abs/2307.09288)
[DPO](./papers/language-models/19-dpo/summary.md) (2023) - 🔥 CRITICAL - Simpler than RLHF - Direct preference optimization - No reward model needed - [Paper](https://arxiv.org/abs/2305.18290)	[DeepSeek-R1](./papers/language-models/26-deepseek-r1/summary.md) (2025) - 🔥 CRITICAL - Pure RL reasoning - Matches OpenAI o1 - Fully open source - [Paper](https://arxiv.org/abs/2501.12948)	[DeepSeek-V3](./papers/language-models/27-deepseek-v3/summary.md) (2024) - 🔥 CRITICAL - $5.76M training cost - 671B MoE architecture - Matches GPT-4 efficiency - [Paper](https://arxiv.org/abs/2412.19437)
[Qwen3](./papers/language-models/28-qwen3/summary.md) (2025) - 🔥 CRITICAL - Unified thinking/non-thinking - Adaptive reasoning modes - Best of both worlds - [Paper](https://arxiv.org/abs/2505.09388)	[Claude 3.5 Sonnet](./papers/language-models/30-claude-3.5-sonnet/summary.md) (2024) - 🔥 CRITICAL - Computer use capability - Best coding model (49% SWE-Bench) - AI controls computers - [Announcement](https://www.anthropic.com/news/3-5-models-and-computer-use) - [Paper](https://www.anthropic.com/news/3-5-models-and-computer-use)	[OpenAI o1](./papers/language-models/31-openai-o1/summary.md) (2024) - 🔥 CRITICAL - Started reasoning model era - PhD-level performance - RL for reasoning - [Announcement](https://openai.com/index/learning-to-reason-with-llms/) - [Paper](https://openai.com/index/learning-to-reason-with-llms/)
[LLaMA 3.3](./papers/language-models/33-llama3.3/summary.md) (2024) - ⭐ HIGH - Distillation breakthrough - 70B matches 405B performance - Knowledge transfer success - [Paper](https://www.meta.ai/blog/meta-llama-3-3/)	[GPT-4](./papers/language-models/36-gpt4/summary.md) (2023) - 🔥 CRITICAL - The capability jump over GPT-3.5 - Bar exam top 10%; first multimodal GPT - Loss predicted in advance from smaller runs - [Paper](https://arxiv.org/abs/2303.08774)	[GPT-4o](./papers/language-models/40-gpt4o/summary.md) (2024) - ⭐ HIGH - Text, audio and vision in one model - 232ms average voice response - Half the price and 2x the speed of GPT-4 Turbo - [Paper](https://cdn.openai.com/gpt-4o-system-card.pdf)
[Llama 4](./papers/language-models/41-llama4/summary.md) (2025) - ⭐ HIGH - Meta's first MoE models - 10M-token context window on Scout - Natively multimodal, open weights - [Paper](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)	[GPT-5](./papers/language-models/42-gpt5/summary.md) (2025) - 🔥 CRITICAL - One model routing fast vs deep reasoning - 94.6% AIME 2025, 74.9% SWE-bench Verified - Ended the GPT-4 / o1 / o3 product split - [Paper](https://cdn.openai.com/gpt-5-system-card.pdf)	[Claude 4 Family](./papers/language-models/43-claude4/summary.md) (2025) - 🔥 CRITICAL - State of the art at agentic work - 80.9% SWE-bench Verified (Opus 4.5) - Extended thinking with controllable depth - [Paper](https://www.anthropic.com/news/claude-4)
[Codex](./papers/language-models/56-codex/summary.md) (2021) - ⭐ HIGH - The model behind GitHub Copilot - Introduced HumanEval and the pass@k metric - Ancestor of every code model since - [Paper](https://arxiv.org/abs/2107.03374)	[GPT-2](./papers/language-models/64-gpt2/summary.md) (2019) - 📚 HISTORICAL - GPT-1 scaled to 1.5B parameters - Zero-shot transfer with no fine-tuning - The staged-release safety debate started here - [Paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)	[T5](./papers/language-models/65-t5/summary.md) (2019) - ⭐ HIGH - Every NLP task as text-in, text-out - Systematic ablation of the whole pretraining recipe - Gave the field the C4 corpus - [Paper](https://arxiv.org/abs/1910.10683)
[GPT-1](./papers/language-models/93-gpt1/summary.md) (2018) - 📚 HISTORICAL - Where the GPT recipe begins - Generative pretraining, then task fine-tuning - 117M parameters, decoder-only Transformer - [Paper](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)	[PaLM](./papers/language-models/94-palm/summary.md) (2022) - ⭐ HIGH - 540B, largest dense Transformer of its day - Trained across two TPU pods via Pathways - Where chain-of-thought results broke out - [Paper](https://arxiv.org/abs/2204.02311)	[Mistral 7B](./papers/language-models/95-mistral-7b/summary.md) (2023) - 🔥 CRITICAL - Made open weights genuinely competitive - Beat LLaMA 2 13B on every benchmark tested - Apache 2.0, with GQA and sliding-window attention - [Paper](https://arxiv.org/abs/2310.06825)
[Llama Guard](./papers/language-models/96-llama-guard/summary.md) (2023) - ⭐ HIGH - Safety classification as an LLM task - Moderates both the prompt and the response - The open-source production reference - [Paper](https://arxiv.org/abs/2312.06674)

[GANs](./papers/image-generation/02-generative-adversarial-networks/summary.md) (2014) - 📚 HISTORICAL - Generative modeling origins - Adversarial training - Generator vs discriminator - [Paper](https://arxiv.org/abs/1406.2661)	[Diffusion Models (DDPM)](./papers/image-generation/06-diffusion-models/summary.md) (2020) - 📖 THEORY - Diffusion foundations - Iterative denoising - Better than GANs - [Paper](https://arxiv.org/abs/2006.11239)	[Stable Diffusion](./papers/image-generation/07-stable-diffusion/summary.md) (2022) - ⭐ HIGH - Practical implementation - Latent space diffusion (10-100× faster) - Open-source, democratized AI art - [Paper](https://arxiv.org/abs/2112.10752)
[Sora & Diffusion Transformers](./papers/image-generation/44-sora-dit/summary.md) (2022-2024) - 🔥 CRITICAL - Diffusion on a Transformer backbone - Video as spacetime patches, any resolution - Launched the AI video generation industry - [Paper](https://arxiv.org/abs/2212.09748)	[DALL-E 3](./papers/image-generation/48-dalle3/summary.md) (2023) - ⭐ HIGH - Follows the prompt you actually wrote - Recaptioned training data was the whole trick - First model to render readable text - [Paper](https://cdn.openai.com/papers/dall-e-3.pdf)	[Variational Autoencoder (VAE)](./papers/image-generation/57-vae/summary.md) (2013) - 🔥 CRITICAL - The reparameterization trick - A smooth, interpolatable latent space - The latent half of latent diffusion - [Paper](https://arxiv.org/abs/1312.6114)
[Classifier-Free Guidance](./papers/image-generation/69-classifier-free-guidance/summary.md) (2021) - 🔥 CRITICAL - The "CFG scale" slider - Two predictions, one difference vector - Also how negative prompts work - [Paper](https://arxiv.org/abs/2207.12598)	[DDIM](./papers/image-generation/70-ddim/summary.md) (2020) - 🔥 CRITICAL - 1,000 steps down to 20 - Deterministic sampling, no retraining - Basis of diffusion image editing - [Paper](https://arxiv.org/abs/2010.02502)	[ControlNet](./papers/image-generation/71-controlnet/summary.md) (2023) - 🔥 CRITICAL - Edges, depth, pose control - Zero convolutions, base model frozen - ICCV 2023 best paper - [Paper](https://arxiv.org/abs/2302.05543)
[Flow Matching / SD3](./papers/image-generation/72-flow-matching-sd3/summary.md) (2022-2024) - 🔥 CRITICAL - What replaced diffusion - Straight noise-to-image paths - Powers SD3, SD3.5 and Flux - [Paper](https://arxiv.org/abs/2403.03206)	[VQ-VAE](./papers/image-generation/89-vq-vae/summary.md) (2017) - ⭐ HIGH - Discrete codebook latents - Straight-through estimator for the argmin - Made images tokenizable like text - [Paper](https://arxiv.org/abs/1711.00937)	[VQ-GAN](./papers/image-generation/90-vq-gan/summary.md) (2020) - ⭐ HIGH - VQ-VAE latents plus an adversarial loss - Transformers autoregress over image tokens - The compress-then-model recipe - [Paper](https://arxiv.org/abs/2012.09841)
[Imagen](./papers/image-generation/91-imagen/summary.md) (2022) - ⭐ HIGH - A frozen T5 encoder beat bigger image models - Text understanding was the real bottleneck - Cascaded super-resolution diffusion - [Paper](https://arxiv.org/abs/2205.11487)	[DreamBooth](./papers/image-generation/92-dreambooth/summary.md) (2022) - 🔥 CRITICAL - Your subject, any scene, from 3-5 images - Rare-token binding plus prior-preservation loss - The blueprint every personalization method copies - [Paper](https://arxiv.org/abs/2208.12242)

[NeRF](./papers/image-generation/117-nerf/summary.md) (2020) - 🔥 CRITICAL - A 3D scene stored in a neural network - Photoreal novel views from photos - Volume rendering through an MLP - [Paper](https://arxiv.org/abs/2003.08934)	[3D Gaussian Splatting](./papers/image-generation/118-3d-gaussian-splatting/summary.md) (2023) - 🔥 CRITICAL - Real-time photoreal 3D - Millions of 3D Gaussians, rasterised fast - NeRF quality at real-time speed - [Paper](https://arxiv.org/abs/2308.04079)	[DALL-E 2 (unCLIP)](./papers/image-generation/119-dalle2-unclip/summary.md) (2022) - ⭐ HIGH - CLIP latents to images - Prior maps text to CLIP image embedding - Diffusion decoder paints the image - [Paper](https://arxiv.org/abs/2204.06125)
[Consistency Models](./papers/image-generation/120-consistency-models/summary.md) (2023) - ⭐ HIGH - Diffusion quality in one step - Map any noisy point straight to the output - Distilled or trained from scratch - [Paper](https://arxiv.org/abs/2303.01469)

[CLIP](./papers/multimodal/08-clip/summary.md) (2021) - ⭐ HIGH - Vision-language bridge - Vision-language contrastive learning - Zero-shot image classification - Powers text-to-image models - [Paper](https://arxiv.org/abs/2103.00020)	[GPT-4V(ision)](./papers/multimodal/23-gpt4v/summary.md) (2023) - 🔥 CRITICAL - Multimodal frontier model - GPT-4 with vision capabilities - State-of-the-art VQA and OCR - Real-world applications - [Paper](https://cdn.openai.com/papers/GPTV_System_Card.pdf)	[Gemini 2.5](./papers/multimodal/29-gemini-2.5/summary.md) (2025) - 🔥 CRITICAL - Most advanced multimodal AI - Native multimodal (text, image, audio, video) - 1M context, 3-hour video understanding - Integrated thinking mode - [Paper](https://arxiv.org/abs/2507.06261)
[SAM 2](./papers/multimodal/32-sam2/summary.md) (2024) - ⭐ HIGH - Universal video segmentation - 44 FPS real-time performance - Zero-shot generalization - Segment anything in video - [Paper](https://arxiv.org/abs/2408.00714)	[LLaVA](./papers/multimodal/46-llava/summary.md) (2023) - 🔥 CRITICAL - Vision encoder + projection + LLM - Instruction data generated by text-only GPT-4 - Democratized open vision-language models - [Paper](https://arxiv.org/abs/2304.08485)	[Gemini 3](./papers/multimodal/47-gemini3/summary.md) (2025) - ⭐ HIGH - First model past 1500 LMArena Elo - 95% AIME 2025, 91.8% MMLU - Deep Think mode for the hardest reasoning - [Paper](https://blog.google/products-and-platforms/products/gemini/gemini-3/)
[Whisper](./papers/multimodal/49-whisper/summary.md) (2022) - 🔥 CRITICAL - 680,000 hours of weakly supervised audio - 99 languages, zero-shot, no fine-tuning - The default open speech recognition model - [Paper](https://arxiv.org/abs/2212.04356)