What we're reading

Yao et al.

Showed that interleaving reasoning traces with actions lets language models solve complex tasks by thinking and acting in alternation. ReAct is the conceptual foundation for most modern AI agent architectures — reason about what to do, then do it, then reason again.

Schick et al.

Demonstrated that language models can learn to use external tools (calculators, search engines, APIs) through self-supervised learning. Established that tool use is a learnable skill, not just a prompting trick — a key insight for building capable AI agents.

Touvron et al.

Meta's release of high-quality open-weight models with permissive licensing catalyzed the open-source AI ecosystem. Llama 2 proved that open models could approach proprietary performance, launching a wave of community fine-tuning and derivative models.

Jiang et al.

Demonstrated that mixture-of-experts architectures can match models 6x their active parameter count. By activating only a subset of parameters per token, MoE models achieve large-model quality at small-model inference cost — a key efficiency breakthrough.

DeepSeek AI

Trained for an estimated $6 million, DeepSeek-R1 matched OpenAI o1's reasoning capabilities and was released under the MIT license. Validated that frontier-level reasoning can be achieved through RL without expensive supervised fine-tuning, fundamentally altering the economics of AI development.

Qwen Team

Alibaba's Qwen2.5 series demonstrated that open-source models trained on 18 trillion tokens across 29 languages could match or exceed proprietary models on coding, math, and reasoning benchmarks. The subsequent Qwen3 variants outperformed OpenAI O3 on advanced mathematics.

Gemini Team, Google

Google's natively multimodal model family demonstrated that training on interleaved text, image, audio, and video from the start produces stronger cross-modal reasoning than bolting modalities onto a text model. Set new benchmarks for multimodal understanding.

Anthropic

Anthropic's detailed system card for Claude 3.5 set a new standard for AI transparency, documenting model capabilities, safety evaluations, and known limitations. Demonstrated how responsible AI development can coexist with frontier capabilities.

Vaswani et al.

Introduced the Transformer architecture, replacing recurrence with self-attention for sequence modeling. This paper is the foundation of every modern large language model — GPT, BERT, Llama, Claude, and Gemini all descend from this architecture.

Devlin et al.

Demonstrated that pre-training a bidirectional transformer on unlabeled text, then fine-tuning on specific tasks, dramatically outperforms training from scratch. Established the pre-train/fine-tune paradigm that defines modern NLP.

Brown et al.

Showed that scaling language models to 175 billion parameters enables few-shot learning — performing tasks from just a few examples without fine-tuning. Proved that scale itself is a path to general capability.

Ouyang et al.

Introduced RLHF (Reinforcement Learning from Human Feedback) to align language models with human intent. This technique transformed raw language models into useful assistants — the key innovation behind ChatGPT and every instruction-tuned model since.

Kaplan et al.

Established precise mathematical relationships between model size, dataset size, compute budget, and performance. These scaling laws became the strategic blueprint for training larger and more capable models — directly informing investment decisions across the industry.

Wei et al.

Demonstrated that prompting models to show their reasoning step-by-step dramatically improves performance on math, logic, and multi-step tasks. Chain-of-thought is now a standard technique in both prompting and model training.

Bai et al.

Introduced a method for training AI systems to be helpful and harmless using a set of principles (a 'constitution') rather than extensive human labeling. Pioneered AI-to-AI feedback for alignment, reducing dependence on human annotation.

Lewis et al.

Combined retrieval systems with generative models, allowing language models to access external knowledge at inference time. RAG is now the standard architecture for building AI systems that need to work with specific, up-to-date, or proprietary information.