CS229 2018 Lecture Notes
Machine Learning — Stanford University, Autumn 2018
CS229 2018 Lecture Notes
CS229 lecture notes · Stanford · Autumn 2018 · Unofficial
Andrew Ng’s CS229,
written up from the board.
Linear regression to reinforcement learning — the derivations worked at the blackboard, the intuition given out loud, and a timestamp into the 2018 recording for every section.
Each lecture is a standalone page written to be read straight through. The recording stays one click away: the chapter links its video under the title, and every section carries a timestamp that opens the lecture at the moment the idea is introduced, so you can watch the derivation being built when the prose is not enough. Sections, equations and figures are numbered and individually linkable.
Lectures
The course runs to twenty lectures and one discussion section, and every one has a chapter; the roadmap groups them by theme.
Lecture 1
What machine learning is, the five areas the course covers, and the board pictures every later lecture is drawn on.
Complete
Lecture 2
Linear Regression and Gradient Descent
Fitting a hypothesis by least squares: batch and stochastic gradient descent, and the normal equations that skip iteration altogether.
Complete
Lecture 3
Locally Weighted and Logistic Regression
Least squares as maximum likelihood, a regression that refits around every query, and the logistic model for classification with Newton’s method.
Complete
Lecture 4
Perceptron and Generalized Linear Models
The perceptron, the exponential family, and the recipe that turns a choice of distribution into a learning algorithm, softmax regression included.
Complete
Lecture 5
Modelling how each class generates its inputs instead of the boundary between them: Gaussian discriminant analysis and naive Bayes.
Complete
Lecture 6
Laplace smoothing and the event models for text, then the geometric margin and the optimisation problem that maximises it.
Complete
Lecture 7
The dual of the margin problem, inner products in feature spaces you never compute, and the soft margin for data that will not separate.
Complete
Lecture 8
Data Splits, Models and Cross-Validation
Bias and variance, regularisation, and how to choose a model with data you did not train on.
Complete
Discussion section
Why training error says anything about test error: the union bound, Hoeffding’s inequality, uniform convergence and the VC dimension.
Complete
Lecture 9
Decision Trees and Ensemble Methods
Greedy recursive splits, the losses that choose them, and the bagging, random forests and boosting that make trees competitive.
Complete
Lecture 10
Introduction to Neural Networks
Logistic regression as a single neuron, and what stacking them into layers buys.
Complete
Lecture 11
Backprop and Improving Neural Networks
Backpropagation worked out by hand, then initialisation, normalisation and the optimisers that make training work in practice.
Complete
Lecture 12
Debugging ML Models and Error Analysis
Diagnosing a learning algorithm that does not work: bias versus variance, optimisation versus objective, and error and ablative analysis.
Complete
Lecture 13
Expectation-Maximization Algorithms
Clustering with k-means, mixtures of Gaussians, and the EM algorithm that fits a model with latent variables.
Complete
Lecture 14
EM Algorithm and Factor Analysis
EM’s concrete updates and its view as coordinate ascent, and factor analysis for few examples in many dimensions.
Complete
Lecture 15
Projecting data onto the directions of greatest variance, when that helps and when it does not, and the cocktail-party problem ICA solves.
Complete
Lecture 16
Independent Component Analysis and RL
Finishing ICA by maximum likelihood, then Markov decision processes and the Bellman equations.
Complete
Lecture 17
MDPs and Value/Policy Iteration
Solving an MDP by value iteration and policy iteration, and learning its model from experience.
Complete
Lecture 18
Continuous State MDPs and Model Simulation
Discretization and its limits, learning a simulator from flight logs, and fitted value iteration for continuous states.
Complete
Lecture 19
Finite-Horizon MDPs and the Linear Quadratic Regulator
State-action rewards, finite horizons and non-stationary policies, linearization, and the linear quadratic regulator.
Complete
Lecture 20
RL Debugging and Direct Policy Search
The helicopter diagnostics in RL notation, stochastic policies, REINFORCE and its variance, partial observability.
Complete
What these notes are
The official CS229 notes are the formal account of the course, and a good one. What they cannot carry is the lecture itself: the order in which Andrew Ng motivates an idea before he formalises it, the picture he draws to make an update rule obvious, the question from the room that exposes the step everyone was about to skip.
These notes are written from the Autumn 2018 recordings rather than from the official notes, and they are meant to sit beside both. Each page aims to be:
- Faithful to the lecture — it follows what was taught and why, and links each section to the moment in the video where it happens.
- Complete in its mathematics — the derivations done at the board are written out in full, with every symbol defined.
- A bridge, not a copy — where the official notes go further, the chapter says which section to read next rather than reproducing it.
The whole book is also a single PDF. For an AI assistant, llms.txt maps every chapter to a Markdown copy with the maths in TeX, and llms-full.txt holds the book in one file. The same treatment of Stanford’s deep learning for computer vision course is in the companion CS231n 2025 Lecture Notes.
Roadmap
- Supervised learning — linear and logistic regression, generalized linear models, generative learning, support vector machines and kernels
- Generalization — bias and variance, model selection, learning theory
- Trees and neural networks — decision trees and ensembles, neural networks and backpropagation
- Applying machine learning — debugging learning algorithms and error analysis
- Unsupervised learning — k-means, EM, factor analysis, PCA and ICA
- Reinforcement learning — MDPs, value and policy iteration, continuous states, LQR
Contributing
Found an error? Corrections are genuinely welcome — there is an Edit this page link at the bottom of every lecture, which opens a pull request in your browser. For anything larger, open an issue.
Disclaimer
This is an unofficial resource, not affiliated with Stanford University. For official course materials see cs229.stanford.edu and the 2018 lecture videos.
The notes are released under the MIT licence, which covers the writing and the diagrams made for this project. Any figure reproduced from elsewhere remains the property of its original authors and is credited in its caption — see NOTICE.