Lecture Notes | Convex Analysis and Optimization | Electrical Engineering and Computer Science

This section contains lecture notes and some associated readings.

Lecture notes files.
TOPICS	LECTURE NOTES	READINGS
The role of convexity in optimization Duality theory Algorithms and duality	Lecture 1 (PDF - 1.2MB)
Convex sets and functions Epigraphs Closed convex functions Recognizing convex functions	Lecture 2 (PDF)	Section 1.1
Differentiable convex functions Convex and affine hulls Caratheodory's theorem	Lecture 3 (PDF)	Sections 1.1, 1.2
Relative interior and closure Algebra of relative interiors and closures Continuity of convex functions Closures of functions	Lecture 4 (PDF)	Section 1.3
Recession cones and lineality space Directions of recession of convex functions Local and global minima Existence of optimal solutions	Lecture 5 (PDF - 1.0MB)	Sections 1.4, 3.1, 3.2
Nonemptiness of closed set intersections Existence of optimal solutions Preservation of closure under linear transformation Hyperplanes	Lecture 6 (PDF - 1.4MB)
Review of hyperplane separation Nonvertical hyperplanes Convex conjugate functions Conjugacy theorem Examples	Lecture 7 (PDF)	Sections 1.5, 1.6
Review of conjugate convex functions Min common / max crossing duality Weak duality Special cases	Lecture 8 (PDF - 1.2MB)	Sections 1.6, 4.1, 4.2
Minimax problems and zero-sum games Min common / max crossing duality for minimax and zero-sum games Min common / max crossing duality theorems Strong duality conditions Existence of dual optimal solutions	Lecture 9 (PDF)	Sections 3.4, 4.3, 4.4, 5.1
Min common / max crossing Theorem III Nonlinear Farkas' lemma / linear constraints Linear programming duality Convex programming duality Optimality conditions	Lecture 10 (PDF)	Sections 4.5, 5.1, 5.2, 5.3.1–5.3.2
Review of convex programming duality / counterexamples Fenchel duality Conic duality	Lecture 11 (PDF)	Sections 5.3.1–5.3.6
Subgradients Fenchel inequality Sensitivity in constrained optimization Subdifferential calculus Optimality conditions	Lecture 12 (PDF)	Section 5.4
Problem structure Conic programming	Lecture 13 (PDF)
Conic programming Semidefinite programming Exact penalty functions Descent methods for convex optimization Steepest descent method	Lecture 14 (PDF)	Chapter 6: Convex Optimization Algorithms (PDF)
Subgradient methods Calculation of subgradients Convergence	Lecture 15 (PDF)
Approximate subgradient methods Approximation methods Cutting plane methods	Lecture 16 (PDF)
Review of cutting plane method Simplicial decomposition Duality between cutting plane and simplicial decomposition	Lecture 17 (PDF)
Generalized polyhedral approximation methods Combined cutting plane and simplicial decomposition methods	Lecture 18 (PDF)	Bertsekas, Dimitri, and Huizhen Yu. "A Unifying Polyhedral Approximation Framework for Convex Optimization." SIAM Journal on Optimization 21, no. 1 (2011): 333–60.
Proximal minimization algorithm Extensions	Lecture 19 (PDF)
Proximal methods Review of proximal minimization Proximal cutting plane algorithm Bundle methods Augmented Lagrangian methods Dual proximal minimization algorithm	Lecture 20 (PDF - 1.1MB)
Generalized forms of the proximal point algorithm Interior point methods Constrained optimization case: barrier method Conic programming cases	Lecture 21 (PDF)
Incremental methods Review of large sum problems Review of incremental gradient and subgradient methods Combined incremental subgradient and proximal methods Convergence analysis Cyclic and randomized component selection	Lecture 22 (PDF)	Bertsekas, Dimitri. "Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey." (PDF) Laboratory for Information and Decision Systems Report LIDS-P-2848, MIT, August 2010. Sra, Suvrit, Sebastian Nowozin, and Stephen Wright, eds. Optimization for Machine Learning. MIT Press, 2011. ISBN: 9780262016469.
Review of subgradient methods Application to differentiable problems: gradient projection Iteration complexity issues Complexity of gradient projection Projection method with extrapolation Optimal algorithms	Lecture 23 (PDF)
Gradient proximal minimization method Nonquadratic proximal algorithms Entropy minimization algorithm Exponential augmented Lagrangian method Entropic descent algorithm	Lecture 24 (PDF)	Beck, Amir, and Marc Teboulle. "Gradient-Based Algorithms with Applications to Signal-Recovery Problems." In Convex Optimization in Signal Processing and Communications. Edited by Daniel Palomar and Yonina Eldar. Cambridge University Press, 2010. ISBN: 9780521762229. Beck, Amir, and Marc Teboulle. "Mirror Descent and Nonlinear Projected Subgradient Methods for Convex Optimization." Operations Research Letters 31, no. 3 (2003): 167–75. Bertsekas, Dimitri. Nonlinear Programming. Athena Scientific, 1999. ISBN: 9781886529007.
Convex analysis and duality Convex optimization algorithms	Lecture 25 (PDF - 2.0MB)