| Finite Horizon Problems |
| Lecture 1 (PDF) |
- Introduction to Dynamic Programming
- Examples of Dynamic Programming
- Significance of Feedback
|
| Lecture 2 (PDF) |
- The Basic Problem
- Principle of Optimality
- The General Dynamic Programming Algorithm
- State Augmentation
|
| Lecture 3 (PDF) |
- Deterministic Finite-State Problem
- Backward Shortest Path Algorithm
- Forward Shortest Path Algorithm
- Alternative Shortest Path Algorithms
|
| Lecture 4 (PDF) |
- Examples of Stochastic Dynamic Programming Problems
- Linear-Quadratic Problems
- Inventory Control
|
| Lecture 5 (PDF) |
- Stopping Problems
- Scheduling Problems
- Minimax Control
|
| Lecture 6 (PDF) |
- Problems with Imperfect State Info
- Reduction to the Perfect State Info Cas
- Linear Quadratic Problems
- Separation of Estimation and Control
|
| Lecture 7 (PDF) |
- Imperfect State Information
- Sufficient Statistics
- Conditional State Distribution as a Sufficient Statistic
- Finite-State Analysis
|
| Lecture 8 (PDF) |
- Suboptimal Control
- Cost Approximation Methods: Classification
- Certainty Equivalent Control
- Limited Lookahead Policies
- Performance Bounds
- Problem Approximation Approach
- Parametric Cost-To-Go Approximation
|
| Lecture 9 (PDF) |
- Rollout Algorithms
- Cost Improvement Property
- Discrete Deterministic Problems
- Approximations to Rollout Algorithms
- Model Predictive Control (MPS)
- Discretization of Continuous Time
- Discretization of Continuous Space
- Other Suboptimal Approaches
|
| Simple Infinite Horizon Problems |
| Lecture 10 (PDF) |
- Infinite Horizon Problems
- Stochastic Shortest Path (SSP) Problems
- Bellman's Equation
- Dynamic Programming – Value Iteration
- Discounted Problems as a Special Case of SSP
|
| Lecture 11 (PDF) |
- Review of Stochastic Shortest Path Problems
- Computation Methods for SSP
- Computational Methods for Discounted Problems
|
| Lecture 12 (PDF) |
- Average Cost Per Stage Problems
- Connection With Stochastic Shortest Path Problems
- Bellman's Equation
- Value Iteration, Policy Iteration
|
| Lecture 13 (PDF) |
- Control of Continuous-Time Markov Chains: Semi-Markov Problems
- Problem Formulation: Equivalence to Discrete-Time Problems
- Discounted Problems
- Average Cost Problems
|
| Advanced Infinite Horizon Problems |
| Lecture 14 (PDF) | - Introduction to Advanced Infinite Horizon Dynamic Programming and Approximation Methods
|
| Lecture 15 (PDF) |
- Review of Basic Theory of Discounted Problems
- Monotonicity of Contraction Properties
- Contraction Mappings in Dynamic Programming
- Discounted Problems: Countable State Space with Unbounded Costs
- Generalized Discounted Dynamic Programming
- An Introduction to Abstract Dynamic Programming
|
| Lecture 16 (PDF) |
- Review of Computational Theory of Discounted Problems
- Value Iteration (VI)
- Policy Iteration (PI)
- Optimistic PI
- Computational Methods for Generalized Discounted Dynamic Programming
- Asynchronous Algorithms
|
| Lecture 17 (PDF) |
- Undiscounted Problems
- Stochastic Shortest Path Problems
- Proper and Improper Policies
- Analysis and Computational Methods for SSP
- Pathologies of SSP
- SSP Under Weak Conditions
|
| Lecture 18 (PDF) |
- Undiscounted Total Cost Problems
- Positive and Negative Cost Problems
- Deterministic Optimal Cost Problems
- Adaptive (Linear Quadratic) Dynamic Programming
- Affine Monotomic and Risk Sensitive Problems
|
| Lecture 19 (PDF) |
- Introduction to approximate Dynamic Programming
- Approximation in Policy Space
- Approximation in Value Space, Rollout / Simulation-based Single Policy Iteration
- Approximation in Value Space Using Problem Approximation
|
| Lecture 20 (PDF) |
- Discounted Problems
- Approximate (fitted) VI
- Approximate PI
- The Projected Equation
- Contraction Properties: Error Bounds
- Matrix Form of the Projected Equation
- Simulation-based Implementation
- LSTD, LSPE, and TD Methods
|
| Lecture 21 (PDF) |
- Review of Approximate Policy Iteration
- Projected Equation Methods for Policy Evaluation
- Simulation-Based Implementation Issues, Multistep Projected Equation Methods
- Bias-Variance Tradeoff
- Exploration-Enhanced Implementations, Oscillations
|
| Lecture 22 (PDF) |
- Aggregation as an Approximation Methodology
- Aggregate Problem
- Simulation-based Aggregation
- Q-Learning
|
| Lecture 23 (PDF) |
- Additional Topics in Advanced Dynamic Programming
- Stochastic Shortest Path Problems
- Average Cost Problems
- Generalizations
- Basis Function Adaptation
- Gradient-based Approximation in Policy Space
- An Overview
|
