CS980: Advanced ML: Reinforcement Learning

Please see the class homepage for schedule and other up-to-date information.

Where

Kingsbury N233

When

MW 3:40pm - 5:00pm

What

In this seminar, we will cover reinforcement learning, or how to make good decisions driven by data. The goal in reinforcement learning is to learn how to act while interacting with a dynamic and complex environment. Reinforcement learning methods can be applied in various domains, such as when managing ecosystems, optimizing website, to robotics, and healthcare.

Our focus will be on reinforcement learning that can learn from batch data sets without interacting with the environment. The algorithms have to learn how to interact with the environment based on a historical data. These batch methods are important when the cost of failure is high, such as in healthcare of agriculture, when using trial and error is impractical.

Some of the topics that we will cover are:

  • Markov decision processes: Policy iteration, Value iteration, Linear programming
  • Policy evaluation (online and offline): TD, LSTD
  • Batch policy improvement: Q-learning, LSPI
  • Convex optimization: Linear programming

The focus of the class will be on depth rather than breadth. The main goal of the class is delivering an interesting reinforcement learning group project. As much as possible we will work on the same code-base using a subset of C++, Python, and R. The class will require independent study of reading materials and in-class group problem solving and paper discussions.

Textbooks

Follow the links for free online versions.

Our main textbook will be:

Additional material can be found in:

  • Puterman, M. L. (2005). Markov decision processes: Discrete stochastic dynamic programming. John Wiley & Sons, Inc.
  • Sutton, R. S., & Barto, A. (1998). Reinforcement learning. MIT Press.

Pre-requisites

Statistics, some linear algebra and ideally familiarity with machine learning.

More details