An Analysis of Linear Models, Linear Value-Function Approximation, and Feature Selection for Reinforcement Learning

Download PDF

Parr, Ronald and Li, Lihong and Taylor, Gavin and Painter-Wakefield, Christopher and Littman, Michael

We show that linear value function approximation is equivalent to a form of linear model approximation. We derive a relationship between the model approximation error and the Bellman error, and show how this relationship can guide feature selection for model improvement and/or value function improvement. We also show how these results give insight into the behavior of existing feature-selection algorithms.