We consider the task of reinforcement learning with linear value function approximation. Temporal-difference algorithms, and in particular the Least Squares Temporal-Difference (LSTD) algorithm, provide a method to learn the parameters of the value function, but when the number of features is large this algorithm can over-fit to the data and is computationally expensive. In this paper, we propose a regularization framework for the LSTD algorithm that overcomes these difficulties. In particular, we focus on the case of L_1 regularization, which is robust to irrelevant features and also serves as method for feature selection. Although the L_1 regularized LSTD solution cannot be expressed as as a convex optimization problem, we present an algorithm similar to the Least Angle Regression (LARS) algorithm that can efficiently compute the optimal solution. Finally, we demonstrate the performance of the algorithm experimentally.