Reinforcement learning is from our perspective the automatic design of approximately optimal controllers from measurements. In traditional (optimal) control, the smart human in the loop decides how to measure and model the system. In RL, on the other hand, the optimal controller is constructed by the RL system directly from measurements; however, the way to the optimal controller can require extensive prestructuring through structured policies, value functions or models. In this page, I want to list some of the projects I am working on or have worked on but this list will always be fairly incomplete.

Our general goal in reinforcement learning is the development of methods which scale into the dimensionality of humanoid robots and can generate actions for seven or more degrees of freedom, e.g., for a human arm. Such problems are a tremendous challenge for reinforcement learning as they require a state space of 21 or more dimensions (one dimension for each joint position, velocity and acceleration) and an action space of seven dimensions.

While supervised statistical learning techniques have significant applications for model and imitation learning, they do not suffice for all motor learning problems, particularly when no expert teacher or idealized desired behavior is available. Thus, both robotics and the understanding of human motor control require reward (or cost) related self-improvement. The developement of efficient reinforcement learning methods is therefore essential for the success of learning in motor control.

However, reinforcement learning in high-dimensional spaces such as manipulator and humanoid robotics is extremely difficult as a complete exploration of the underlying state-action spaces is impossible and few existing techniques scale into this domain.

Nevertheless, it is obvious that humans also never need such an extensive exploration in order to learn new motor skills and instead rely upon a combination of both watching a teacher and subsequent self-improvement. In more technical terms: first, a control policy is obtained by imitation and then improved using reinforcement learning. It is essential that only local policy search techniques, e.g., policy gradient methods, are applied as a rapid change to the policy would result into a complete unlearning of the policy and might also result into an unstable control policies which can damage the robot.

In order to bring reinforcement learning to robotics and computational motor control, we have both improved existing reinforcement learning methods as well as developed a variety of novel algorithms. At this point, we can only give a short overview of these methods:

• Policy Gradient Methods: One class of methods which are particularly interesting, are policy gradient methods duer to their stronger guarantees. A nice tutorial to get started can be found in the Policy Gradient Toolbox which I created for an upcoming survey.
• Natural Actor-Critic: The natural actor-critic makes use of the fact, that a natural gradient usually beats a vanilla gradient. We have developed several versions and have realized that algorithms such as Sutton's Actor-Critic and Bradtke & Bartos' Q-Learning for the traditional problem of Linear Quadratic-Regulation can be derived from this setting.
• EM-like Reinforcement Learning: If we had a teacher labeling all actions as good or bad in a binary fashion, we would have an imitation learning problem. However, if we consider these labels as hidden variables and use the returns/action values as improper distributions over the labels, we obtain an inference problem. This problem has led to the reward-weighted regression and the PoWER algorithm.

Related Publications

Hachiya,H.; Akiyama, T.; Sugiyama, M.; Peters, J. (2009). Adaptive Importance Sampling for Value Function Approximation in Off-policy Reinforcement Learning, Neural Networks, 22, 10, pp.1399-1410.
[Keywords: off-policy reinforcement learning; value function approximation; policy iteration; adaptive importance sampling; importance-weighted cross-validation; efficient sample reuse]

Peters, J.; Kober, J.; Nguyen-Tuong, D. (2008). Policy Learning – a unified perspective with applications in robotics, Proceedings of the European Workshop on Reinforcement Learning (EWRL).
[Keywords: reinforcement learning, policy gradient, weighted regression] [PDF]

Peters, J.;Schaal, S. (2008). Natural actor critic, Neurocomputing, 71, 7-9, pp.1180-1190.
[Keywords: reinforcement learning, policy gradient, natural actor-critic, natural gradients] [PDF]

Peters, J.;Schaal, S. (2008). Learning to control in operational space, International Journal of Robotics Research, 27, pp.197-212.
[Keywords: operational space control, learning, EM ALGORITHM, redundancy resolution, reinforcement learning] [PDF]

Peters, J.;Schaal, S. (2008). Reinforcement learning of motor skills with policy gradients, Neural Networks, 21, 4, pp.682-97.
[Keywords: Reinforcement learning, Policy gradient methods, Natural gradients, Natural Actor-Critic, Motor skills, Motor primitives] [PDF]

Peters, J., Schaal, S. (2007). Policy Learning for Motor Skills, Proceedings of 14th International Conference on Neural Information Processing (ICONIP).
[Keywords: Machine Learning, Reinforcement Learning, Robotics, Motor Primitives, Policy Gradients, Natural Actor-Critic, Reward-Weighted Regression] [PDF]

Wierstra, D.; Foerster, A.; Peters, J.; Schmidhuber, J. (2007). Solving Deep Memory POMDPs with Recurrent Policy Gradients, Proceedings of the International Conference on Artificial Neural Networks (ICANN).
[Keywords: policy gradients, reinforcement learning] [PDF]

Theodorou, E; Peters, J; Schaal, S. (2007). Reinforcement Learning for Optimal Control of Arm Movements, Abstracts of the 37st Meeting of the Society of Neuroscience..
[Keywords: Optimal Control,Reinforcement Learning, Arm Movements]

Peters, J. (2007). Machine Learning of Motor Skills for Robotics, Ph.D. Thesis, Department of Computer Science, University of Southern California.
[Keywords: Machine Learning, Reinforcement Learning, Robotics, Motor Primitives, Policy Gradients, Natural Actor-Critic, Reward-Weighted Regression]

Peters, J.;Schaal, S. (2007). Reinforcement learning for operational space control, International Conference on Robotics and Automation (ICRA2007), pp.2111-2116.
[Keywords: operational space control, reinforcement learning, weighted regression, EM-Algorithm] [PDF]

Peters, J.;Schaal, S. (2007). Using reward-weighted regression for reinforcement learning of task space control, Proceedings of the 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning.
[Keywords: reinforcement learning, cart-pole, policy gradient methods] [PDF]

Peters, J.;Schaal, S. (2007). Applying the episodic natural actor-critic architecture to motor primitive learning, Proceedings of the 2007 European Symposium on Artificial Neural Networks (ESANN).
[Keywords: reinforcement learning, policy gradient methods, motor primitives, natural actor-critic] [PDF]

Peters, J.;Schaal, S. (2007). Reinforcement learning by reward-weighted regression for operational space control, Proceedings of the International Conference on Machine Learning (ICML2007).
[Keywords: reinforcement learning, operational space control, weighted regression] [PDF]

Peters, J.;Theodorou, E.;Schaal, S. (2007). Policy gradient methods for machine learning, INFORMS Conference of the Applied Probability Society.
[Keywords: policy gradient methods, reinforcement learning, simulation-optimization]

Riedmiller, M.;Peters, J.;Schaal, S. (2007). Evaluation of policy gradient methods and variants on the cart-pole benchmark, Proceedings of the 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning.
[Keywords: reinforcement learning, cart-pole, policy gradient methods] [PDF]

Peters, J.;Schaal, S. (2006). Reinforcement Learning for Parameterized Motor Primitives, Proceedings of the 2006 International Joint Conference on Neural Networks (IJCNN 2006).
[Keywords: motor primitives, reinforcement learning] [PDF]

Peters, J.;Schaal, S. (2006). Policy gradient methods for robotics, Proceedings of the IEEE International Conference on Intelligent Robotics Systems (IROS 2006).
[Keywords: policy gradient methods, reinforcement learning, robotics] [PDF]

Peters, J.;Vijayakumar, S.;Schaal, S. (2005). Natural Actor-Critic, in: Gama, J.;Camacho, R.;Brazdil, P.;Jorge, A.;Torgo, L. (eds.), Proceedings of the 16th European Conference on Machine Learning (ECML 2005), 3720, pp.280-291, Springer.
[Keywords: Reinforcement Learning, Policy Gradients, Natural Gradients] [PDF]

Peters, J.;Vijayakumar, S.;Schaal, S. (2003). Reinforcement learning for humanoid robotics, IEEE-RAS International Conference on Humanoid Robots (Humanoids2003).
[Keywords: reinforcement learning, policy gradients, movement primitives, behaviors, dynamic systems, humanoid robotics] [PDF]

Peters, J.;Vijayakumar, S.;Schaal, S. (2003). Scaling reinforcement learning paradigms for motor learning, Proceedings of the 10th Joint Symposium on Neural Computation (JSNC 2003).
[Keywords: Reinforcement learning, neurodynamic programming, actorcritic methods, policy gradient methods, natural policy gradient] [PDF]

Kwee, I.;Hutter, M.;Schmidhuber, J. (2001). Gradient-based reinforcement planning in policy-search methods, IDSIA.
[Keywords: reinforcement learning policy gradients model-based] [PDF]