Fast model identification via physics engines for data-efficient policy search

Shaojun Zhu, Andrew Kimmel, Kostas Bekris, Abdeslam Boularias

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

This paper presents a method for identifying mechanical parameters of robots or objects, such as their mass and friction coefficients. Key features are the use of off-the-shelf physics engines and the adaptation of a Bayesian optimization technique towards minimizing the number of real-world experiments needed for model-based reinforcement learning. The proposed framework reproduces in a physics engine experiments performed on a real robot and optimizes the model's mechanical parameters so as to match real-world trajectories. The optimized model is then used for learning a policy in simulation, before real-world deployment. It is well understood, however, that it is hard to exactly reproduce real trajectories in simulation. Moreover, a near-optimal policy can be frequently found with an imperfect model. Therefore, this work proposes a strategy for identifying a model that is just good enough to approximate the value of a locally optimal policy with a certain confidence, instead of wasting effort on identifying the most accurate model. Evaluations, performed both in simulation and on a real robotic manipulation task, indicate that the proposed strategy results in an overall time-efficient, integrated model identification and learning solution, which significantly improves the data-efficiency of existing policy search algorithms.

Original languageEnglish (US)
Title of host publicationProceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018
EditorsJerome Lang
PublisherInternational Joint Conferences on Artificial Intelligence
Pages3249-3256
Number of pages8
ISBN (Electronic)9780999241127
StatePublished - Jan 1 2018
Event27th International Joint Conference on Artificial Intelligence, IJCAI 2018 - Stockholm, Sweden
Duration: Jul 13 2018Jul 19 2018

Publication series

NameIJCAI International Joint Conference on Artificial Intelligence
Volume2018-July
ISSN (Print)1045-0823

Other

Other27th International Joint Conference on Artificial Intelligence, IJCAI 2018
CountrySweden
CityStockholm
Period7/13/187/19/18

Fingerprint

Identification (control systems)
Physics
Engines
Trajectories
Robots
Reinforcement learning
Robotics
Experiments
Friction

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

Cite this

Zhu, S., Kimmel, A., Bekris, K., & Boularias, A. (2018). Fast model identification via physics engines for data-efficient policy search. In J. Lang (Ed.), Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018 (pp. 3249-3256). (IJCAI International Joint Conference on Artificial Intelligence; Vol. 2018-July). International Joint Conferences on Artificial Intelligence.
Zhu, Shaojun ; Kimmel, Andrew ; Bekris, Kostas ; Boularias, Abdeslam. / Fast model identification via physics engines for data-efficient policy search. Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018. editor / Jerome Lang. International Joint Conferences on Artificial Intelligence, 2018. pp. 3249-3256 (IJCAI International Joint Conference on Artificial Intelligence).
@inproceedings{871b62f0199c41959d43dfab5931b091,
title = "Fast model identification via physics engines for data-efficient policy search",
abstract = "This paper presents a method for identifying mechanical parameters of robots or objects, such as their mass and friction coefficients. Key features are the use of off-the-shelf physics engines and the adaptation of a Bayesian optimization technique towards minimizing the number of real-world experiments needed for model-based reinforcement learning. The proposed framework reproduces in a physics engine experiments performed on a real robot and optimizes the model's mechanical parameters so as to match real-world trajectories. The optimized model is then used for learning a policy in simulation, before real-world deployment. It is well understood, however, that it is hard to exactly reproduce real trajectories in simulation. Moreover, a near-optimal policy can be frequently found with an imperfect model. Therefore, this work proposes a strategy for identifying a model that is just good enough to approximate the value of a locally optimal policy with a certain confidence, instead of wasting effort on identifying the most accurate model. Evaluations, performed both in simulation and on a real robotic manipulation task, indicate that the proposed strategy results in an overall time-efficient, integrated model identification and learning solution, which significantly improves the data-efficiency of existing policy search algorithms.",
author = "Shaojun Zhu and Andrew Kimmel and Kostas Bekris and Abdeslam Boularias",
year = "2018",
month = "1",
day = "1",
language = "English (US)",
series = "IJCAI International Joint Conference on Artificial Intelligence",
publisher = "International Joint Conferences on Artificial Intelligence",
pages = "3249--3256",
editor = "Jerome Lang",
booktitle = "Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018",

}

Zhu, S, Kimmel, A, Bekris, K & Boularias, A 2018, Fast model identification via physics engines for data-efficient policy search. in J Lang (ed.), Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018. IJCAI International Joint Conference on Artificial Intelligence, vol. 2018-July, International Joint Conferences on Artificial Intelligence, pp. 3249-3256, 27th International Joint Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, 7/13/18.

Fast model identification via physics engines for data-efficient policy search. / Zhu, Shaojun; Kimmel, Andrew; Bekris, Kostas; Boularias, Abdeslam.

Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018. ed. / Jerome Lang. International Joint Conferences on Artificial Intelligence, 2018. p. 3249-3256 (IJCAI International Joint Conference on Artificial Intelligence; Vol. 2018-July).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Fast model identification via physics engines for data-efficient policy search

AU - Zhu, Shaojun

AU - Kimmel, Andrew

AU - Bekris, Kostas

AU - Boularias, Abdeslam

PY - 2018/1/1

Y1 - 2018/1/1

N2 - This paper presents a method for identifying mechanical parameters of robots or objects, such as their mass and friction coefficients. Key features are the use of off-the-shelf physics engines and the adaptation of a Bayesian optimization technique towards minimizing the number of real-world experiments needed for model-based reinforcement learning. The proposed framework reproduces in a physics engine experiments performed on a real robot and optimizes the model's mechanical parameters so as to match real-world trajectories. The optimized model is then used for learning a policy in simulation, before real-world deployment. It is well understood, however, that it is hard to exactly reproduce real trajectories in simulation. Moreover, a near-optimal policy can be frequently found with an imperfect model. Therefore, this work proposes a strategy for identifying a model that is just good enough to approximate the value of a locally optimal policy with a certain confidence, instead of wasting effort on identifying the most accurate model. Evaluations, performed both in simulation and on a real robotic manipulation task, indicate that the proposed strategy results in an overall time-efficient, integrated model identification and learning solution, which significantly improves the data-efficiency of existing policy search algorithms.

AB - This paper presents a method for identifying mechanical parameters of robots or objects, such as their mass and friction coefficients. Key features are the use of off-the-shelf physics engines and the adaptation of a Bayesian optimization technique towards minimizing the number of real-world experiments needed for model-based reinforcement learning. The proposed framework reproduces in a physics engine experiments performed on a real robot and optimizes the model's mechanical parameters so as to match real-world trajectories. The optimized model is then used for learning a policy in simulation, before real-world deployment. It is well understood, however, that it is hard to exactly reproduce real trajectories in simulation. Moreover, a near-optimal policy can be frequently found with an imperfect model. Therefore, this work proposes a strategy for identifying a model that is just good enough to approximate the value of a locally optimal policy with a certain confidence, instead of wasting effort on identifying the most accurate model. Evaluations, performed both in simulation and on a real robotic manipulation task, indicate that the proposed strategy results in an overall time-efficient, integrated model identification and learning solution, which significantly improves the data-efficiency of existing policy search algorithms.

UR - http://www.scopus.com/inward/record.url?scp=85055699217&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055699217&partnerID=8YFLogxK

M3 - Conference contribution

T3 - IJCAI International Joint Conference on Artificial Intelligence

SP - 3249

EP - 3256

BT - Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018

A2 - Lang, Jerome

PB - International Joint Conferences on Artificial Intelligence

ER -

Zhu S, Kimmel A, Bekris K, Boularias A. Fast model identification via physics engines for data-efficient policy search. In Lang J, editor, Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018. International Joint Conferences on Artificial Intelligence. 2018. p. 3249-3256. (IJCAI International Joint Conference on Artificial Intelligence).