Planner Arena is a site for benchmarking sampling-based planners. The site is set up to show the performance of implementations of various sampling-based planning algorithms in the Open Motion Planning Library (OMPL). We have chosen a few benchmark problems that highlight some interesting aspects of motion planning.
Planner Arena is also a site you can use to analyze your own motion planning benchmark data. The easiest way to do so is to use either the ompl_benchmark program in OMPL.app or the Benchmark class in your own code. See the relevant documentation on the OMPL site. The log files that are produced by the OMPL benchmarking facilities get turned into a SQLite database using a script. The database schema is described on this page as well. This means that you could produce benchmark databases with some other software for entirely different planning algorithms (or different implementations of algorithms in OMPL) and use Planner Arena to visualize the data. Much of the Planner Arena user interface is dynamically constructed based on the contents of the benchmark database. In particular, if you store different types of performance measures in your tables, Planner Arena will still be able to plot the results.
If you use Planner Arena or the OMPL benchmarking facilities, then we kindly ask you to include the following citation in your publications:
@article{moll2015benchmarking-motion-planning-algorithms,
Author = {Mark Moll and Ioan A. {\c{S}}ucan and Lydia E. Kavraki},
Doi = {10.1109/MRA.2015.2448276},
Journal = {{IEEE} Robotics \& Automation Magazine},
Month = {September},
Number = {3},
Pages = {96--102},
Title = {Benchmarking Motion Planning Algorithms: An Extensible Infrastructure for Analysis and Visualization},
Volume = {22},
Year = {2015}
}
The default database used by the Planner Arena server contains results for a number of sample benchmarks described below. Most of them were produced by running the ompl_benchmark tool on the following configuration files included with the OMPL.app distribution:
cubicles.cfg: A fairly straightforward 3D rigid body planning problem. There are a few path homotopy classes. The path is a little convoluted, since it has go through the whole environment. A large part of the “basement” is not connected to the rest of the environment. A sample solution is shown in the first movie below.cubicles_opt.cfg: The same problem, but configured to be solved with a number of optimizing planners. The planners are given more time than in the previous benchmark so that we can compare convergence rates in the progress plots.Abstract.cfg: This is a more challenging 3D rigid body planning problem with several narrow passages, and several homotopy classes.Home.cfg: This is also a challenging 3D rigid body planning problem. There is a long path between start and goal that is relatively easy to find, but there also other, shorter paths that are much harder to find. This is therefore a good benchmark for optimizing planners.pipedream_ring.cfg: A 3D rigid body planning problem that contains one long, curvy narrow passage. Nevertheless, most planners can solve this problem within a few seconds.BugTrap_dcar.cfg: A challenging kinodynamic motion planning problem: a second-order car has to drive out of a “bug trap” obstacle.Maze_kcar.cfg: Another kinodynamic motion planning problem: a first-order car has to navigate through a maze. The dynamics are simpler than in the previous benchmark, but the obstacles are more complex.We have also included the results of a few benchmarks where the robot cannot be modeled by a rigid body. These benchmarks are included with OMPL and OMPL.app as demo programs. We have included the results:
demo_KinematicChainBenchmark 20): A kinematic chain with 20 degrees of freedom has to move out of a curved narrow passage by essentially folding up onto itself (while avoiding self-collisions!) before fully extending outside of the narrow passage. The implementation of this benchmark has improved significantly, so the benchmark times will not exactly match the graphs shown in this paper, where the benchmark is introduced.demo_AnytimePathShortening easy alternate 30 rrtconnect and demo_AnytimePathShortening easy none 30 rrtstar): This benchmark illustrates the benefit of Anytime Path Shortening, a generic wrapper around one or more geometric motion planners that repeatedly applies shortcutting and hybridization to a set of solution paths. As dimensionality of the configuration space increases, this approach starts to compare very favorably to asymptotically optimal planners like RRT*. The benchmark consists of two rigid bodies separated by a wall with a not-so-narrow passage having to swap positions. The configuration space is thus 12-dimensional.The overall performance plots can show how different planners compare on various measures. The most common performance measure is the time it took a planner to find a feasible solution. For very hard problems where most planners time out without finding a solution, it might be informative to look at solution difference: the gap between the best found solution and the goal. Explanations of the various benchmark data collected by OMPL can be found here.
The overall performance page allows you to select a motion planning problem that was benchmarked, a particular benchmark attribute to plot, the OMPL version (in case the database contains data for multiple versions), and the planners to compare. If there is only choice in a particular category, it will be disabled (since there is no other choice available).
Most of the measures are plotted as box plots. Missing data is ignored. This is very important to keep in mind: if a planner failed to solve a problem 99 times out of a 100 runs, then the average solution length is determined by one run! To make missing data more apparent, a table below the plot shows how many data points there were for each planner and how many of those were missing values (i.e., NULL, None, NA, etc.).
If your benchmark database contains results for parametrized benchmarks, then you can select results for different parameter values. By default, results are aggregated over all parameter values. You can also choose to show performance for selected planners across all parameter values by selecting “all (separate)” from the corresponding parameter selection widget.
The plots can be downloaded in two formats:
load command (or just double-click on the file if you have RStudio installed). The plot can be completely customized, further analysis can be applied to the data, or the data can be plotted in an entirely different way.Some planners in OMPL can not only report information after a run is completed, but also periodically report information during a run. In particular, for asymptotically optimal planners it is interesting to look at the convergence rate of the best path cost. Typically, the path cost is simply path length, but OMPL allows you to specify different optimization objectives. See also the benchmarking tutorial for information on how to specify objectives in the input files for ompl_benchmark.
By default, Planner Arena will plot the smoothed mean as well as a 95% confidence interval for the mean. Analogous to the performance plots, missing data is ignored. During the first couple seconds of a run, a planner may never find a solution path. Below the progress plot, we therefore plot the number of data points available for a particular planner at a particular 1 second time interval.
Regression plots show how the performance of the same planners change over different versions of OMPL. This is mostly a tool for the OMPL developers that can help in the identification of changes with unintended side-effects on performance. However, it also allows a user to easily compare the performance of a user's modifications to the planners in OMPL to the latest official release.
In regression plots, the results are shown as a bar plot with error bars.
On the “Database info” page there are two tabs. Both show information for the motion planning problem selected under “Overall performance.” The first tab show how the benchmark was set up and on what kind of machine the benchmark was run. The second tab shows more detailed information on how the planners were configured. Almost any planner in OMPL has some parameters and this tab will show exactly the parameter values for each planner.
Finally, it is possible to upload your own database of benchmark data. We have limited the maximum database size to 30MB. If your database is larger, you can run Planner Arena locally. The “Change database” page allows you to switch back to the default database after you have uploaded your own database. You can also download the default database. This might be useful if you want to extend the database with your own benchmarking results and compare our default benchmark data with your own results.
See this page for detailed instructions on how to run Planner Arena on your own computer.