Home
build details

Show: section status errors & todos local changes recent changes last change in-page changes feedback controls

Rules

Modified 2018-08-07 by julian121266

Maintainer: Julian Zilly

$$ \newcommand{\AC}[1]{{\color{blue}AC: #1}} \newcommand{\JZ}[1]{{\color{olive}JZ: #1}} \newcommand{\fix}{\marginpar{FIX}} \newcommand{\new}{\marginpar{NEW}} % Robot: \newcommand{\dynamical}{\mathcal{D}} \newcommand{\robot}{\mathcal{R}} % Robot \newcommand{\config}{\mathcal{Q}} % Configuration space (of robot) \newcommand{\sensors}{\{z\}} % Sensor set \newcommand{\bandwidth}{\mathcal{B}} \newcommand{\computation}{\mathcal{C}} \newcommand{\memory}{\mathcal{M}} \newcommand{\actuators}{\mathcal{A}} \newcommand{\knowledge}{\mathcal{K}} \newcommand{\perception}{P} \newcommand{\control}{U} \newcommand{\actions}{\mathcal{U}} % Robot mathematics \newcommand{\operator}{T} % Groups: \newcommand{\groups}{G} %\newcommand{\group}{g} \newcommand{\groupalgebra}{\mathfrak{g}} % Scene space \newcommand{\timespace}{\mathbb{T}} \newcommand{\environment}{E} \newcommand{\scene}{\xi} \newcommand{\scenespace}{\Xi} \newcommand{\universe}{U} % Sensor space \newcommand{\sensor}{\zeta} \newcommand{\sensorproj}{z} \newcommand{\sensorspace}{Z} \newcommand{\projection}{\pi} \newcommand{\projectionspace}{\Pi} \newcommand{\viewport}{v} \newcommand{\viewportspace}{\mathcal{V}} % Data space \newcommand{\dataspace}{\mathcal{X}} \newcommand{\data}{x} \newcommand{\dataproj}{\phi} \newcommand{\datakernel}{\psi} % Output space \newcommand{\outputy}{y} \newcommand{\outputspace}{\mathcal{Y}} % Task space \newcommand{\task}{T} \newcommand{\taskspace}{\mathcal{T}} \newcommand{\objective}{\mathcal{J}} \newcommand{\robotictask}{RT} \newcommand{\rules}{\Phi} \newcommand{\constraints}{\Lambda} % Action space \newcommand{\action}{u} \newcommand{\actionspace}{\mathcal{U}} \newcommand{\nuisance}{\nu} % Other characteristics / symbols \newcommand{\place}{\eta} \newcommand{\image}{I} \newcommand{\noise}{n} \newcommand{\pose}{p} \newcommand{\shape}{S} \newcommand{\albedo}{\rho} % Information theory \newcommand{\information}{\mathcal{I}} \newcommand{\expectation}{\mathbb{E}} % Optimization \newcommand{\loss}{L} $$

Performance metrics

Measuring performance in robotics is less clear cut and more multidimensional than traditionally encountered in machine learning settings. Nonetheless, to achieve reliable performance estimates we assess submitted code on several episodes with different initial settings and compute statistics on the outcomes. We denote $\objective$ to be an objective or cost function to optimize, which we evaluate for every experiment. In the following formalization, objectives are assumed to be minimized.

In the following we summarize the objectives used to quantify how well an embodied task is completed. We will produce scores in three different categories:

Note that the these objectives are not merged into one single number.

For general rules about participation and what happens with submitted code, collected logs and evaluated scores, please see general rules.

This concludes the exposition of the rules of the AI Driving Olympics. Rules and their evaluation are subject to changes to ensure practicability and fairness of scoring.

Because of mathjax bug