Virtual sensor models for real-time applications

. Increased complexity and severity of future driver assistance systems demand extensive testing and validation. As supplement to road tests, driving simulations offer various beneﬁts. For driver assistance functions the perception of the sensors is crucial. Therefore, sensors also have to be modeled. In this contribution, a statistical data-driven sensor-model, is described. The state-space based method is capable of modeling various types behavior. In this contribution, the modeling of the position estimation of an automotive radar system, including autocorrelations, is presented. For rendering real-time capability, an efﬁcient implementation is presented.


Introduction
For showing whether fully automated driving is safer than human operation, a very large distance is necessary (see e.g.Maurer et al. (2015, pp. 451-458) for first estimations).As searching rare errors is a challenge, simulations might support street tests.Being able to focus on the most crucial scenarios might decrease the required distance for road testing.Moreover finding weaknesses of driver assistance systems in early stages of the development helps improving the quality of the end product.
To enable virtual testing, most driving simulators are capable of modeling vehicle dynamics and rendering the environment.Furthermore various efforts address the generation of scenarios (Behrisch and Weber, 2015;Prialé Olivares et al., 2016;Gruyer et al., 2013).For advanced driver assistance systems (ADAS) the main input is the perception of the sensors.Therefore, implementing realistic sensor behavior is crucial for virtual ADAS development.
Approaches for simulating perception include Hardware-In-The-Loop (HiL) systems: for cameras HiL (Gans et al., 2009) setups or Software-In-The-Loop rendering (Gruyer et al., 2012) are viable alternatives.However, distance measuring sensors remain challenging to simulate.HiL setups for radar enable the simulation of one or a small number of targets (Heuel, 2015;Rohde&Schwarz, 2015).Difficulties particularly regarding the simulation of different angles remain.Lidar HiL simulation is equally problematic.Even if a well suited HiL system for distance measuring sensors would exist, the question, what values to simulate (e.g. which position), remains.
Former statistical approaches mainly focused on simple parametric sensor models (Schubert et al., 2014;Bernsteiner et al., 2013Bernsteiner et al., , 2015;;Rasshofer et al., 2005;Hanke et al., 2015).However, as shown in the subsequent section, nonparametric models present various benefits regarding realism, changes in dimensionality of the simulated quantities and generic applicability.
Whilst electromagnetic wave propagation simulation can yield accurate results, its computation might be too timeconsuming some applications in ADAS development, even when neglecting real-time requirements.
In the first section a statistical data-driven sensor model is shortly reviewed.Next the approach is extended to simulate the sensed position, including autocorrelations, of an automotive radar system.The subsequent section provides a critical discussion of the new approach, compared to classical parametric models.Afterwards, a newly developed efficient implementation, rendering the reviewed approach real-time capable, is presented.
Published by Copernicus Publications on behalf of the URSI Landesausschuss in der Bundesrepublik Deutschland e.V.
N. Hirsenkorn et al.: Virtual sensor models for real-time applications 2 Modeling In this section a sensor model generated by real test drive measurements is shown.The approach was first introduced in Hirsenkorn et al. (2015).It is shortly summarized and then extended to further increase the fidelity.At the end of this section, the advantages and disadvantages compared to parametric statistical models are presented.
In the context of probability density functions, capitalized variables are commonly denoted by random variables.Lower case letters indicate realizations of random variables.Functions marked by a hat indicate that they are an estimate (e.g. the result of an estimator).Variables marked by a tilde ∼ indicate that the quantity was measured by the sensor which should be modeled.
The task of a statistical sensor model is to estimate the probability density function (PDF) pZ sim |X sim of the simulated sensor-output Z sim given a state X sim = x sim of the simulation.During run-time the simulator draws a sample z sim from this PDF.Depending on the model, the PDF needs to be calculated at run-time.

Theory
The basic idea is illustrated in two sentences: for each simulated situation, recorded situations that are similar to this situation are identified.The simulated measurement should then be close to the recorded measurements in the identified situations.
For a vivid derivation and further explanations, refer to Hirsenkorn et al. (2015).
The conditional PDF pZ sim |X sim , introduced in the previous section, can be transformed to a joint PDF using Bayes' theorem Here p(x sim ) = c can be interpreted as a normalization constant to integrate to one.It is constant, since x sim is a fixed quantity, regarding the state of the simulator at a specific time step.For the estimation of the joint PDF, a non-parametric, kernel density estimation (KDE) approach is used.As the joint PDF is at least two-dimensional, a multivariate KDE has to be performed.For the estimation a set of N tuples of recorded measurements {z mea, t , x mea, t } t=1:N is used.Adapting the general definition (Hwang et al., 1994) to the case on hand, leads to In this equation the function K quantifies the equality of the current vector of quantities in the simulation (z T sim , x T sim ) T and the same quantities in recorded test drives (z T mea, t , x T mea, t ) T .However Sect.2.2.2 will further discuss the function K. z describes quantities of the sensor, which should be modeled (e.g. a measured position of the target vehicle).x is a state which describes the influences on the quantities that should be modeled.An example for x mea, t is the exact position of the target vehicle measured by a high precision reference system at step in time t.The states z mea, t and x mea, t were recorded at the same time.
A challenging task is the choice of proper combinations of quantities in the tuples.For example, the quantities included in the state X should contain as much information as possible, to allow a precise prediction of its associated measurement Z.However, this would lead to a high dimensionality of X. Considering the curse of dimensionality the state-space would become sparse.Sparse areas contain little information for the estimation of p Z sim |X sim .
Parametric models share the problem of choosing a proper state description.Section 4 includes a proposal of supporting a reasonable choice of state variables.

Autocorrelated position modeling
In this section, an exemplary use of the modeling mentioned in the previous section is described.The task is a detailed modeling of the sensed position using an automotive radar system.It is assumed that only one unobstructed vehicle is being measured.However, this assumption can be dropped by splitting the model into multiple subtasks (Hanke et al., 2015).

Choice of the state description
As the state description (i.e. the state variables used in X or Z) is equal for the measurements and the simulation, the subscript indexes sim and mea can be dropped in this section.Figure 1 visualizes some of the used quantities.
The state variables of the sensor-output z t consist of the estimated target position ( o x,t , o y,t ) T (e.g. of a radar system) relative to the true position of the closest corner of the vehicle (o x,corner, t , o y,corner, t ) T .Summing up, the sensor output is described by (4) The index t denotes a discrete step in time (e.g.t ∈ 1 : N for the measurements or a certain step in time of the simulation).An exemplary state would be z The reason for making z t relative to a point of the target vehicle is to increase the similarity of the PDF pZ sim |X sim at different target vehicle positions.This makes it easier for the non-parametric model to properly estimate the PDF since it does not change fast.In contrast, if not choosing relative values, the PDF pZ sim |X sim would be shifted depending on the position of the vehicle.Choosing the quantities of the influences x t on the sensoroutput, depends on the effects which should be included.On one hand the true position of the closest corner of the target vehicle is used (e.g.o x,corner,T = 46 [m] in front of the sensor and o y,corner,T = 0.7 [m] to the left).As the estimated target position is usually located at the observed side of the vehicle, choosing an unsuited position on the vehicle (e.g. the middle) would require a different model for each vehicle type (e.g.long trucks).Furthermore, the previous estimated target position, relative to the true position of the closest corner in the previous time step z t−1 , is used.This leads to In Hirsenkorn et al. (2015) we solely used the current true position.Therefore no autocorrelation was implied.This led to big changes in the sensed position in subsequent time steps.However, as raw measurements are filtered, such jumps do not occur: even if the raw measurement jumps, it will be smoothed by a filter (e.g. a Kalman Filter, Venhovens and Naab, 1999;Bar-Shalom et al., 2004).Containing this information, the PDF in time step T is located around the position of the measurement in the previous time step T −1 .This can be observed in Fig. 3.It should be remarked, that by including the previous measurement in x t , this process can be seen as a Markov-chain, which is visualized in Fig. 2.
The tuples {z mea, t , x mea, t } t=1:N were recorded using an automotive radar system (i.e.required for the measured target position) and a high precision carrier phase, differential GPS including inertial measurement unit (i.e.required for the true corner positions).To obtain a reference for the relative position of the ego-and the target-vehicle, the high precision reference system is located in both vehicles.

Choice of the kernel function
The second degree of freedom in the application of the model is the selection of the kernel function K.This function can be interpreted as a quantification of similarity.Various classes of kernel functions exist.As their statistical properties are similar, the choice should be based mainly on practical criteria (Simonoff, 2012).Due to the ease of computation and the common use in literature, the Gaussian kernel was selected.Moreover sampling from a Gaussian distribution can be implemented efficiently.benefits from this.A drawback of this kernel is the infinite support.However we neglect very low values, as their contribution to the resulting PDF is practically irrelevant.To extend the kernel function to higher dimensionalities a product kernel is used (Simonoff, 2012), which leads to a Gaussian distribution with diagonal covariance matrix.
D is the sum of the dimensionalities of z and x. c is a normalization constant.
It should be noticed, that choosing a Gaussian kernel does not imply the assumption of a Gaussian PDF, as the resulting PDF can be regarded as a weighted sum of multiple Gaussian PDFs.Because of the same reason, the independence of dimensions in the kernel does not imply independence of the dimensions in the resulting PDF.
Applying this kernel on the current example, leads to K rel quantifies the relevance of a tuple using the difference between the simulated state x sim and the tth environmentstate x t .Choosing the variances properly is a trade-off: on one hand low variances assure the situation is as similar as possible.On the other hand this leads to a low number of tuples close to the simulated state.Equal to the curse of dimensionality, it would be hardly possible to estimate the PDF.To optimize this trade-off, the variances should decrease when the number of measurements N increases.Furthermore the values depend on the unit of the dimension (e.g.variances need to be bigger using the unit mm than using m).The deployed kernel is denotes the covariance matrix in use.It is adapted to the unit m.The values were carefully, manually chosen, rewww.adv-radio-sci.net/14/31/2016/Adv.Radio Sci., 14, 31-37, 2016 Here T is the current step in time of the simulation and t is the index of a certain measurement.For the sake of completeness, the former sentences are only valid if t − 1 and t are subsequent steps in time of the same recording, and the time difference between two steps in time is equal in the simulation and in the measurements (e.g.50 ms).
The second kernel function K con specifies the shape of the contribution to the resulting PDF around the tth measured sensor-output z t .Due to its simplicity, the variances deployed in K con are computed using an automatic leave one out cross-validation approach (Simonoff, 2012).Computing the variances can only be performed after knowledge of the current simulator state x sim, t (i.e. at run-time, in each time step).The simple cross-validation is computationally expensive.To increase performance, the variances at several points of X sim are computed before run-time and stored in a table.Due to the smoothing of K rel the values are continuous.At run-time, a table-lookup with interpolation is carried out.
An alternative way of selecting the variances is using an automatic variance selection, for the relevance and the contribution at once.Result would be the same output variance in all areas.This would not account to different variances needed for different shapes of the PDFs in different areas of the state-space.For further information about variance selection (called bandwidth selection in non-parametric literature) and non-parametric statistics in general Scott (2015); Simonoff (2012) and Jiang (2010) can be consulted.

Critical discussion of the non-parametric model
This section compares the introduced approach to classic, parametric models starting with the advantages, followed by the disadvantages.Classical, parametric probability distributions are defined by a fixed amount of parameters.Often the parameters are adapted to fit the distribution to recorded measurements.However, the possible realism is limited since the distribution is restricted to a certain class Jiang (2010).Using real sensor data, the true underlying distribution will almost certainly not belong to the chosen class.Furthermore, the sensor behavior changes throughout the whole field of view.For example, in most cases the accuracy of a sensor is higher in the middle of the field of view than at the borders.For sensor models this implies, that the shape of the PDF differs depending on the position in the state-space X sim .In parametric sensor models this is sometimes treated heuristically by a linear increasing standard deviation at the borders (Bernsteiner et al., 2013(Bernsteiner et al., , 2015)).The non-parametric model provides a more accurate adaption: the PDF pZ sim |X sim is calculated solely using measurements close to the current state of the simulation x sim,T (i.e.measurements characteristic for the current state).
Data-driven, non-parametric models are very flexible.An asymptotic view reveals this strength: with an infinite amount of data, a non-parametric model will converge almost sure to the true distribution (Wied and Weißbach, 2012).In contrast, parametric models lack this property.
The described model can also treat changes in dimensionality of z or x.This is of high practical relevance, for example when encountering object-losses.In non-parametric modeling this can be included due to the separation into relevance and contribution.This article will not go into further detail here, however the next section and Hirsenkorn et al. (2015) will clarify this.
Furthermore, the behavior of the model at a certain time step can be linked to one tuple of a test drive.This means unexpected behavior in the simulation can be traced back to a specific measurement.The subsequent section will show this.
Due to the flexibility of the model, some readers might have an association to the problem of overfitting.In other words, the model sticks too close to recorded measurements, shortcoming of generalization.Preventing overfitting comes at the cost of adding assertions on the behavior.As explained earlier in this section, the assumption of a specific distribution in parametric models destroys asymptotic properties.Moreover, overfitting is desirable in certain cases: the model sticks to real measurements but sacrifices completeness: behavior, which has not been encountered, will not be modeled, but the model also will not add behavior that does not exist.
However, the authors also want to describe the drawbacks: the model is less interpretable and therefore hard to adapt to other behavior.Moreover, the modeling of a future sensor, of which no measurements are available yet, is barely possible.This is relevant for parallel development of ADAS and the sensor.
Another drawback is the dependence on sensor measurements including reference-measurements.This may be one of the main reasons, the approach has not been discussed earlier: accurate reference measurements were not available.However, parametric models with parameters adapted to measurements share this drawback.
A big drawback of directly implementing the nonparametric approach is the high computational complexity.It increases with the number of measurements.Classical parametric models, such as a Gaussian model, are of constant complexity.The implementation introduced in the subsequent section further discusses this topic and provides a solution to minimize the drawback.

Computational improvements
This section investigates the computational complexity of the approach.Next solutions are presented to decrease the complexity to a fraction of the initial amount, whilst increasing the numerical accuracy.
Combining Eqs. ( 1), ( 3) and ( 7) leads to In practical use, the number of tuples N may consist of 10 4 to 10 5 measurements or more (i.e.∼ 1 h of test drive, considering 20 Hz update frequency).This is why processing this equation needs to be highly optimized.Since the relevance computation K rel ( x t ), can be seen as the inverse of a distance measure, large distances lead to negligible values.Therefore, highly efficient standard algorithms of fixed radius near neighbors search (Muja and Lowe, 2009), to obtain the relevant samples, can be used.The solutions often contain optimizations such as a graph construction at the beginning, which can be performed before run-time.After identifying the closest points, the more expensive relevance computation is performed only to the closest points -a small fraction of all samples.The remaining points are neglected for further computation since their relevance is practically zero.We use an R-tree (Beckmann et al., 1990) implementation of the C++ library boost.
The direct way of drawing from Eq. ( 11) is the evaluation of the PDF at a lot of support points.Besides being computationally expensive, reconstructing the PDF using support points is practically always a lossy approximation.Next the drawing has to be performed from the multidimensional complexly shaped PDF.This step also inefficient.
A more accurate and faster way of evaluating the PDF can be executed by a two stage drawing process.Following the law of total probability and comparing to Eq. ( 11) leads to At first, the tuple is selected which should be followed.This is done by a weighted drawing using w 1:N .Then drawing samples from the Gaussian distribution K con ( z t ), located at the sensor-output z t of the selected tuple, is executed.Figure 4 visualizes this process for N = 3 measurements.Summing up, drawing values using this implementation is statistically equivalent to drawing using the original PDF.Furthermore this algorithm is missing the necessity of an explicit representation of the PDF, which could only be approximated by support points.Therefore it is able to reconstruct the desired PDF more accurately than the direct evaluation.
It should be noted that this derivation also shows the possibility to identify the origin x t of a certain behavior in simulation.10).The lower figure presents a binned relative frequency using 95 000 samples drawn by the two stage drawing process described in Eq. ( 12).

Verification
Figure 5 compares an analytically computed, binned PDF (i.e. the PDF we actually want to draw from, Eq. 10), and the binned relative frequency resulting from 95 000 samples drawn using the two stage process (Eq.12).The values of the borders of the bins are the same in both figures.This ensures the comparability and shows that the PDF is not shifted.
Besides the random fluctuations, which were to be expected due to the random nature of the drawing process, both plots show good agreement.This result could be reproduced on arbitrary other PDFs, i.e. at other locations of the statespace X sim and other choices of the state-variables.

Conclusions and outlook
In this article a sensor model, simulating the position output was presented.The data-driven model was generated using real test drives including an automotive radar system in addition to reference sensors.Whilst the scope of the effects reach from noise to field of view and object-losses, mainly autocorrelation of the sensed position was discussed.Next a critical discussion of the advantages and disadvantages, compared to classical parametric models, was presented.Furthermore an efficient implementation was derived, enabling real-time operation whilst achieving accurate results.The improvements were verified using an optical comparison.
Future works should focus on data-driven models, not using high precision reference sensors but common sensors, which are available in production vehicles.This enhancement sets up test drives on any public road, leading to realistic scenarios.As production vehicles would contain the necessary equipment, this would also enable crowdsourcing of sensor models.The potential of fleet data regarding various tasks in the automotive industry has already been shown in Ruhhammer et al. (2014), Klanner and Ruhhammer (2015) and Protschky et al. (2015).This approach may not be possible for all quantities or microscopic effects, such as noise.However, macroscopic behavior, for instance the field of view depending on weather conditions, might be observable and quantifiable.
A quality criterion for quantifying the degree of realism would enable further improvements.Besides a well founded choice of the variances in the input kernel function, the the state representation could be optimized.For example, the selection of the state-representation could be automated.Furthermore, abstract state representations, acquired from dimensionality reduction techniques, might boost the performance.

Figure 1 .
Figure 1.Visualization of the state description.

Figure 2 .
Figure 2. Depiction of the model as a Markov-chain.

Figure 3 .
Figure 3. Course of the distribution of the PDF of the sensor-output in subsequent time steps.The sensor is located at the origin pointing in longitudinal direction.The white rectangle indicates the bounding box of the simulated target vehicle.The heat map shows the PDF p(z sim |X sim = x sim ).The green dot indicates the drawn position at the current time step.The cyan dot indicates the former sensor-output, maintaining the offset of the previous time step z sim, t−1 to the current closest corner.The white dots indicate the positions of recorded measurements z k .The size of each dot reveals the relevance of the kth tuple.For the shown plots, only ∼ 5000 measurements, spread over the whole field of view, were used.

Figure 4 .
Figure 4.The two stage drawing process.

Figure 5 .
Figure5.The upper figure shows an analytically computed, binned relative frequency using an integration of Eq. (10).The lower figure presents a binned relative frequency using 95 000 samples drawn by the two stage drawing process described in Eq. (12).
The efficient implementation in Sect. 3