Off-Road Autonomy Validation Using Scalable Digital Twin Simulations Within High-Performance Computing Clusters (2024)

3.1.1 Suspension Dynamics

The stiffness ${{}^{i}K}={{}^{i}M}*{{}^{i}\omega_{n}}^{2}$ and damping ${}^{i}B=2*^{i}\zeta*\sqrt{{{}^{i}K}*{{}^{i}M}}$ coefficients of the suspension system are computed based on the sprung mass ${{}^{i}M}$, natural frequency ${{}^{i}\omega_{n}}$, and damping ratio ${{}^{i}\zeta}$ parameters. The point of suspension force application ${{}^{i}Z_{F}}$ is calculated based on the suspension geometry:${{}^{i}Z_{F}}={{}^{i}Z_{\text{COM}}}-{{}^{i}Z_{w}}+{{}^{i}r_{w}}-{{}^{i}Z_{f}}$, where ${}^{i}Z_{\text{COM}}$ denotes the Z-component of vehicle’s center of mass, ${}^{i}Z_{w}$ is the Z-component of the relative transformation between each wheel and the vehicle frame (${}^{V}T_{w_{i}}$), ${}^{i}r_{w}$ is the wheel radius, and ${}^{i}Z_{f}$ is the force offset determined by the suspension geometry. Lastly, the suspension displacement ${}^{i}Z_{s}$ at any given moment can be computed as ${{}^{i}Z_{s}}=\frac{{{}^{i}M}*g}{{{}^{i}Z_{0}}*{{}^{i}K}}$, where $g$ represents the acceleration due to gravity, and ${}^{i}Z_{0}$ is the suspension’s equilibrium point. Additionally, the vehicle model also has a provision to include anti-roll bars, which apply a force on the left ${{}^{L}F_{r}}=K_{r}*{{}^{R}Z}-{{}^{L}Z}$ and right ${R^{F}_{r}}=K_{r}*{{}^{L}Z}-{{}^{R}Z}$ wheels as long as they are grounded at the contact point $Z_{c}$. This force is directly proportional to the stiffness of the anti-roll bar, $K_{r}$. The left and right wheel travels are given by ${{}^{L}Z}=\frac{-{{}^{L}Z_{c}}-{{}^{L}r_{w}}}{{}^{L}Z_{s}}$ and ${{}^{R}Z}=\frac{-{{}^{R}Z_{c}}-{{}^{R}r_{w}}}{{}^{R}Z_{s}}$.

3.1.2 Powertrain Dynamics

The powertrain comprises an engine, transmission and differential. The engine is modeled based on its torque-speed characteristics. The engine RPM is updated smoothly based on its current value $RPM_{e}$, the idle speed $RPM_{i}$, average wheel speed $RPM_{w}$, final drive ratio $FDR$, current gear ratio $GR$, and the vehicle velocity $v$. The update can be expressed as $RPM_{e}:=\left[RPM_{i}+\left(|RPM_{w}|*FDR*GR\right)\right]_{(RPM_{e},v)}$ where, $[\mathscr{F}]_{x}$ denotes evaluation of $\mathscr{F}$ at $x$.

The total torque generated by the powertrain is computed as $\tau_{\text{total}}=\left[\tau_{e}\right]_{RPM_{e}}*\left[GR\right]_{G_{\#}}*$. Here, $\tau_{e}$ is the engine torque, $\tau$ is the throttle input, and $\mathscr{A}$ is a non-linear smoothing operator which increases the vehicle acceleration based on the throttle input.

The automatic transmission decides to upshift/downshift the gears based on the transmission map of a given vehicle. This keeps the engine RPM in a good operating range for a given speed: $RPM_{e}=\frac{{v_{\text{MPH}}*5280*12}}{{60*2*\pi*R_{\text{tire}}}}*FDR*GR$. It is to be noted that while shifting the gears, the total torque produced by the powertrain is set to zero to simulate the clutch disengagement. It is also noteworthy that the auto-transmission is put in neutral gear once the vehicle is in standstill condition and parking gear if handbrakes are engaged in standstill condition. Additionally, switching between drive and reverse gears requires that the vehicle first be in the neutral gear to allow this transition.

The total torque $\tau_{\text{total}}$ from the drivetrain is divided to the wheels based on the drive configuration of the vehicle:$\tau_{\text{out}}=\begin{cases}\frac{\tau_{\text{total}}}{2}&\text{if FWD/RWD}$.The torque transmitted to wheels $\tau_{w}$ is modeled by dividing the output torque $\tau_{\text{out}}$ to the left and right wheels based on the steering input. The left wheel receives a torque amounting to ${}^{L}\tau_{w}=\tau_{\text{out}}*(1-\tau_{\text{drop}}*|\delta^{-}|)$, while the right wheel receives a torque equivalent to ${}^{R}\tau_{w}=\tau_{\text{out}}*(1-\tau_{\text{drop}}*|\delta^{+}|)$. Here, $\tau_{\text{drop}}$ is the torque-drop at differential and $\delta^{\pm}$ indicates positive and negative steering angles, respectively. The value of $(\tau_{\text{drop}}*|\delta^{\pm}|)$ is clamped between $[0,0.9]$.

3.1.3 Steering Dynamics

The steering mechanism operates by employing a steering actuator, which applies a torque $\tau_{\text{steer}}$ to achieve the desired steering angle $\delta$ with a smooth steering rate $\dot{\delta}$, without exceeding the steering limits $\pm\delta_{\text{lim}}$. The rate at which the vehicle steers is governed by its speed $v$ and steering sensitivity $\kappa_{\delta}$, and is represented as $\dot{\delta}=\kappa_{\delta}+\kappa_{v}*\frac{v}{v_{\text{max}}}$. Here, $\kappa_{v}$ is the speed-dependency factor of the steering mechanism. Finally, the individual angle for left $\delta_{l}$ and right $\delta_{r}$ wheels are governed by the Ackermann steering geometry, considering the wheelbase $l$ and track width $w$ of the vehicle:$\left\{\begin{matrix}\delta_{l}=\textup{tan}^{-1}\left(\frac{2*l*\textup{tan}($.

3.1.4 Brake Dynamics

The braking torque is modeled as ${{}^{i}\tau_{\text{brake}}}=\frac{{{}^{i}M}*v^{2}}{2*D_{\text{brake}}}*R_{b}$ where $R_{b}$ is the brake disk radius and $D_{\text{brake}}$ is the braking distance at 60 MPH, which can be obtained from physical vehicle tests. This braking torque is applied to the wheels based on the type of brake input: for combi-brakes, this torque is applied to all the wheels, and for handbrakes, it is applied to the rear wheels only.

3.1.5 Tire Dynamics

Tire forces are determined based on the friction curve for each tire $\left\{\begin{matrix}{{}^{i}F_{t_{x}}}=F(^{i}S_{x})\\{{}^{i}F_{t_{y}}}=F(^{i}S_{y})\\\end{matrix}\right.$, where ${}^{i}S_{x}$ and ${}^{i}S_{y}$ represent the longitudinal and lateral slips of the $i$-th tire, respectively. The friction curve is approximated using a two-piece spline, defined as $F(S)=\left\{\begin{matrix}f_{0}(S);\;\;S_{0}\leq S<S_{e}\\f_{1}(S);\;\;S_{e}\leq S<S_{a}\\\end{matrix}\right.$, with $f_{k}(S)=a_{k}*S^{3}+b_{k}*S^{2}+c_{k}*S+d_{k}$ as a cubic polynomial function. The first segment of the spline ranges from zero $(S_{0},F_{0})$ to an extremum point $(S_{e},F_{e})$, while the second segment ranges from the extremum point $(S_{e},F_{e})$ to an asymptote point $(S_{a},F_{a})$. Tire slip is influenced by factors including tire stiffness ${}^{i}C_{\alpha}$, steering angle $\delta$, wheel speeds ${}^{i}\omega$, suspension forces ${}^{i}F_{s}$, and rigid-body momentum ${{}^{i}P}={{}^{i}M}*{{}^{i}v}$. The longitudinal slip ${}^{i}S_{x}$ of $i$-th tire is calculated by comparing the longitudinal components of its surface velocity $v_{x}$ (i.e., the longitudinal linear velocity of the vehicle) with its angular velocity ${}^{i}\omega$: ${{}^{i}S_{x}}=\frac{{{}^{i}r}*{{}^{i}\omega}-v_{x}}{v_{x}}$. The lateral slip ${}^{i}S_{y}$ depends on the tire’s slip angle $\alpha$ and is determined by comparing the longitudinal $v_{x}$ (forward velocity) and lateral $v_{y}$ (side-slip velocity) components of the vehicle’s linear velocity: ${{}^{i}S_{y}}=\tan(\alpha)=\frac{v_{y}}{\left|v_{x}\right|}$.

3.1.6 Aerodynamics

The vehicle digital twin is modeled to simulate variable air drag $F_{\text{aero}}$ acting on it, which is computed based on the vehicle’s operating condition:$F_{\text{aero}}=\begin{cases}F_{d_{\text{max}}}&\text{if }v\geq v_{\text{max}}$where, $v$ is the vehicle velocity, $v_{\text{max}}$ is the vehicle’s designated top-speed, $v_{\text{rev}}$ is the vehicle’s designated maximum reverse velocity, $G_{\#}$ is the operating gear, and $RPM_{w}$ is the average wheel RPM. Apart from this, a linear angular $T_{d}$ drag acts on the vehicle, which is directly proportional to its angular $\omega$ velocity. Finally, the downforce acting on the vehicle is also modeled proportional to its velocity: $F_{\text{down}}=K_{\text{down}}*|v|$, where $K_{\text{down}}$ is the downforce coefficient.

3.2.1 Actuator Feedbacks

Throttle ($\tau$), steering ($\delta$), brake ($\chi$) and handbrake ($\xi$) sensors are simulated using a simple feedback loop. These variables keep track of the commands relayed to the vehicle actuators using a get() method.

3.2.2 Incremental Encoders

Simulated incremental encoders measure wheel rotations ${}^{i}N_{\text{ticks}}={{}^{i}PPR}*{{}^{i}CGR}*{{}^{i}N_{\text{rev}}}$, where ${}^{i}N_{\text{ticks}}$ represents the measured ticks, ${}^{i}PPR$ is the encoder resolution (pulses per revolution), ${}^{i}CGR$ is the cumulative gear ratio, and ${}^{i}N_{\text{rev}}$ represents the wheel revolutions.

3.2.3 Inertial Navigation Systems

Positioning systems and inertial measurement units (IMU) are simulated based on temporally coherent rigid-body transform updates of the vehicle $\{v\}$ with respect to the world $\{w\}$: ${{}^{w}\mathbf{T}_{v}}=\left[\begin{array}[]{c | c}\mathbf{R}_{3\times 3}&$. The positioning systems provide 3-DOF positional coordinates $\{x,y,z\}$ of the vehicle, while the IMU supplies linear accelerations $\{a_{x},a_{y},a_{z}\}$, angular velocities $\{\omega_{x},\omega_{y},\omega_{z}\}$, and 3-DOF orientation data for the vehicle, either as Euler angles $\{\phi_{x},\theta_{y},\psi_{z}\}$ or as a quaternion $\{q_{0},q_{1},q_{2},q_{3}\}$.

3.2.4 Cameras

Simulated cameras are parameterized by their focal length $f$, sensor size $\{s_{x},s_{y}\}$, target resolution, as well as the distances to the near $N$ and far $F$ clipping planes. The viewport rendering pipeline for the simulated cameras operates in three stages.

First, the camera view matrix $\mathbf{V}\in SE(3)$ is computed by obtaining the relative hom*ogeneous transform of the camera $\{c\}$ with respect to the world $\{w\}$: $\mathbf{V}=\begin{bmatrix}r_{00}&r_{01}&r_{02}&t_{0}\\r_{10}&r_{11}&r_{12}&t_{1}\\r_{20}&r_{21}&r_{22}&t_{2}\\0&0&0&1\\\end{bmatrix}$, where $r_{ij}$ and $t_{i}$ denote the rotational and translational components, respectively.

Next, the camera projection matrix $\mathbf{P}\in\mathbb{R}^{4\times 4}$ is calculated to project world coordinates into image space coordinates: $\mathbf{P}=\begin{bmatrix}\frac{2*N}{R-L}&0&\frac{R+L}{R-L}&0\\0&\frac{2*N}{T-B}&\frac{T+B}{T-B}&0\\0&0&-\frac{F+N}{F-N}&-\frac{2*F*N}{F-N}\\0&0&-1&0\\\end{bmatrix}$, where $L$, $R$, $T$, and $B$ denote the left, right, top, and bottom offsets of the sensor. The camera parameters $\{f,s_{x},s_{y}\}$ are related to the terms of the projection matrix as follows: $f=\frac{2*N}{R-L}$, $a=\frac{s_{y}}{s_{x}}$, and $\frac{f}{a}=\frac{2*N}{T-B}$. The perspective projection from the simulated camera’s viewport is given as $\mathbf{C}=\mathbf{P}*\mathbf{V}*\mathbf{W}$, where $\mathbf{C}=\left[x_{c}\;\;y_{c}\;\;z_{c}\;\;w_{c}\right]^{T}$ represents image space coordinates, and $\mathbf{W}=\left[x_{w}\;\;y_{w}\;\;z_{w}\;\;w_{w}\right]^{T}$ represents world coordinates.

Finally, this camera projection is transformed into normalized device coordinates (NDC) by performing perspective division (i.e., dividing throughout by $w_{c}$), leading to a viewport projection achieved by scaling and shifting the result and then utilizing the rasterization process of the graphics API (e.g., DirectX for Windows, Metal for macOS, and Vulkan for Linux).

Additionally, a post-processing step physically simulates non-linear lens and film effects, such as lens distortion, depth of field, exposure, ambient occlusion, contact shadows, bloom, motion blur, film grain, chromatic aberration, etc.

3.2.5 Planar LIDARs

2D LIDAR simulation employs iterative ray-casting raycast{${}^{w}\mathbf{T}_{l}$, $\vec{\mathbf{R}}$, $r_{\text{max}}$} for each angle $\theta\in\left[\theta_{\text{min}}:\theta_{\text{res}}:\theta_{\text{max}}\right]$ at a specified update rate. Here, ${{}^{w}\mathbf{T}_{l}}={{}^{w}\mathbf{T}_{v}}*{{}^{v}\mathbf{T}_{l}}\in SE(3)$ represents the relative transformation of the LIDAR {$l$} with respect to the vehicle {$v$} and the world {$w$}, $\vec{\mathbf{R}}=\left[\cos(\theta)\;\;\sin(\theta)\;\;0\right]^{T}$ defines the direction vector of each ray-cast $R$, where $r_{\text{min}}$ and $r_{\text{max}}$ denote the minimum and maximum linear ranges, $\theta_{\text{min}}$ and $\theta_{\text{max}}$ denote the minimum and maximum angular ranges, and $\theta_{\text{res}}$ represents the angular resolution of the LIDAR, respectively. The laser scan ranges are determined by checking ray-cast hits and then applying a threshold to the minimum linear range of the LIDAR, calculated as ranges[i]$=\begin{cases}d_{\text{hit}}&\text{ if }\texttt{ray[i].hit}\text{ and }d_{$, where ray.hit is a Boolean flag indicating whether a ray-cast hits any colliders in the scene, and $d_{\text{hit}}=\sqrt{(x_{\text{hit}}-x_{\text{ray}})^{2}+(y_{\text{hit}}-y_{$ calculates the Euclidean distance from the ray-cast source $\{x_{\text{ray}},y_{\text{ray}},z_{\text{ray}}\}$ to the hit point $\{x_{\text{hit}},y_{\text{hit}},z_{\text{hit}}\}$.

3.2.6 Spatial LIDARs

3D LIDAR simulation adopts multi-channel parallel ray-casting raycast{${}^{w}\mathbf{T}_{l}$, $\vec{\mathbf{R}}$, $r_{\text{max}}$} for each angle $\theta\in\left[\theta_{\text{min}}:\theta_{\text{res}}:\theta_{\text{max}}\right]$ and each channel $\phi\in\left[\phi_{\text{min}}:\phi_{\text{res}}:\phi_{\text{max}}\right]$ at a specified update rate, with GPU acceleration (if available). Here, ${{}^{w}\mathbf{T}_{l}}={{}^{w}\mathbf{T}_{v}}*{{}^{v}\mathbf{T}_{l}}\in SE(3)$ represents the relative transformation of the LIDAR {$l$} with respect to the vehicle {$v$} and the world {$w$}, $\vec{\mathbf{R}}=\left[\cos(\theta)*\cos(\phi)\;\;\sin(\theta)*\cos(\phi)\;\;-$ defines the direction vector of each ray-cast $R$, where $r_{\text{min}}$ and $r_{\text{max}}$ denote the minimum and maximum linear ranges, $\theta_{\text{min}}$ and $\theta_{\text{max}}$ denote the minimum and maximum horizontal angular ranges, $\phi_{\text{min}}$ and $\phi_{\text{max}}$ denote the minimum and maximum vertical angular ranges, and $\theta_{\text{res}}$ and $\phi_{\text{res}}$ represent the horizontal and vertical angular resolutions of the LIDAR, respectively. The thresholded ray-cast hit coordinates $\{x_{\text{hit}},y_{\text{hit}},z_{\text{hit}}\}$, from each of the casted rays is encoded into byte arrays based on the LIDAR parameters, and given out as the point cloud data (PCD).

4.2.1 AutoDRIVE Simulator Containerization

The AutoDRIVE Simulator container is built on top of nvidia/vulkan base container and installs the necessary software dependencies, an X virtual framebuffer (Xvfb) server, fast forward moving picture experts group (FFmpeg) [56], and a Python-based HTTP server. The Xvfb server is initialized in order to render simulation streams from a headless container. FFmpeg is used to grab frames of the virtual display and forward them to an HTTP live streaming (HLS) client, which is published to an HTTP server that is ported outside of the container. FFmpeg can also be complemented or replaced with a virtual network computing (VNC) server if GUI control is needed to interact with the simulations. Additionally, in case the simulator supports a “render off-screen” mode, which publishes the sensor data and any video feeds to programmable outputs, initializing an Xvfb server is not necessary.

4.2.2 AutoDRIVE Devkit Containerization

The container for the AutoDRIVE Devkit was built from the python:3.8.10 base image. Inside the image are necessary software dependencies, the Python script controlling and initializing the simulation with AutoDRIVE Python API, as well as the HPC framework’s data logging module. The data logging module exposes an API, which a developer integrates into the simulation Python script to gather simulation parameters from the control server and record any desired vehicle metrics.

4.2.3 Control Server Containerization

The control server is built from node:14, the official node 14 Docker image, and runs a node.js webserver to orchestrate communication between simulation pods and connections outside the cluster.

4.2.4 WebViewer Containerization

The WebViewer container is built off of the nginx:alpine image and contains a webpage that loads HLS streams from active simulations in the cluster. The WebViewer is an HTML/Javascript webpage featuring a 4$\times$4 grid of interactive windows for rendering simulation streams (refer Fig. 1).

4.3.1 AutoDRIVE Deployment

The AutoDRIVE deployment is responsible for the orchestration of simulation pods running the AutoDRIVE Simulator container and the AutoDRIVE Devkit container as outlined earlier. The Devkit container communicates with the Simulator container to initialize and control simulations (send vehicle commands, analyze sensor input, change environmental parameters, etc.) over a pod’s local network. Every AutoDRIVE pod requires a GPU in order to run a simulation instance, resulting in the number of replicas in this deployment being dependent on the cluster’s GPU (or GPU replicas if a GPU is either time-sliced or in multi-instance mode) availability.

4.3.2 WebViewer Deployment

The WebViewer deployment contains only one pod, which hosts a container for the control server and another container for the WebViewer to render live video streams from all the simulation instances as they spin-up and run.

4.3.3 Services

A service is employed to expose a specific aspect of a Kubernetes cluster, typically an application or endpoint, making it accessible to connections both within the cluster and externally. Two NodePort services were used to expose the control server and WebViewer containers outside of the cluster.

Additionally, a headless service was used to expose pods in the AutoDRIVE deployment to the control server. Unlike a NodePort service, when queried, a headless service returns a list of all corresponding resource addresses. This behavior is ideal for the control server to dynamically acquire the addresses of simulation pods and proxy video streams outside of the cluster.

4.3.4 Time Slicing Configmap

In the cluster, GPUs were configured to facilitate GPU time-slicing. This approach was adopted to generate “replicas” of each GPU, thereby enabling the registration of multiple GPU-enabled pods per video card within a node. The preference for time-slicing over the multiple instance graphics (MIG) mode was primarily due to the incompatibility of MIG with certain graphics libraries found on prevalent HPC video cards, such as the NVIDIA A100 [57]. As mentioned earlier, our study was conducted on Node 003 (refer Table 1), which was time-sliced into 8 segments per card resulting in a total of 16 GPU replicas. The choice of 8 replicas per GPU was selected as a starting point after preliminary experimentation of simulator GPU utilization on a non-sliced A100 card.

4.4.1 Kubernetes Python Client

The Kubernetes official Python client wraps calls to the RESTful Kubernetes API server in order to allow for complete cluster control via Python scripts. This client is used in the HPC Framework Python Library for cluster orchestration.

4.4.2 HPC Framework Python Library

A custom Python library was developed, serving two primary purposes: (i) integrating with the existing simulation scripts to accurately log simulation metrics and (ii) providing an abstracted API through which the users can automate the verification and validation of all the necessary test cases via appropriately configured simulation instances.

The aforementioned data logging module stores specified metrics in a comma-separated value (CSV) file within the simulation pod while a simulation is running. Once a test case concludes, the module contains functionality to send collected metrics back to the control plane for transmission outside of the cluster. To introduce variability among otherwise identical simulation containers, the logging module can request initialization parameters from the control plane (e.g., weather conditions, time of day, or any other variable).

The automation module allows Python scripts to automate cluster processes and run simulations in larger batches. Functionally, this is achieved through a combination of calls to the Kubernetes Python API and requests made to the control plane within the cluster.

4.5.1 WebViewer and Control Server Communication

In order to account for the dynamic IP addresses of simulation pods (since pods are created and terminated after each batch of simulations) the control server opens a proxy to running simulation pod video streams by performing a DNS lookup on the headless service configured with the AutoDRIVE deployment. This allows an endpoint URL structure of /stream/$<$sim#$>$/video.m3u8 to route to simulations based on position in a batch regardless of that simulation’s IP address. Pod addresses are updated and proxies are rebuilt by the server after each batch of simulations is completed.

4.5.2 Pod and Control Server Communication

Outside of querying the headless service for pod addresses, the control server mediates metrics reported from simulation pods and stores them to be extracted by the HPC Framework Python Client. Once completed, simulation pods post metrics recorded to an endpoint on the control server. The control server hosts another endpoint that the simulation pods connect to using the HPC Framework Python Client in order to fetch any variation in simulation parameters.

4.5.3 HPC Framework Python Client and Control Server Communication

The HPC Framework Python Client communicates with the control server for three purposes. Firstly, to request a dump of the simulation data stored in the server and save each batch of simulations to a CSV file locally. Secondly, to post a change in the server’s stored simulation configuration parameters, which are then passed to pods when they are initializing. Thirdly, to command the control server to query the headless service and refresh the stored pod IP addresses when a new batch of simulations is launched.