\section{Introduction}\label{sec_intro}

\subsection{Background}
\subsubsection{Significance of V2X Communications}
Vehicle-to-Everything (V2X) communications have the potential to significantly bring down the number of vehicle crashes, thereby reducing the number of associated fatalities \cite{gdot_1}. The capability gave V2X communications the central role in constitution of intelligent transportation system (ITS) for connected and autonomous vehicles (CAVs).

\subsubsection{Challenge}
Due to the very high mobility of vehicles, there will be frequent change in the topology of the network. Mainly, routing, mobility and security are affected by this frequent topology change \cite{icc19}.

Analysis of such ever-dynamic environments surrounding a vehicle has been attempted in the literature of \textit{usage-based insurance (UBI)}. This technique is a means to incorporate analysis of the driving behaviors (e.g., speed, mileage, and harsh braking/accelerating) into determination of an auto insurance rate. The best practice that has been adopted by many insurance programs to protect users' location privacy is the use of driving speed rather than Global Positioning System (GPS) data \cite{allstate18}.

However, the current UBI-based approaches have limitations to be applied to make a V2X network adapt itself to a dynamic environment. In practice, many environmental factors, such as traffic amount/pattern and regulations, influence the risk level to which a vehicle is exposed in ``real-time.'' Nevertheless, the current UBI schemes cannot reflect dynamic traffic patterns and environmental change.

\subsubsection{Approach Adopted by This Paper}
In a CAV environment, the vehicles must be able to explore, learn from, make accurate evaluation of the level of an accident risk, and adapt their transmission of basic safety messages (BSMs) to the environment. To date, there is no proposal fostering such a capability at each vehicle. This makes a compelling case that a V2X network should employ a machine learning functionality to keep itself nimble on the fly without the need for external update, which causes additional delay.

\textit{Online machine learning} is a method in which data becomes available in a \textit{streaming} manner \cite{ms19}, as opposed to batch (or offline) learning which is trained by an entire training data set at once. As such, online learning is known to be particularly efficient in areas where it is computationally infeasible to train over the entire dataset, which includes V2X environments.

Notice that this paper focuses on defining the level of ``accident risk'' that will be used as a metric, to which a V2X networking protocol will be adapted. There are a plethora of factors determining an accident risk, which makes the training data ``streamed'' over time. Hence, being able to dynamically adapt to new inputs, an online learning algorithm will very well suit this paper's learning model.

Furthermore, our proposed learning framework must be resilient. We adopt \textit{reinforcement learning} so the learning framework can enhance itself in the dynamic V2X environments. In particular, in order to reflect the high dynamicity, this paper characterizes the learning framework as a \textit{contextual multi-armed bandit (MAB)} problem.

\subsection{Contributions of This Paper}
This paper presents the following contributions:
\begin{enumerate}
\item \textit{Extendibility of the proposed learning framework:} While this paper selects (i) variance of speed and (ii) inter-vehicle distance as contexts to be learned, the proposed learning framework is designed in such a way that it can be easily extended to (i) accommodate any other accident risk-related contexts as ``online'' inputs and (ii) adapt the networking behavior to an ever-changing environment.
\item \textit{Generality of the accident risk measurement model:} While the existing UBI models consider driver-oriented factors only, this paper proposes an accident risk measurement model that can be extended to any factors to be learned: not only driver-related factors but external factors such as road condition and weather as well. It makes the proposed model far more generally applicable than the typical UBI models.
\item \textit{Capability of completely autonomous learning and networking adaptation:} The proposed protocol can be applied to any ``distributed'' and ``listen-before-talk (LBT)'' type of V2X network--e.g., dedicated short-range communications (DSRC) and cellular V2X (C-V2X) mode 4. As such, it is a remarkable improvement from the traditional UBI in such a way that each vehicle is able to learn the environment according to the context and hence improve the UBI model itself on the fly without external update nor prior knowledge.
\end{enumerate}

\begin{table}[t]
\caption{Key abbreviations and notations}
\centering
\begin{tabular}{ |l|l|}
\hline
\textbf{\cellcolor{gray!30}Abbreviation} & \cellcolor{gray!30}\textbf{Description}\\
\hline\hline
BSM & Basic safety message\\
C-V2X & Cellular V2X\\
DSRC & Dedicated short-range communications\\
EXP & Packet expiration\\
LBT & Listen-before-talk\\
MAB & Multi-armed bandit\\
PPP & Poisson point process\\
RL & Reinforcement learning\\
UBI & Usage-based insurance\\
VANET & Vehicular ad-hoc network\\
V2X & Vehicle-to-everything communications\\
\hline
\hline
\textbf{\cellcolor{gray!30}Notation} & \cellcolor{gray!30}\textbf{Description}\\
\hline\hline
$w_{j}$ & Weight for context in context $j$, $\mathbf{c}_{j}$, in $\mathcal{AR}_{t,i}$\\
$\mathcal{AR}_{t,i}$ & Accident risk of vehicle $i$ at time instant $t$\\
$\mathsf{bo}$ & Backoff counter, i.e., $\left[0, \text{CW}-1\right]$\\
$\mathbf{c}_{i}$ & Vector of ``context'' $i$, i.e., $\left(c_{v,i}, c_{d,i}\right)$\\
CW & Contention window\\
$d^{\text{min}}_{i}$ & Minimum inter-vehicle distance at vehicle $i$\\
$\Psi$ & Variance of speed from the road's speed limit\\
$\mathsf{N}_{\text{arm}}$ & Number of arms\\
$\mathcal{S}_{\text{ctx}}$ & Set of contexts, i.e., $\mathcal{S}_{\text{ctx}} = \left\{ v, d \right\}$\\
$\mathsf{bo}_{t,i}$ & Backoff value for vehicle $i$ at time $t$\\
$v_{i}$ & Speed of vehicle $i$\\
$\mathbf{w}_{i}$ & An arm of the bandit $i$, i.e., weights $\left( w_{v}, w_{d} \right)$\\
\hline
\end{tabular}
\label{table_abbreviations}
\end{table}

\section{Related Work}\label{sec_related}

\subsubsection{Analysis of V2X Networking Performance}
It is with particular challenge to analyze the performance of a V2X network. Analysis frameworks based on stochastic geometry for DSRC have been proposed \cite{arxiv19}-\cite{arxiv20}. They commonly rely on the fact that uniform distributions of nodes on $X$ and $Y$ axes of a Cartesian-coordinate two-dimensional space yield a Poisson point process (PPP) on the number of nodes in the space \cite{haenggi05}. This paper also applies the stochastic geometry framework for analysis of the proposed mechanism.

\subsubsection{Accident Risk Quantification}
As has been mentioned earlier, the UBI is the main literature where the most effort on accident risk quantification has been found \cite{survey18}. The technique is quite stable and reliable, which makes itself widely applied in practice already by the commercial insurance providers \cite{allstate18}.

Nevertheless, as pointed out in Section \ref{sec_intro}, the key limitation of the current literature on UBI is that only driver-related factors are taken into consideration. There are too many other factors that should be considered in order to accurately model an accident risk. Even further, the type, number, and influence in concert of the factors are changing all over time. Yet the existing proposals are not flexible enough to precisely characterize such dynamicity.

\subsubsection{Machine Learning in V2X}
Several bodies of prior work have been found on application of machine learning to V2X networking: (i) detection of anomaly and misbehaviors for security of a V2X network \cite{icc19}\cite{elephant18}; (ii) beam selection \cite{tnet18} and situational awareness \cite{asilomar18} in millimeter wave bands; and (iii) transmission path optimization via obstruction detection by training using satellite images \cite{lett20}.

However, no contribution has been found on adapting the BSM prioritization and networking lightening. We highlight the significance of the network load lightening for the following two reasons: (i) there is increasing demand for V2X traffic as more connected and autonomous vehicles are emergent on the road; (ii) second, the spectrum contention for V2X communications are getting more severe especially in the 5.9 GHz band in the United States, due to the recent spectrum re-alignment by the Federal Communications Commission (FCC) \cite{nprm}.

\begin{figure}
\centering
\includegraphics[width = \linewidth]{figs/fig_scatter}
\caption{Distribution of vehicles on $\mathbb{R}^2$ (with $l = w = $ 1e3 m and $\lambda = $ 15e-3 vehicles/m$^2$)}
\label{fig_scatter}
\end{figure}

\section{System Model}\label{sec_system_model}
\subsubsection{Network}
In the vehicular network model presented in this paper, a vehicle is assumed to be mobile. The network is completely \textit{distributed}, \textit{i.e.}, vehicular ad-hoc network (VANET), in which no central coordinator node nor infrastructure exists. Specifically, a \textit{distributed} type of V2X networks is assumed such as (i) DSRC \cite{11pstd} and (ii) mode 4 of cellular V2X (C-V2X) \cite{lbt19}, both of which are based on listen-before-talk (LBT). Also, exchange of information among vehicles is achieved via \textit{periodic broadcast} of BSMs.

Also, a network is \textit{fully connected}: every node is supposed to be equipped with communication functionality and hence is able to broadcast it whenever needed.

This system model can generally suit currently operating VANET systems in practice including IEEE 802.11-based system such as DSRC \cite{11pstd} and 802.11bd \cite{80211bd}. The model can also be applied to C-V2X mode 4 where the nodes are connected directly in a distributed manner without going through the network core, e.g., sidelink-based broadcast or groupcast as defined in the 3rd Generation Partnership Project (3GPP) Release 16 \cite{tr22886}. Commonly in the standards, a basic safety message (BSM) is \textit{broadcast without any feedback available}. This paper assumes 10 BSMs being broadcast per second.

\subsubsection{Geometry}
As shown in Fig. \ref{fig_scatter}, a two-dimensional space $\mathbb{R}^2$ is defined with the length and width of $l$ and $w$ meters (m), respectively. In order to capture a more dynamic and realistic movement of nodes in a vehicular network, this system model considers \textit{no separation of lanes}. Notice that such a generalized model enables the subsequent analyses more widely applicable \cite{access19}. Furthermore, to consider the most generic vehicle movement characteristic, this model assumes that every vehicle can move in any direction, which enables the system to capture every possible movement scenario including flight of UAVs, lane changing, intersection, and pedestrian walking.

The distribution of the nodes follows Poisson point process (PPP). This paper considers a homogeneous PPP $\Phi$ with the vehicle density $\lambda$. As such, the position of each node is uniformly distributed on each axis of X and Y between $\left[0, l\right]$ and $\left[0, w\right]$, respectively \cite{uniform}.

\section{Proposed RL Framework}\label{sec_proposed}
This section articulates the proposed RL algorithm enabling a vehicle to autonomously update its accident risk according to the environment.

As Fig. \ref{fig_overview} illustrates, in each time slot, a bandit (i.e., a vehicle) makes a selection among arms accounting the ``context'' that has been observed in the time slot. Each arm selection leads to quantification of (i) an accident risk level, (ii) backoff counter, (iii) successful transmission or EXP, and (iv) a reward. The vehicle learns from the rewards collected in the arms to make the choice of an arm in the current iteration. Over time, the vehicle's aim is to collect enough information about how the contexts and arms are mapped to each other, so that it can predict the next best arm to play by looking at the incoming context.

\subsection{Problem Formulation}
\subsubsection{MAB Problem}
In essence, the proposed RL framework is defined to learn $\mathbf{w}_{i} = \left( w_{v}, w_{d} \right)$ at each vehicle $i$ within a time slot $t$, in accordance with a context $\mathbf{c}_{i}$.

If vehicle $i$ (the bandit) knew the expected weight $\mathbf{w}_{i}$ for each context $\mathbf{c}_{i,t}$ in time slot $t$ a priori, it could simply (i) evaluate the accident risk for the given context based on (\ref{eq_AR}) and (ii) adjust its transmission priority in the network. However, since the vehicle $i$ does not know the environment, it has to learn the weight $\mathbf{w}_{i}$ over time.

In order to learn these values, vehicle $i$ has to try out different values for weight $\mathbf{w}_{i}$ on different contexts over time, which forms a \textit{MAB problem}. At the same time, it should ensure that those $\mathbf{w}_{i}$'s already proven to be accurate are used sufficiently often. Hence, the vehicle has to find a tradeoff between ``exploring'' $\mathbf{w}_{i}$'s, of which it has little knowledge, and ``exploiting'' the proven $\mathbf{w}_{i}$'s for high average learning performance. As such, the selection of the learning algorithm depends on the history of $\mathbf{w}_{i}$'s that have been selected in previous periods and the corresponding observed rewards, $\mathbf{r}_{i}$'s.

Let $\mathbf{c}_{i}$ denote a vector of contexts, which is formulated as
\begin{align}\label{eq_c}
\mathbf{c}_{i} = ( c_{v}, c_{d} )
\end{align}
where contexts $c_{v}$ and $c_{d}$ represent speed of each vehicle and inter-vehicle distance at each vehicle, respectively.

\subsubsection{Accident Risk}
We remind that a V2X environment is extremely dynamic presenting a wide variety of factors that cause an accident. By ``dynamic,'' we mean not only the \textit{quantity} of a factor but the \textit{type} of factors as well. As such, each vehicle needs to keep teaching itself how to deal with the factors ``in concert'' in such a dynamic environment. More specifically, in accordance with the environment, the relative significance of each factor should be differentiated. For instance, while a highly speeding vehicle marks a higher crash risk in a highway, a vehicle with a smaller inter-vehicle distance, $d^{\text{min}}_{i}$, poses a higher risk in a denser traffic.

While a plethora of factors must be considered, the proposed RL framework highlights its contribution by remaining flexible for any additional factor. Notice from (\ref{eq_c}) that as more contexts are considered to quantify an accident risk, the context vector $\mathbf{x}_{i}$ gets longer accordingly.

\begin{figure}
\centering
\includegraphics[width = \linewidth]{figs/fig_overview}
\caption{A learning cyle of the proposed RL framework within a time slot $t$}
\label{fig_overview}
\end{figure}

\begin{definition}\label{definition_objective}
(Learning objective: Accident risk). \textit{Accident risk of vehicle $i$ at time instant $t$ is formulated as}
\begin{align}\label{eq_AR}
0 \le \mathcal{AR}_{t,i} = \displaystyle \sum_{j \in \mathcal{S}_{\text{ctx}}} w_{j,i} \le \mathsf{N}_{\text{arm}}
\end{align}
\textit{where $\mathcal{S}_{\text{ctx}}$ denotes the set of contexts (i.e., $\mathcal{S}_{\text{ctx}} = \left\{ v,d \right\}$ as shall be detailed in Definitions \ref{definition_cv} and \ref{definition_cd}). Since the coefficient $w$ for each arm ranges in $\left[ 0,1 \right]$, the quantity of an entire $\mathcal{AR}$ ranges in $\left[0, \mathsf{N}_{\text{arm}}\right]$.}
\end{definition}

\subsubsection{Contexts and Arm-Mapping Policies}
Now, one wishes to formulate the weights $w_{j,i}$ defined in (\ref{eq_AR}), which indicate the level of an accident risk at a vehicle.

The first type of context that this paper adopts is the \textit{variance of speed}. Operating speed at a vehicle has been found to be normally distributed with the mean being the speed limit of the road \cite{nhtsa09}. We take advantage of this premise to proceed to define our first accident risk factor.

The rationale behind this is as follows. First, it becomes dangerous being far from the mean. In other words, not only a too high speed, but is a too low speed dangerous. By defining as ``variance,'' a single quantity can express both. Second, a speed can be obtained at a vehicle without any external information. For instance, a speed limit is an easy number to obtain in many commercial GPS applications (e.g., Google Maps). Reliance on such an easily available quantity increases the applicability of the proposed algorithm.

\begin{definition}\label{definition_cv}
(Context 1: Variance of speed).
\begin{align}\label{eq_cv}
c_{v,i} = \left| v_{i} - v_{\text{ref}}\right|
\end{align}
where $v_{\text{ref}}$ gives the reference speed, i.e., the speed limit of the road in which vehicle $i$ is currently being driven.
\end{definition}

\begin{definition}\label{definition_wv}
(Optimal strategy for context 1: Mapping context $c_{v}$ to arm $w_{v}$). \textit{We remind that context $c_{v,i}$ is designed to be dangerous as the variance $c_{v,i}$ goes further from $v_{\text{ref}}$. As such, this paper proposes that the weight for this context, $w_{v,i}$, as}
\begin{align}\label{eq_wv}
w_{v,i} = \tanh \left( k_{v} c_{v,i} \right), \hspace{0.1 in} c_{v,i} \ge 0
\end{align}
\textit{where $k_{v}$ is a constant that controlls the steepness of $w_{v,i}$.}

\textit{It is interpreted as follows. The weight $w_{v,i}$ becomes neutral (i.e., 0.5) when vehicle $i$ keeps its speed $v_{i} = v_{\text{ref}}$, i.e., $c_{v,i} = 0$. As $w_{v,i}$ gets smaller/greater than 0.5, as the weight $w_{v,i}$ grows.}
\end{definition}

In addition to the speed, this paper adopts the \textit{inter-vehicle distance} as another critical factor in determining an accident risk at a vehicle \cite{dsafe}.

\begin{definition}\label{definition_cd}
(Context 2: Minimum inter-vehicle distance in reference to a safe braking distance).
\begin{align}\label{eq_cd}
c_{d,i} = \frac{d^{\text{min}}_{i}}{d_{\text{ref},i}}
\end{align}
\textit{where $d_{\text{ref},i}$ denotes the safe stopping distance, which is given by \cite{dsafe}}
\begin{align}
d_{\text{ref},i} = r v_{i} \frac{10}{36} + \frac{v_{i}^2}{b}
\end{align}
\textit{where $v_{i}$ is measured in km/h; $r$ gives the driver reaction time (secs), which is set to 1.5 secs in this paper; and $b$ indicates the braking coefficient factor, which is nominally set to 170 assuming dry, level pavement.}

\textit{Also, the first term in (\ref{eq_cd}) is defined as}
\begin{align}
d^{\text{min}}_{i} = \min_{i \neq j, \hspace{0.02 in} j \in \mathcal{S}_{\text{nbr}} } d_{i,j}
\end{align}
\textit{where $\mathcal{S}_{{\rm{nbr}}}$ is the set of neighboring vehicles.}
\end{definition}

\begin{figure}
\centering
\includegraphics[width = \linewidth]{figs/fig_w_vs_c}
\caption{Weights as functions of contexts $c_{v,i}$ and $c_{d,i}$ (with coefficients $k_{v}$ = 2/3 and $k_{d}$ = 5)}
\label{fig_w_vs_c}
\end{figure}

\begin{definition}\label{definition_wd}
(Optimal strategy for context 2: Mapping context $c_{d}$ to arm $w_{d}$). \textit{Recall that context $c_{d,i}$ is designed to be (i) dangerous when $c_{d,i} < 1$, i.e., $d^{\text{min}}_{i} < d_{\text{ref}}$; (ii) safe while being kept $c_{d,i} > 1$, i.e., $d^{\text{min}}_{i} > d_{\text{ref}}$; and (iii) on the border if equal to 1, i.e., $d^{\text{min}}_{i} = d_{\text{ref}}$. In accordance, this paper proposes to allocate the weight for this context, $w_{d,i}$, as a sigmoid, which is formally written as}
\begin{align}\label{eq_wd}
w_{d,i} = \frac{1}{1 + e^{k_{d}\left(c_{d,i}-1\right)}}, \hspace{0.1 in} c_{d,i} \ge 0
\end{align}
\textit{where $k_{d}$ is a constant that controlls the steepness of the weight function.}

\textit{It is interpreted as follows. The weight $w_{d}$ becomes neutral (i.e., 0.5) when vehicle $i$ keeps $d^{\text{min}}_{i} = d_{\text{ref}}$. Now, $w_{d}$ gets (i) smaller than 0.5 as $c_{d}$ goes negative and (ii) greater than 0.5 as $c_{d}$ goes positive.}
\end{definition}

\begin{remark}\label{remark_w_vs_c}
(Weight versus context). \textit{Fig. \ref{fig_w_vs_c} demonstrates the allocation of $w$ versus $c$ as have been proposed in Definitions \ref{definition_wv} and \ref{definition_wd}. As a remark, we remind that (i) $c_{v}$ is considered the safest when the current speed is equivalent to the speed limit of the current road, which allocates $w_{v} = 0$ indicating ``no accident danger''; and (ii) $c_{d}$ becomes safer as the value of $c_{d}$ gets greater.}

\textit{In particular, for context $c_{d}$, $w_{d}$ becomes 0.5 at $c_{d} = 1$, the point at which the context $c_{d}$ becomes not either safe nor dangerous. Also, an inflection point on $w_{d}$ is noteworthy: it shows that one can control the steepness of a weight function by varying the value of the coefficient $k_{d}$.}
\end{remark}

\begin{algorithm}[t]
\SetAlgoLined
Input: $T$, $\textbf{c}_{i}$, $\epsilon$

\For{t = 1, $\cdots$, T}{
\eIf{$t \le T_{\text{trn}}$}{
\textbf{``Training''}\\
Observe context $\mathbf{c}_{i}$\\
Give a random value for an arm $\mathbf{w}_{i}$\\
Compute the danger $\mathcal{AR}$\\
Allocate a backoff value according to $\mathcal{AR}$\\
Compute reward $\mathbf{r}_{i}$
}{
\eIf{rand $\le \epsilon$}{
\textbf{``Explore''}\\
Observe context $\mathbf{c}_{i}$\\
Repeat Lines 5 through 9
}{
\textbf{``Exploit''}\\
Observe context $\mathbf{c}_{i}$\\
Select an arm $\mathbf{w}_{i}$ based on (\ref{eq_wi_xt})\\
Repeat Lines 7 through 9
}
}
}
\caption{Proposed Algorithm for Learning and Network Adaptation}
\end{algorithm}

\subsection{Algorithm for Learning and Network Adaptation}
\subsubsection{Overview of the Proposed Algorithm}
In each discrete time slot $t$, the bandit (i.e., arbitrary vehicle $i$ in the network) performs the proposed algorithm to (i) learn the context in the changing environment and (ii) be given an opportunity of transmitting a BSM in accordance with the result of the learning. A detailed procedure follows as below.

The bandit is given a certain length of \textit{training phase} at the beginning. In a training phase, upon observation of a certain context, an arm is chosen uniformly randomly among the $\mathsf{N}_{\text{arm}}$ arms. The weight for the context selected via the selected arm is used to compute a value for $\mathcal{AR}$.

After the training is completed, the bandit still goes into an \textit{explore} operation with a certain probability $\epsilon$. When exploring, the same operation is performed as in a training phase. However, when the bandit \textit{exploits} with the probability of $1 - \epsilon$, it selects an arm $\mathbf{w}_{t,i}$, referring to the history of $\mathbf{r}$ that has been collected so far, as shall be formulated in (\ref{eq_wi_xt}).

\subsubsection{Explore vs. Exploit}
We formulate the \textit{exploit} and \textit{explore} as follows.
\begin{proposition}\label{proposition_exploit}
($\mathbf{w}_{i}$ when exploited). \textit{When vehicle $i$ exploits at a time instant $t = t_{0}$, the bandit selects the arm satisfying:}
\begin{align}\label{eq_wi_xt}
\mathbf{w}_{\text{xt},i} = \max_{t < t_{0}} \big( \mathbf{w}_{t,i} | \text{c}_{i} \big)
\end{align}
\end{proposition}

One can compare (\ref{eq_wi_xt}) to the bandit's \textit{exploring} among the arms, which can be formally written as
\begin{align}\label{eq_wi_xe}
\mathbf{w}_{\text{xe},i} = \mathcal{U}\left( 1, \mathsf{N}_{\text{arm}}\right)
\end{align}
where $\mathcal{U}\left( a,b \right)$ denotes a discrete uniform distribution with the interval of $\left[a, b\right]$ where $b > a$.

\begin{figure*}
\centering
\begin{subfigure}{.45\textwidth}
\centering
\includegraphics[width = \linewidth]{figs/fig_heatmap_e1}
\caption{A/B testing (i.e., $\epsilon = 1$)}
\label{fig_heatmap_e1}
\end{subfigure}\hspace{0.5 in}
\begin{subfigure}{.45\textwidth}
\centering
\includegraphics[width = \linewidth]{figs/fig_heatmap_e01}
\caption{$\epsilon$-greedy (with $\epsilon = 0.1$)}
\label{fig_heatmap_e01}
\end{subfigure}
\caption{Distribution of reward ($r_{d,i}$) versus arm ($w_{d,i}$) and context ($c_{d,i}$) according to MAB strategy (with $T = 10^{5}$ and $T_{\text{trn}} = T/10$)}
\label{fig_heatmap}
\end{figure*}

\subsubsection{Network Adaptation Based on the Learning Result}
We remind that the proposed protocol intends a more dangerous vehicle (i.e., with a higher $\mathcal{AR}$) to take a higher priority in a multiple access process. Thus, a backoff value in a CSMA process is allocated to be inversely proportional to the value of $\mathcal{AR}$, which is given by

The proposed protocol uses the packet expiration (EXP) rate as the metric associated with the reward in MAB. The EXP rate can be obtained within each vehicle completely autonomously. Specifically, according to the result of the learning, the vehicle (i.e., the bandit) calculates a backoff counter and allocates the counter to itself.

\begin{definition}\label{definition_bo}
(Backoff counter allocation according to $\mathcal{AR}$).
\begin{align}\label{eq_bo}
\mathsf{bo}_{t,i} = \round*{ \frac{1 - \text{CW}}{\mathbb{N}\left[ \mathcal{S}_{\text{ctx}} \right]} \mathcal{AR}_{t,i} + \text{CW} - 1 }
\end{align}
\textit{where $\round{\cdot}$ denotes rounding to the closest integer.}

\textit{Note that function $\mathsf{bo}_{t,i}$ is designed to become (i) the maximum (i.e., $\text{CW} - 1$) at the minimum $\mathcal{AR}_{t,i}$ (i.e., 0) and (ii) the minimum (i.e., $0$) at the maximum $\mathcal{AR}_{t,i}$ (i.e., $\mathsf{N}_{\text{arm}}$).}

\textit{With (\ref{eq_bo}), one can easily distinguish the proposed backoff allocation scheme with the traditional one where a backoff value is allocated completely uniformly.}
\end{definition}

As has been illustrated in Fig. \ref{fig_overview}, a reward is granted according to the result of a packet transmission at vehicle $i$, which is formally written as
\begin{definition}\label{definition_r}
(Reward according to packet transmission success or failure).
\begin{align}\label{eq_r}
\mathbf{r}_{t,i} = 1 - \mathbbm{1}\left(\text{EXP}\right),
\end{align}
\textit{which means that if vehicle $i$ succeeds to transmit a packet (i.e., no EXP), a reward of 1 is granted. We assume that a success in transmission does not guarantee a successful reception, which is a key characteristic in a distributed V2X network such as DSRC and C-V2X mode 4.}
\end{definition}

\subsubsection{Performance Measurement Metrics}\label{sec_proposed_metrics}
We proceed to evaluate the performance of the proposed mechanism. There are two separate metrics for quantifying the performance of (i) learning and (ii) networking.

The learning performance is measured via the \textit{regret of learning} \cite{ms19}. Specifically, it is defined as the expected difference in the amount of received data achieved (i) via an optimal way (i.e., a priori knowledge on the optimal $\left(w_{v}, w_{d}\right)$) and (ii) via the proposed RL algorithm. The regret after $T$ rounds of learning at vehicle $i$ is formally written as
\begin{align}\label{eq_rho}
\rho_{i}\left(T\right) &= \mathbb{E}\left[ \displaystyle \sum_{t=1}^{T} \bigg( \mathbf{r}^{\ast}\left(\mathbf{c}_{t,i}\right) - \mathbf{r}\left(\mathbf{c}_{t,i}\right) \bigg) \right]\nonumber\\
&= \displaystyle \sum_{t=1}^{T} \mathbb{E}\left[ \mathbf{r}^{\ast}\left(\mathbf{c}_{t,i}\right) \right] - \sum_{t=1}^{T} \mathbb{E}\left[ \mathbf{r}\left(\mathbf{c}_{t,i}\right) \right]
\end{align}
where $\mathbf{r} = \left(r_{v}, r_{d}\right)$ and $\mathbf{r}^{\ast}$ denotes the reward associated with an optimal strategy.

For evaluating the networking performance, we define the \textit{probability that a packet is transmitted}, which is denoted by $\tau$ \cite{arxiv19}. Specifically, $\tau$ is the probability that a packet goes through a backoff process, which is (i) initiated by allocation of a backoff counter and (ii) processed longer or shoter depending on the level of competition for the medium within the vehicle's carrier-sense range.

\section{Numerical Results}\label{sec_numerical}

\subsection{Setting}

\begin{table}[hbtp]
\caption{Key parameters}
\centering
\begin{tabular}{ |l|l|}
\hline
\textbf{\cellcolor{gray!30} Parameter (symbol)} & \cellcolor{gray!30}\textbf{Value}\\
\hline\hline
\# arms ($\mathsf{N}_{\text{arm}}$) & 100\\
\# periods ($T$) & 10$^5$ secs\\
Length of training phase ($T_{\text{trn}}$) & $T/10$ secs\\
Reference speed ($v_{\text{ref}}$) & 60 mph\\
CW & 15\\
\hline
\end{tabular}
\label{table_parameters}
\end{table}

This section demonstrates the results of the proposed framework. Specifically, numerical computations were performed to evaluate the proposed (i) RL and (ii) networking performances. The performances were measured at an arbitrary vehicle $i$ in a network. Also, notice that we employ a numerical solve for evaluation of $\tau$, which has been mentioned in Section \ref{sec_proposed_metrics}, due to massive computation load caused by recursiveness in a Markov process \cite{arxiv19}. Key parameters and the values are summarized in Table \ref{table_parameters}.

This section shows the results on context $c_{d,i}$ only, but it is obvious that the same numerical approaches can be applied to the other context $c_{v,i}$, because the proposed RL framework learns the significance $w_{d,i}$ and $w_{v,i}$ independently. Notice that this RL framework design enables even further extension to accommodation of other contexts for learning their weights to more accurately measure the accident risk $\mathcal{AR}$.

\begin{figure}
\centering
\includegraphics[width = \linewidth]{figs/fig_convergence}
\caption{Time convergence of the proposed RL algorithm according to MAB strategy (with $c_{d} = 1.1$)}
\label{fig_convergence}
\end{figure}

\subsection{Results and Discussion}

\subsubsection{Performance of RL according to MAB Strategy}
Fig. \ref{fig_heatmap} provides heatmaps to demonstrate how the reward is distributed versus the context and arm. Figs. \ref{fig_heatmap_e1} and \ref{fig_heatmap_e01} compare different representative MAB strategies--i.e., A/B testing and $\epsilon$-greedy. One can observe from the comparison that a higher probability of ``exploitation'' (i.e., $1 - \epsilon = $ 0 and 0.9 in Figs. \ref{fig_heatmap_e1} and \ref{fig_heatmap_e01}, respectively) yields a clearer tendency that has been observed in Fig. \ref{fig_w_vs_c}.

\subsubsection{Performance of RL in terms of convergence}
Fig. \ref{fig_convergence} sets a certain value of context (i.e., $c_{d} = 1.1$) to focus on the time convergence. As such, the figure presents the convergence of the proposed algorithm according to the same types of MAB strategy as in Fig. \ref{fig_heatmap}. Notice that this plot is a result with a significantly short duration (i.e., $T = 50$ sec) just for observation of the convergence, which, in turn, is with $T_{\text{trn}} = T/10 = 5$ sec. (We remind that this is just an example: in practice, we suggest one to elongate the initial training time as more accident causing contexts are considered in (\ref{eq_AR}).) While an A/B test (blue line) remains to be random all through $T$, an $\epsilon$-greedy strategy (red line) explores only at a given probability (i.e., $\epsilon$ = 0.1). The optimal $w_{d}$ for $c_{d} = 1.1$ can be found from either Fig. \ref{fig_w_vs_c} or \ref{fig_heatmap}.

\begin{figure}
\centering
\includegraphics[width = \linewidth]{figs/fig_tau}
\caption{Probability of a packet transmission at an arbitrary vehicle versus the number of competing vehicles (with CW $= 15$)}
\label{fig_tau}
\end{figure}

Also notice that the $\epsilon$-greedy strategy is not able to find a completely accurate arm as the optimal strategy does, mainly due to (i) a short training phase and (ii) discretization in the categorization of $c_{d}$ into a $w_{d}$. The regret $\rho$ is approximated to be 0.15 in the example shown in Fig. \ref{fig_convergence}. (See (\ref{eq_rho}) for formulation of $\rho$.)

\subsubsection{Performance of Networking}
Fig. \ref{fig_tau} compares $\tau$, the probability that a packet goes through the backoff process and is able to be transmitted, versus $n_{cs}$, the number of vehicles in vehicle $i$'s carrier-sense range. This result shows that the proposed protocol presented in Definition \ref{definition_bo} prioritizes a packet with a higher danger (i.e., a larger value for $\mathcal{AR}$), while a packet with a less accident risk marks a lower transmission probability.

Notice the inverse relationship of EXP with $\tau$, which is given by $\mathbb{P}\left[\text{EXP}\right] = 1 - \tau$. As such, referring to (\ref{eq_r}), one can infer that the reward is piled faster with a larger $\tau$: i.e., $\mathbf{r}_{t,i} \propto \tau$. This relationship forms the chain between the result of a packet transmission linked to the vehicle's learning in the next round.

\section{Conclusions}
This paper proposed (i) a RL framework enabling a vehicle to autonomously adapt itself to dynamic environments and (ii) a V2X networking protocol prioritizing a vehicle with a high accident risk. This paper formulated the RL framework as a contextual MAB problem where the multiple arms are used to test all possible values for the ``weight'' of a certain context. The weight selected in a best arm was used for calculation of the accident risk at a vehicle in a time slot. The vehicle's priority of transmitting the current packet in a network was determined based on the accident risk. Whether the packet could be transmitted or not determined the reward, which was used in the selection of the next arm.

We foresee that this paper's finding will have direct impacts to the practice. For instance, the current autonomous driving functionalities (i.e., Auto Pilot by Tesla) can benefit from this framework. Our results showed that a vehicle could autonomously train itself on a policy of speed and inter-vehicle distance. This trained data could adapt the transmission priority according to the result of learning, which consequently increased the performance of a vehicle at a high crash risk.

Possible future extensions include performance evaluation of the proposed RL framework with more accident causing factors.


\section{Introduction}

\textbf{A major issue for the implementation of large scale superconducting quantum circuits  is the interaction with interfacial two-level system defects (TLS) that leads to qubit relaxation and impedes qubit operation in certain frequency ranges that also drift in time.
Another major challenge comes from non-equilibrium quasiparticles (QPs) that result in qubit dephasing and relaxation. }
\textbf{
In this work we show that such QPs can also serve as a source of TLS.
Using spectral and temporal mapping of TLS-induced fluctuations in frequency tunable resonators, we identify a subset of the general TLS population that are highly coherent TLS with a low reconfiguration temperature $\sim$ 300 mK, and a non-uniform density of states.
These properties can be understood if these TLS are formed by QPs trapped in shallow subgap states formed by spatial fluctutations of the superconducting order parameter $\Delta$.
Magnetic field measurements of one such TLS reveals a link to superconductivity. Our results imply that trapped QPs can induce qubit relaxation.}

It is becoming increasingly evident that the most coherence limiting TLS reside outside the qubit junctions
 on the surface of metals and dielectrics surrounding them \cite{klimov2018,lisenfeld2019,burnett2014,degraaf2017,bilmes2019} and their slowly fluctuating dynamics poses a significant challenge for quantum computation \cite{supremacy,klimov2018,lisenfeld2019,schlor2019, burnett2019}.
A wide range of techniques has been developed to study and mitigate such TLS, by developing qubits
and resonators into probes of a wider range of material properties \cite{schneider2019,degraaf2017,degraaf2018,grabovskij2012,lisenfeld2019,geaney2019,degraaf2015, lisenfeld2015,lisenfeld2015b}.
However, charged surface TLS and paramagnetic impurities also result in a stochastic and locally varying backdrop for QPs in the superconductor itself. It is thus important to consider the implications on the ever-present excess number of non-equilibrium QPs in superconducting quantum devices \cite{Henrique2019, jin2015, devoretPRL} which must be eliminated in order to increase qubit dephasing times. It is also likely that trapped QPs
are responsible for the non-equilibrium relaxation of transmons recently observed \cite{devoretPRL}.
Non-equlibrium QPs can be generated by electromagnetic radiation \cite{devisser2012} and from rare high energy cosmic particles impinging the sample, the latter inducing correlated errors in all qubits of a surface code architecture \cite{moore2012}.

Here we reveal the implications of a spatially fluctuating $\Delta$ on such QPs.
Scanning tunneling spectroscopy studies typically find spatial variations in $\Delta$ of the order of $\delta\Delta = 10-20\%$  in moderately
disordered superconductors \cite{carbillet2019,lemarie2013,Liao2019}.
$\delta\Delta$ gets smaller in very clean superconductors, however,
even in exceptionally clean films with negligible intrinsic magnetic
disorder an everpresent surface spin density
of the order of $\sim5\cdot10^{17}$ m$^{-2}$ \cite{degraaf2017, anton2013, saveskul2019} results in both flux noise \cite{degraaf2017, quintana2017, anton2013, kumar2016, paladino2014,muller2017} and spatially
non-uniform gap suppression \cite{saveskul2019}. In thin films the gap is also non-uniformely suppressed
by the Altshuler-Aronov effect  due to impurity scattering \cite{Altshurer_Aronov, carbillet2019}, as well as thickness variations \cite{ivry2014}, particularly relevant for Al \cite{aldelta}. The clean, low resistance NbN films that we chose to study in this work are characterized by a low carrier density \cite{Wong2017}. In such materials it is very likely that the DOS varies significantly, resulting in gap variations.

Crucially,
we find that the optimal fluctuation able to trap a QP only has a few bound states. As a result a trapped QP behaves similar to `textbook' TLS, as shown in Fig. \ref{fig:sketch}. We hereafter refer to this type of quasiparticle TLS as `qTLS', which coexist with a large number of conventional glassy TLS.  The spatial extent of the gap fluctuation sets the frequency of the qTLS, which means that the density of states (DOS) of these objects is very non-uniform (in contrast to conventional TLS in glasses). We predict a maximum in the qTLS DOS around 6-10 GHz, the frequency range of most qubits (transmons).
Relatively shallow traps also imply that even a modest temperature can reshuffle the QPs.
We expect the underlying qualitative physics of charge defects formed by trapped QPs to be applicable to most materials, including
superconductors in the dirty limit, such as Al used for qubits, but the relevant frequency ranges may differ somewhat due to different material parameters.

\begin{figure*}[t!]
\includegraphics[scale=0.6]{fig1.png}
\caption{qTLS formation. a) Illustration of typical fluctuations of the order parameter $\Delta$ and the formation of wells with multiple bound states in which quasiparticles can be trapped, effectively forming a TLS (`qTLS'). The scale of the trap determines the qTLS frequency and escape rate. b) qTLS are separated by large distances compared to length-scales at which fluctuations in $\Delta$ occur, fluctuations which may originate from impurity scattering or surface spins. Importantly the qTLS are separated in distance from the superconductor oxide surface where conventional TLS defects, adsorbants etc. reside, and form the usual bath of TLS.  c) Numerically evaluated typical elongated fluctuation of the order parameter capable of hosting multiple bound states for quasiparticles, forming a qTLS. Here $\tilde{\xi}=\sqrt{\Delta_0/E} \xi_0$, the scaling assumes $(k_F \xi_0)^{-1}= 0.14$, $\Delta_0/E = 40$, appropriate for our NbN samples.  }
\label{fig:sketch}
\end{figure*}

This model is supported by a series of experimental observations from detailed mapping of parameter fluctuations in high-Q frequency tunable superconducting NbN resonators \cite{sumedh2019}.
Such resonators allow us to map individual TLS in energy and trace them in a broad range of external parameters, such as magnetic or electric field \cite{lisenfeld2019}.
Through analysis of a large number of individual TLS we find:
(i) most qTLS are highly coherent, some with detected linewidths down to $\lesssim 70$ kHz, the resonator linewidth. (ii) The qTLS landscape irreversibly reshuffles at very modest temperatures of $\sim 300$ mK,  (iii) The qTLS DOS appears strongly suppressed at lower frequencies. And (iv) tracking a qTLS in magnetic field reveals a qTLS energy that scale in a way similar to the expected scaling of $\Delta$.
In particular, the presence of a low energy scale  $\lesssim1\text{K},$ is inconsistent with conventional TLS theory in which all scales
are set by chemical energies \cite{phillips}. The low energy scale is instead natural for TLS formed by QPs trapped in regions of locally smaller gap.
We start by outlining the main features of the model that explains these observations.

\addtocontents{toc}{\protect\setcounter{tocdepth}{-1}}
\section*{Results}
To understand the emergence of qTLS from a fluctuating $\Delta = \Delta_0 + \delta\Delta({\bf r})$
we consider a general model that assumes gaussian
fluctuations of the gap, $\left\langle \delta\Delta({\bf r})\delta\Delta({\bf r'})\right\rangle =g\delta({\bf r}-{\bf r'})$, where $g\simeq \delta \Delta\Delta_0\xi_0^2$ is the strength of fluctuations, and $\xi_0$ the BCS coherence length.
The exact origin of these fluctuations is not relevant
for the conclusions. The density of subgap states is determined by
 rare fluctuations of the gap that reduce the QP
energy sufficiently below the gap edge in a uniform material.
The key point is that such fluctuations  can create traps with a shape and depth that allow them to effectively trap QPs.
As the QPs near the gap edge move with momenta close to the Fermi momentum $p_{F}$, it follows that trapped QPs will have similar momenta, which can point in an arbitrary direction.
This freedom results in the optimal trap being very anisotropic, with a shape elongated along the direction
of $p_{F}$. In Fig. \ref{fig:sketch}c we show the typical shape of such a trap obtained through numerical simulations.
In general, the probability to find a fluctuation of depth E scales exponentially with the area $A$ of the fluctuation, $P\sim \exp{(-E^2A)}$. Efficient traps thus have small area, however, a too small dimension along the direction of $p_F$ will forbid the formation of a bound state due to the large kinetic energy of the QP. Therefore optimal traps do not favor isotropic fluctuations.
The wave function of the QP oscillates quickly along the direction of $p_F$. The
ground and excited states in the trap differ in the number of oscillations:
since the total number of oscillations is large it is not surprising
that each trap typically contains more than one bound state.

\begin{figure*}[t!]
\includegraphics[scale=0.63]{fig2.png} \caption{Spectral \& temporal fluctuations of the internal quality factor and
spectral reconfiguration of TLS. (a) Initial measurement taken at
10 mK and $\langle n\rangle\approx40$. For each pixel the resonator
line-shape is measured and fitted to extract the internal quality
factor $Q_{i}$.
The measurement is repeated as the resonator is tuned in frequency (the typical frequency resolution is $\approx 50$ kHz), this measurement
vs detuning is then repeated over time. The color scale shows the normalised
variation $\delta Q_{i}=(Q_{i}-\langle Q_{i}\rangle)/\langle Q_{i}\rangle$.
(b) same measurement performed after
(a) with the temperature
set to 300 mK. (c) the same measurement was repeated again after returning
to 10 mK. The red arrows indicate a broad TLS jumping into the measurement window. Dashed boxes indicates two strong
TLS. For repeated experiments see \cite{suppl}. d) The time-averaged data from panels (a)-(c) respectively.}
\label{fig:spectraltemporal}
\end{figure*}

Due to the presence of excited bound states, a QP in such a
trap forms an effective TLS with typical energy splitting $\hbar\omega_{\text{qTLS}}=\zeta\left|\widetilde{E}\right|$
where $\widetilde{E}$ is the energy of the trapped QP as measured
from the gap edge and  $\zeta\approx0.7$ is found from theory \cite{suppl}.
One consequence of these traps is that, in contrast to conventional TLS, the DOS of qTLS is not constant.
A large $\omega_{qTLS}$ implies very deep traps that are exponentially
rare. We find the density
\begin{equation}
\rho_{\text{qTLS}}(\omega)\simeq \nu(\widetilde{E}) = \frac{1}{\xi_{0}^{2}\Delta_{0}}\text{exp}\left[-\frac{\eta}{g}\frac{\xi_{0}^{2}\Delta_{0}^{2}}{\sqrt{k_{F}\xi_{0}}}\left(\frac{\hbar\omega}{\zeta\Delta_{0}}\right)^{5/4}\right],
\end{equation}
where $\eta\approx 10$. Likewise, small $\omega_{qTLS}$ are suppressed since
QPs in shallow traps anihilate each other efficiently, leading to
\begin{equation}
\rho_{\text{qTLS}}(\omega) \leq \nu_0(\widetilde{E}) = \frac{\sqrt{2k_{F}\xi_{0}}}{\xi_{0}^{2}\Delta_{0}[ \text{ln}(\tau_{\text{exp}} \Delta_0/\hbar) ]^2}\left({\frac{\hbar\omega}{\zeta\Delta_{0}}}\right)^{3/4},
\end{equation}
in the absence of QP generation. Here $\tau_{\text{exp}}$ is an experimental timescale.

Due to the spatial asymmetry, the transition between the lowest and
first excited state in each trap is expected to have a significant electric dipole
moment, resulting in a strong interaction with quantum devices.
In this picture the TLS is formed in a single well, in contrast to the more conventional double-well picture.

In similar films  $\delta\Delta$ constitute 10\% of the total gap \cite{carbillet2019, lemarie2013, Liao2019}, which gives in our devices $\delta\Delta\approx2.5$ K.
At $T=10$ mK tunneling out of such a local minima is exponentially suppressed, however, at temperatures approaching fractions of $\delta\Delta$ there is a finite probability that the QP escapes and either finds another minima or recombines, resulting in a typical escape temperature of the order of $ 200-300$ mK; each event resulting in a different TLS landscape.

The unusual shape of the
traps and the presence of multiple bound states in each trap distinguish clean superconductors from the dirty limit  \cite{Larkin_Ovchinnikov, Lamacraft_Simons, Silva_Ioffe}. In the latter a QP is scattered many times by defects
while moving in a particular direction inside the trap, making
the formation of elongated traps impossible.
The physics of  qTLS traps is  similar to the well studied phenomena of the appearance of states with negative energy
in disordered conductors \cite{Halperin_Lax66,Cardy1978}, with two important differences: electrons at the bottom of the band in a disordered conductor have much smaller momenta, and the wave functions do not have zeros.

\begin{figure*}[t!]
\includegraphics[scale=0.54]{fig3.png} \caption{qTLS Properties. a) Histogram of extracted TLS linewidths (FWHM) obtained
at 10 mK across a series of cooldowns and different resonators in
the frequency range 3-7 GHz.  b)  Expected scaling of the qTLS density of states (blue solid region) bounded by  $\nu(\widetilde{E})$ and $\nu_0(\widetilde{E})$ (see text for details). Not to scale. c) Number of strongly coupled TLS as a function of frequency. The data
points are obtained by counting the number of TLS present in a spectral
and temporal map. Data taken on two different samples (circles and triangles) in seven separate cooldowns (different colors). Duplicate markers at the same frequency are before/after thermal reconfiguration. Multiple markers from the same resonator have been offset somewhat in frequency for better visibility.
Error bars indicate miscounting the number of TLS by one, e.g. if one
TLS happens to be present just outside the measurement range. The solid line is the average of observations, bounded by the standard deviation. The inset shows the same data plotted against the corresponding trap depth from $\Delta_0$. All data obtained at 10 mK and $\langle N\rangle<100$.}
\label{fig:linewidths}
\end{figure*}

We now turn to the experimental work where our observations can be explained by this model.
To be able to study individual TLS and spectral and temporal fluctuations
of superconducting resonator parameters we use kinetic inductance
tunable superconducting resonators with centre frequencies $f_0$ in the
range 3-7 GHz, described in detail in Ref. \cite{sumedh2019}. The resonators
are cooled down to a base temperature of 10 mK in a well filtered
dilution refrigerator. A small dc current ($<1$ mA) is applied to change the kinetic inductance and tune $f_0$.
We measure the transmitted microwave signal, $S_{21}$, using a vector network analyser and
extract the resonator frequency and quality factors Q. In some of the
experiments we also utilise a gate electrode mounted in the lid of
the sample enclosure, such that we can affect the TLS energy through
the applied gate voltage $V_g$: $E_{{\rm TLS}}=\sqrt{\epsilon_{0}^{2}+\gamma^2 V_{g}^{2}}$ \cite{bilmes2019,lisenfeld2019}. Here $\epsilon_{0}$ is the TLS minimum energy and $\gamma$ the TLS voltage coupling strength.

In Fig. \ref{fig:spectraltemporal}a we show the relative variation
of the internal quality factor $\delta Q_{i}=(Q_{i}-\langle Q_{i}\rangle)/\langle Q_{i}\rangle$,
as we tune the resonator frequency. The measurement is continously
repeated over several days to produce a spectral and temporal map
of the variations in loss. This reveals large drops in $\delta Q_{i}$: at detunings of
$3.5$ and $14$ MHz (from 6.995 GHz) TLS dips remain in the same place whereas at
other frequencies both switching-like behaviour and drift can be observed.
We then raise the temperature of the cryostat to 300 mK and repeat
the same measurement (Fig. \ref{fig:spectraltemporal}b), and return to 10 mK (Fig. \ref{fig:spectraltemporal}c). We observe that a strong TLS
remains stable over more than 60 hours in each individual measurement,
but thermal cycling to a mere 300 mK makes another TLS  appear at a different frequency.
This is quite remarkable as the temperature is increased to a value
smaller than the energy level splitting of the TLS itself (7 GHz $\approx$
350 mK), and certainly not expected from conventional TLS-glass physics
in which all scales are set by chemical energies \cite{phillips}.
Separate high power measurements show that these dips are saturable, and not caused by e.g. local variations
in background transmission. We have repeated the same experiment but visiting multiple temperatures
on the way to 300 mK (see  \cite{suppl} for additional data) which confirms that reconfiguration only
happens near 300 mK.

\begin{figure*}[t!]
\includegraphics[scale=0.68]{fig4.png} \caption{A single strongly coupled qTLS in magnetic field. a) Transmission
spectrum of the resonator strongly coupled to a qTLS at selected magnetic
fields, as a function of applied electrostatic gate voltage, showing
the coherent coupling of a single qTLS to the resonator. b) The resonance
frequency as a function of magnetic field (solid blue line) together
with the fitted qTLS minimum energy $\epsilon_{0}$ (red markers).
Other extracted parameters are qTLS-resonator coupling strength $g_0=270$
kHz, damping rate $\gamma_{{\rm qTLS}}=200$ KHz and $\gamma=10$ MHz/V, consistent with $p_0\approx 1$ $\rm e\AA$ in our geometry.}
\label{fig:tlsbfield}
\end{figure*}

Occasionally we observe very dissipative TLS with linewidths in excess of several MHz in our measurement window (red arrow in Fig. \ref{fig:spectraltemporal}c). Here we concentrate instead on the much more coherent TLS that we consistently observe, the narrowest with
line-widths down to $\sim70$ kHz \cite{suppl}, limited by the readout resonator line-width. In Fig. \ref{fig:linewidths}a we show the histogram of
linewidths of the observed TLS, which peaks at values corresponding to coherence times in excess
of several microseconds, much longer than what has been reported for individually resolved TLS in qubits \cite{lisenfeld2019,lisenfeld2015}. The energy leakage caused by  such highly coherent TLS could pose a significant challenge for quantum error correction \cite{Brown2019-1}.
Naturally, qTLS interact with conventional TLS and slow fluctuators that result in parameter fluctuations.

We now turn to the qTLS DOS. Fig. \ref{fig:linewidths}b shows the expected frequency dependence of the qTLS DOS, given by the bounds of eq. (1) and (2).
This is in good agreement with the number of strongly coupled qTLS
that we observe in resonators with different $f_0$, shown in Fig. \ref{fig:linewidths}c; data obtained across several cooldowns and devices. A similar dependence was obtained by tuning the TLS energy by an applied electrostatic field, see \cite{suppl}.
In contrast, for conventional non-interacting TLS in amorphous materials we expect
a DOS $P(E)={\rm {const}}$ \cite{phillips}, while for interacting
TLS we expect a weak pseudogap given by $P(E)=E^{\mu}$ \cite{faoro2015, moshe2020}.  $\mu\approx0.2$ is typically found experimentally \cite{burnett2014,degraaf2018}, which would for $f_{0}=3-6$ GHz only result in a $\sim10\%$
variation in the TLS DOS.
As expected, we find through the temperature dependent shift of $f_0$ which samples all TLS, including the more numerous conventional glassy TLS,  that the intrinsic loss tangent fall within $F\tan\delta_{i}=0.7\pm0.1 \cdot10^{-5}$,
with no observed dependence on $f_0$ \cite{suppl}.

The data in Fig. \ref{fig:linewidths}b has not been
scaled to the surface area occupied by each resonator, which would yield an even stronger non-linear
dependence ($A\propto f_{0}$). If we make the assumption
that the observed qTLS uniformely occupy the surface of the superconductor we get for $f_0=4.9$ GHz an observed density of qTLS of $0.004$ GHz$^{-1}$ ${\rm \mu}$m$^{-2}$, or $1$ GHz$^{-1}$ per $(\sim3000\xi_0)^2$.
In our devices the distance between electrodes is small, resulting
in significant electric fields away from the immediate vicinity
of metal edges. In fact, the maximum observed TLS coupling strengths of
up to $100-200$ kHz are obtained for electric dipoles of size $p_0\sim 1$
${\rm e\lyxmathsym{\AA}}$ (the expected scale of a qTLS dipole moment \cite{suppl} and commonly found also for conventional TLS \cite{lisenfeld2019, sarabi2016}) across a significant part of the device
surface. Near metal edges we would expect coupling strengths well exceeding
1 MHz \cite{suppl}. This we do not observe, implying instead that the qTLS are detected across
most of the device surface.
The above density is much smaller than the total density of TLS we extract
from $F\tan\delta_{i}$. We find a surface density
of $\rho_{{\rm \Delta,A}}\approx50$ GHz$^{-1}$ ${\rm \mu}$m$^{-2}$
of weakly coupled TLS which are mainly responsible for the background
loss and noise \cite{suppl} previously studied in detail \cite{burnett2014, degraaf2018}. These conventional TLS coexist with the qTLS and constitute the much broader background of loss and fluctuations. However,
since the qTLS are located at the metal surface they interact
weakly with the conventional glassy TLS bath which is mainly situated
some distance away, e.g. in the oxide on the superconductor and adsorbants
on top, resulting in higher qTLS coherence.

Finally, to determine if there indeed is a link between the qTLS and
superconductivity we measure a strongly coupled qTLS
in an applied
magnetic field $B$ parallel to the superconducting film, shown in Fig.
\ref{fig:tlsbfield}a. As expected, we observe a quadratic suppression
of $f_0$ owing to the kinetic inductance
increasing as $\Delta$ is suppressed. At the same time
we track the anti-crossing with the qTLS and extract the minimum qTLS
energy $\epsilon_0(B)$. As shown in Fig. \ref{fig:tlsbfield}b
$\epsilon_0$ is also suppressed in magnetic field on a
scale comparable to the weak (few \%) suppression of $\Delta$.
We also note that the conventional TLS bath remains unaffected up to moderate fields \cite{degraafAPL2018}.

High-resolution spectral mapping of TLS in qubits has only been performed in the range $ 5-6$ GHz \cite{klimov2018}.  Below $3$ GHz qubits
in other studies were limited by other loss mechanisms or statistics are insufficient \cite{nguyen2019,quintana2017,yan2015}.
Only Ref. \cite{quintanathesis} hints at a lower TLS density in Al qubits below $\simeq 3$ GHz,
suggesting that the mechanism
responsible for the formation of qTLS could indeed be quite common, and motivating the need to explore lower frequencies.
The critical parameter for the DOS, the strength of fluctuations $g$, is to the best of our knowledge not known for thin Al films. However, it may even be enhanced due to thickness fluctuations contributing to $\delta \Delta$ \cite{ivry2014,aldelta} and the upper frequency for qTLS is bounded by a smaller $\Delta_0$.
We  note that in the dirty limit the details of the theory developed here is not strictly valid. However, the general mechanism would still apply, but the optimal shape of fluctuations is likely to be different.

We now turn to the implications that these qTLS states have for a range of superconducting technologies and for the coherence of qubits. A typical sample of $\sim 1$ cm$^{2}$ size is hit by a high energy
cosmic ray once in every $\sim10$ s, creating a shower of phonons that quickly relax to low frequencies where their relaxation time becomes long. These
phonons produce QPs well above $\Delta$.
Eventually these QPs fall into the traps discussed here where they remain
essentially forever at low temperatures. Such localization of QPs prevents their recombination and equilibration. Those close to a qubit can result in its excitation or relaxation, and hence the QP localization results in a
long period of qubit performance degradation. As this degradation affects all qubits on a
sample it becomes a catastrophic event for quantum computation, a conclusion supported by recent data
on charge noise in qubits \cite{mcdermott}
 and by the studies of QP bursts in granular Al
resonators \cite{Henrique2019} and kinetic inductance detectors \cite{karatsu2019}.

Identifying and eliminating strongly coupled TLS is crucially important for scaling-up of superconducting quantum computing.
All the mechanisms that lead to gap fluctuations are difficult to control in thin films used for resonators and qubits, but our results suggest these must be carefully addressed in order to achieve this goal.

\section*{Acknowledgements}
We thank R. McDermott and M. Gershenson for fruitful discussions. Samples were fabricated in the nanofabrication facilities of the department of microtechnology and nanoscience at Chalmers university. This work was supported by the UK Department of Business, Energy and Industrial Strategy (BEIS), the EU Horizon 2020 research and innovation programme (grant agreement 766714/HiTIMe), the Swedish Research Council (VR) (grant agreements 2016-04828 and 2019-05480), EU H2020 European Microkelvin Platform
(grant agreement 824109), and Chalmers Area of Advance NANO/2018.
J.J.B acknowledges financial support from the Industrial Strategy Challenge Fund Metrology Fellowship as part of BEIS.

\let\oldaddcontentsline\addcontentslin
\renewcommand{\addcontentsline}[3]{