\section{Introduction}
\glsresetall

\begin{figure*}[!ht]
     \centering
     \begin{subfigure}[c]{0.05\textwidth}
         \centering
         \caption{}
         \label{fig:effective_number}
     \end{subfigure}\hfill
     \begin{subfigure}[c]{0.95\textwidth}
         \centering
         \includegraphics[height=0.14\paperheight]{.//effective_number}
     \end{subfigure}
     \begin{subfigure}[c]{0.05\textwidth}
         \centering
         \caption{}
         \label{fig:nonlinear}
     \end{subfigure}\hfill
     \begin{subfigure}[c]{0.95\textwidth}
         \centering
         \includegraphics[height=0.14\paperheight]{.//nonlinear}
     \end{subfigure}
     \begin{subfigure}[c]{0.05\textwidth}
         \centering
         \caption{}
         \label{fig:correlations}
     \end{subfigure}\hfill
     \begin{subfigure}[c]{0.95\textwidth}
         \centering
         \includegraphics[height=0.14\paperheight]{.//correlations}
     \end{subfigure}
     \caption{
       (a) The Vendi Score, \acrshort{VS} in the figure, can be interpreted as the effective number of unique elements in a sample. It increases linearly with the number of modes in the dataset. IntDiv, the expected dissimilarity, becomes less sensitive as the number of modes increases, converging to 1.
      (b) Combining distinct similarity functions can increase the Vendi Score, as should be expected of a diversity metric, while leaving IntDiv unchanged.
      (c) IntDiv does not take into account correlations between features, but the Vendi Score does. The Vendi Score is highest when the items in the sample differ in many attributes, and the attributes are not correlated with each other.
     }
        \label{fig:properties}
\end{figure*}

Diversity is a criterion that is sought after in many areas of \gls{ML}, from dataset curation
and generative modeling to reinforcement learning, active learning, and decoding algorithms. A lack of diversity in datasets and
models can hinder the usefulness of \gls{ML} in many critical applications, e.g. scientific discovery.
It is therefore important to be able to measure diversity.

\parhead{Are we measuring diversity correctly in \gls{ML}?} Several existing metrics for diversity rely on a reference distribution or dataset, which is limiting.
These reference-based metrics define diversity in terms of coverage of the reference.
They assume access to an embedding function--such as a pretrained Inception model~\citep{szegedy2016rethinking}--that maps samples to real-valued vectors.
One example of a reference-based metric is \gls{FID}~\citep{heusel2017gans}, which measures the Wasserstein-2 distance between two Gaussian distributions, one Gaussian fit to the embeddings of the reference sample and another one fit to the embeddings of the sample to be evaluated for diversity.
\gls{FID} was originally proposed for evaluating image \glspl{GAN} but has since been applied to text~\citep{cifka2018eval} and molecules~\citep{preuer2018frechet} using domain-specific neural network encoders.
~\citet{sajjadi2018assessing} proposed a two-metric evaluation paradigm using precision and recall, with precision measuring quality and recall measuring diversity in terms of coverage of the reference distribution. Several other variations of precision and recall have been proposed~\citep{kynkaanniemi2019improved,simon2019revisiting,naeem2020reliable}. Compared to these approaches, the Vendi Score can measure diversity without relying on a reference distribution or dataset and is therefore more general.

Some other existing metrics evaluate diversity using a pre-trained classifier, therefore requiring labeled datasets.
For example, the \gls{IS}~\citep{Salimans2016}, which is mainly used to evaluate the perceptual quality of image generative models,
evaluates diversity using the entropy of the marginal distribution of class labels predicted by an ImageNet classifier.
Another example is \gls{NOM}~\citep{Srivastava2017}, a metric used to evaluate the diversity of \glspl{GAN}.
\gls{NOM} is calculated by using a classifier trained on a labeled dataset and then counting the number of unique labels predicted by the classifier when
using samples from a \gls{GAN} as input.
Both \gls{IS} and \gls{NOM} define diversity in terms of predefined labels, and therefore require knowledge of the ground truth labels and a separate classifier, which limits their applicability.

In some discrete domains, diversity is often evaluated in terms of the distribution of unique features.
For example in \gls{NLP}, a standard metric is n-gram diversity, which is defined as the number of distinct n-grams divided by the total number of n-grams~\citep[e.g.][]{li2016diversity}.
These metrics require an explicit, discrete feature representation.

There are proposed metrics that use similarity scores to define diversity. The most widely used metric of this form is the average pairwise similarity score or the complement, the average dissimilarity.
In text, variants of this metric include \acrshort{PAIRWISE-BLEU}~\citep{shen2019mixture} and \acrshort{D-LEX-SIM}~\citep{fomicheva2020unsupervised}, in which the similarity function is an n-gram overlap metric such as \acrshort{BLEU}~\citep{papineni2002bleu}.
In biology, average dissimilarity is known as IntDiv~\citep{benhenda2017chemgan}, with similarity defined as the Jaccard (Tanimoto) similarity between molecular fingerprints.
Average similarity has some shortcomings, which we highlight in \Cref{fig:properties}.
The figure shows the similarity matrices induced by a shape similarity function and/or a color similarity function. Each entry in the matrices shown is defined to be $1$ if the column and the row share both
shape and color, $0.5$ if they share either shape or color, and $0$ otherwise. As shown in \Cref{fig:properties}, the average similarity--IntDiv in the figure--becomes less sensitive as diversity increases and does not account for correlations between features. This is not the case for the Vendi Score, which accounts for correlations between features and is able to capture the increased diversity resulting from composing two orthogonal similarity functions.

Some other diversity metrics in the \gls{ML} literature fall outside of these categories.
The Birthday Paradox Test~\citep{Arora2018} aims to estimate the size of the support of a generative model, but requires some manual inspection of samples.
\acrshort{GILBO}~\citep{alemi2018gilbo} is a reference-free metric but is only applicable to latent variable generative models.

As discussed above, several attempts have been made to measure diversity in \gls{ML}. However, the proposed metrics can be limited in their applicability in that they require a reference dataset or predefined labels, or are domain-specific and applicable to one class of models. Besides, these metrics don't give their user the agency to decide which type of diversity they want to measure. The existing metrics that don't have those applicability limitations have serious shortcomings when it comes to capturing diversity that we've illustrated in \Cref{fig:properties} and will further demonstrate in our experiments in \Cref{sec:empirical}.

\parhead{Contributions.} We make several contributions in this paper, which we summarize as follows:
\begin{itemize}
	\item We extend ecological diversity to \gls{ML}, and propose the Vendi Score, a metric for evaluating diversity in \gls{ML}.
	We study the properties of the Vendi Score, which provides us with a more formal understanding of desiderata for diversity.
	\item We showcase the flexibility and wide applicability of the Vendi Score and highlight the shortcomings of existing metrics used to measure diversity in different domains.
	\item With the Vendi Score, we are now able to measure the diversity of datasets in \gls{ML}, which opens the door for diversity-informed data augmentation.
\end{itemize}
\parhead{Outline.} In \Cref{sec:method} we define the Vendi Score, state its properties, and discuss its computational complexity. We then apply
it to several datasets, models, and decoding algorithms, in different domains, in \Cref{sec:empirical}. Finally, we conclude and discuss future work in \Cref{sec:discussion}

\section{Measuring Diversity with the Vendi Score}
\label{sec:method}

We now define the Vendi Score, state its properties, and study its computational complexity. (We relegate all proofs of lemmas and theorems to the appendix.)

\subsection{Defining the Vendi Score}

To define a diversity metric in \gls{ML} we look to ecology, the field that centers diversity in its work.
In ecology, one main way diversity is defined is as the exponential of the entropy of the distribution of the species under study~\citep{jost2006entropy}. This is a reasonable index for diversity. Consider a population with a uniform distribution over $N$ species, with entropy $\log(N)$. This population has maximal ecological diversity $N$, the same diversity as a population with $N$ members, each belonging to a different species. The ecological diversity decreases as the distribution over the species becomes less uniform, and is minimized and equal to one when all members of the population belong to the same species.

How can we extend this way of thinking about diversity to \gls{ML}? One naive approach is to define diversity as the exponential of the Shannon entropy of the probability distribution defined by a machine learning model or dataset. However, this approach is limiting in that it requires a probability distribution for which entropy is tractable, which is not possible in many \gls{ML} settings. We would like to define a diversity metric that only relies on the samples being evaluated for diversity. And we would like for such a metric to achieve its maximum value when all samples are dissimilar and its minimum value when all samples are the same. This implies the need to define a similarity function over the samples. Endowed with such a similarity function, we can define a form of entropy that only relies on the samples to be evaluated for diversity. This leads us to the Vendi Score:

\begin{definition}[Vendi Score]\label{def:vendi_score}
  Let $x_1, \ldots, x_n \in {\mathcal{X}}$ denote a collection of samples, let $k: {\mathcal{X}} \times {\mathcal{X}} \to {\mathbb{R}}$ be a positive semidefinite similarity function, with $k(x, x) = 1$ for all $x$, and let ${\bm{K}} \in {\mathbb{R}}^{n \times n}$ denote the kernel matrix with entry $K_{i, j} = k(x_i, x_j)$. Denote by $\lambda_1, \ldots, \lambda_n$ the eigenvalues of ${\bm{K}}/n$. The Vendi Score (${{\text{VS}}}$) is defined as the exponential of the Shannon entropy of the eigenvalues of ${\bm{K}}/n$:
  \begin{align}\label{eq:vendi-score}
    {{\text{VS}}}_k(x_1, \ldots, x_n) =  \exp\left( - \sum_{i=1}^S \lambda_i \log \lambda_i \right),
  \end{align}
where we use the convention $0 \log 0 = 0$.
\end{definition}
To understand the validity of the Vendi Score as a mathematical object, consider the following lemma:
\begin{lemma}\label{lemma:eigen}
  Let ${\bm{K}} \in {\mathbb{R}}^{n \times n}$ denote a positive semi-definite kernel matrix with $K_{i, i} = 1$ for $i \in \{1, \ldots, n\}$.
  Denote by $\Delta_n$ the $(n-1)$-dimensional simplex; its elements are vectors of dimension $n$ whose entries are nonnegative and sum to one. Let ${\bm{\lambda}} = (\lambda_1, \ldots, \lambda_n)$ denote the vector of eigenvalues of ${\bm{K}}/n$.
  Then ${\bm{\lambda}} \in \Delta_n$.
\end{lemma}

The lemma implies that the eigenvalues of ${\bm{K}}/n$ as defined in \Cref{def:vendi_score} are nonnegative and sum to one. The Shannon entropy is therefore well-defined and the Vendi Score is well-defined.
In this form, the Vendi Score can also be seen as the \emph{effective rank} of the kernel matrix ${\bm{K}}/n$. Effective rank was introduced by~\citet{roy2007effective} in the context of signal processing; the effective rank of a matrix is defined as the exponential of the entropy of the normalized singular values.
Effective rank has also been used in machine learning, for example, to evaluate word embeddings~\citep{torregrossa2020correlation} and to study the implicit bias of gradient descent for low-rank solutions~\citep{arora2019implicit}.

The Vendi Score can be expressed directly as a function of the kernel similarity matrix ${\bm{K}}$:
\begin{lemma}\label{lemma:von_neumann}
Consider the same setting as Definition~\ref{def:vendi_score}. Then
  \begin{align}
    {{\text{VS}}}_k(x_1, \ldots, x_n) =  \exp \left (-\tr \left(\frac{{\bm{K}}}{n} \log \frac{{\bm{K}}}{n} \right) \right )
    .
  \end{align}
\end{lemma}
The lemma makes explicit the connection of the Vendi Score to quantum statistical mechanics: the Vendi Score is equal to the exponential of the von Neumann entropy associated with ${\bm{K}}/n$~\citep{bach2022information}.
In quantum statistical mechanics, the state of a quantum system is described by a \emph{density matrix}, often denoted $\rho$.
The von Neumann entropy of $\rho$ quantifies the uncertainty in the state of the system~\citep{wilde2013quantum}.
The normalized similarity matrix ${\bm{K}}/n$ here plays the role of the density matrix.

Our formulation of the Vendi Score assumes that $x_1, \ldots, x_n$ were sampled independently, and so $p(x_i) \approx \frac{1}{n}$ for all $i$.
This is the usual i.i.d setting in \gls{ML} and the setting we study in our experiments. However, we can generalize the Vendi Score to a setting in which
we have an explicit probability distribution over the sample space.
\begin{definition}[Probability-Weighted Vendi Score]\label{def:weighted_vendi_score}
  Let ${\bm{p}} \in \Delta_n$ denote a probability distribution on a discrete space ${\mathcal{X}} = \{x_1, \ldots, x_n\}$, where $\Delta_n$ denotes the $(n-1)$-dimensional simplex.
  Let $k: {\mathcal{X}} \times {\mathcal{X}} \to {\mathbb{R}}$ be a positive semidefinite similarity function, with $k(x, x) = 1$ for all $x$. Let ${\bm{K}} \in {\mathbb{R}}^{n \times n}$ denote the kernel matrix with $K_{i, j} = k(x_i, x_j)$.
Let $\tilde{{\bm{K}}}_{{\bm{p}}} = \mathrm{diag}(\sqrt{{\bm{p}}}){\bm{K}}\mathrm{diag}(\sqrt{{\bm{p}}})$
denote the probability-weighted kernel matrix. Let $\lambda_1, \ldots, \lambda_n$ denote the eigenvalues of $\tilde{{\bm{K}}}_{{\bm{p}}}$.
  The Vendi Score (${{\text{VS}}}$) is defined as the exponential of the Shannon entropy of the eigenvalues of $\tilde{{\bm{K}}}_{{\bm{p}}}$:
  \begin{align}\label{eq:def-general}
    {{\text{VS}}}_k(x_1, \ldots, x_n, {\bm{p}}) =  \exp\left( - \sum_{i=1}^S \lambda_i \log \lambda_i \right).
  \end{align}
\end{definition}
This is the same definition as \Cref{eq:vendi-score}, with the uniform normalization of the similarity matrix ${\bm{K}}$ replaced by normalization with the probability distribution ${\bm{p}}$.
When all elements in the sample are completely dissimilar, the probability-weighted Vendi Score defined in~\Cref{def:weighted_vendi_score} reduces to the exponential of the Shannon entropy of the weighting distribution:
\begin{lemma}\label{lemma:dissimilar}
Let ${\bm{p}} \in \Delta_n$ be a probability distribution over $x_1, \ldots, x_n$ and suppose $k(x_i, x_j) = 0$ for all $i \neq j$. Then the Vendi Score is the exponential of the Shannon entropy of ${\bm{p}}$,
   \begin{itemize}
      	\item[] \centering ${{\text{VS}}}_k(x_1, \ldots, x_n, {\bm{p}}) = \exp\left(H({\bm{p}})\right)$.
      \end{itemize}
\end{lemma}

\subsection{Understanding the Vendi Score}
Figure~\ref{fig:properties} illustrates the behavior of the Vendi Score on simple toy datasets in which each element is defined by a shape and a color, and similarity is
defined to be $1$ if elements share both shape and color, $0.5$ if they share either shape or color, and $0$ otherwise.

First, Figure~\ref{fig:effective_number} illustrates that the Vendi Score is an \textit{effective number}, and can be understood as the effective number of dissimilar elements in a sample.
The value of measuring diversity with effective numbers has been argued in ecology~\citep[e.g.][]{hill1973diversity,patil1982diversity,jost2006entropy} and economics~\citep{adelman1969comment}.
Effective numbers provide a consistent basis for interpreting diversity scores, and make it possible to compare diversity scores using ratios and percentages.
For example, in Figure~\ref{fig:effective_number}, when the number of modes doubles from two to four, the Vendi Score doubles as well.
If we doubled the number of modes from four to eight, the Vendi Score would double once again.

Figures~\ref{fig:nonlinear} and~\ref{fig:correlations} illustrate another strength of the Vendi Score, which is that it accounts for correlations between features.
Given distinct similarity functions $k$ and $k'$, the Vendi Score calculated using the combined similarity function $\frac{1}{2}k(x) + \frac{1}{2}k'(x)$ can be
greater than the average of the individual Vendi Scores if the two similarity functions describe distinct dimensions of variation.
Furthermore, the Vendi Score increases when the items in the sample differ in more attributes, and the attributes become less correlated with each other.

The Vendi Score has several desirable properties as a diversity metric. We summarize them in the following theorem.
\begin{theorem}[Properties of the Vendi Score]\label{thm:properties}
  Consider the same definitions in \Cref{def:vendi_score} and \Cref{def:weighted_vendi_score}.
  \begin{enumerate}
    \item \textbf{Effective number.} If $k(x_i, x_j) = 0$ for all $i \neq j$, then ${{\text{VS}}}_k(x_1, \ldots, x_n)$ is maximized and equal to $n$. If $k(x_i, x_j) = 1$ for all $i, j$, then ${{\text{VS}}}_k(x_1, \ldots, x_n)$ is minimized and equal to $1$.
    \item \textbf{Identical elements.} Suppose $k(x_i, x_j) = 1$ for some $i \neq j$.
      Let ${\bm{p}}'$ denote the probability distribution created by combining $i$ and $j$, i.e. $p'_i = p_i + p_j$ and $p'_j = 0$. Then the Vendi Score is unchanged,
      \begin{itemize}
      	\item[] \centering ${{\text{VS}}}_k(x_1, \ldots, x_n, {\bm{p}}) = {{\text{VS}}}_{k}(x_1, \ldots, x_n, {\bm{p}}')$.
      \end{itemize}
      \item \textbf{Partitioning.} Suppose $S_1, \ldots, S_m$ are collections of samples such that, for any $i \neq j$, for all $x \in S_i, x' \in S_j$, $k(x, x') = 0$.
      Then the diversity of the combined samples depends only on the diversities of $S_1, \ldots, S_m$ and their relative sizes.
      In particular, if $p_i = |S_i|/\sum_{j}|S_j|$ is the relative size of $S_i$ and $H(p_1, \ldots, p_m)$ denotes the Shannon entropy, then the Vendi Score is the geometric mean,
      \begin{itemize}
      	\item[] \centering ${{\text{VS}}}_k(S_1, \ldots, S_m) = \exp(H(p_1, \ldots, p_m))\prod_{i=1}^m {{\text{VS}}}_k(S_i)^{p_i}.$
      \end{itemize}
    \item \textbf{Symmetry.} If $\pi_1, \ldots, \pi_n$ is a permutation of $1, \ldots, n$, then
     \begin{itemize}
      	\item[] \centering ${{\text{VS}}}_k(x_1, \ldots, x_n) = {{\text{VS}}}_k(x_{\pi_1}, \ldots, x_{\pi_n})$.
      \end{itemize}
  \end{enumerate}
\end{theorem}

The effective number property provides a consistent frame of reference for interpreting the Vendi Score: a sample with a Vendi Score of $m$ can be understood to be as diverse as a sample consisting of $m$ completely dissimilar elements.
The identical elements property provides some justification for our use of a sampling approximation: for example, calculating the empirical Vendi Score of a sample of $90$ blue diamonds and $10$ yellow squares is equivalent to calculating the probability-weighted Vendi Score of a sample of one blue spade and one yellow square, with ${\bm{p}} = (0.9, 0.1)$.
The partitioning property is analogous to the partitioning property of the Shannon entropy and means that if two samples are completely dissimilar we can calculate the diversity of the union of the samples using only the diversity of each sample independently and their relative sizes.
The symmetry property means that the Vendi Score will be the same regardless of how we order the rows and columns in the similarity matrix.

\subsection{Calculating the Vendi Score}
Calculating the Vendi Score for a sample of $n$ elements requires finding the eigenvalues of an $n \times n$ matrix, which has a time complexity of $O(n^3)$.
The Vendi Score can be approximated using column sampling methods~\citep[i.e. the Nystr\"{o}m method; ][]{williams2000using}.
However, in many of the applications we consider, the similarity functions we use are inner products between explicit feature vectors $\phi(x) \in {\mathbb{R}}^d$, with $d \ll n$. That is, ${\bm{K}} = {\bm{X}}^{\top}{\bm{X}}$, where ${\bm{X}} \in {\mathbb{R}}^{n \times d}$ is the feature matrix with row ${\bm{X}}_{i, :} = \phi(x_i)$.
The eigenvalues of ${\bm{K}}/n$ are the same as the eigenvalues of the covariance matrix ${\bm{X}}\mX^{\top}/n$, therefore we can calculate the Vendi Score exactly in a time of $O(d^2n + d^3) = O(d^2n)$.
This is the same complexity as existing metrics such as \gls{FID}~\citep{heusel2017gans}, which require calculating the covariance matrix of Inception embeddings.

\section{Experiments}
\label{sec:empirical}
\begin{table}[t]
\centering
\begin{tabular}{lcr}
  \toprule
  Model & IntDiv & \acrshort{VS}\\
  \midrule
  Original & 0.855 & 403.9\\
  \midrule
  \acrshort{AAE} & 0.859 & 501.1\\
  \acrshort{Char-RNN} & 0.856 & 482.4\\
  Combinatorial & 0.873 & 536.9\\
  \acrshort{HMM} & 0.871 & 250.9\\
  \acrshort{JTN} & 0.856 & 489.5\\
  Latent \acrshort{GAN} & 0.857 & 486.4\\
  N-gram & 0.874 & 479.8\\
  \acrshort{VAE} & 0.856 & 475.3\\
  \bottomrule
\end{tabular}
\caption{
\label{tab:molecule_metrics}
IntDiv and \acrshort{VS} for generative models of molecules. The \acrshort{HMM} has one of the highest IntDiv scores, but scores much lower on \acrshort{VS} .
An analysis of $250$ molecules from the \acrshort{HMM} reveals \acrshort{VS} is more accurate in this case. (See \Cref{fig:molecule_kernels}.)
}
\end{table}

 We used the Vendi Score, which we now denote by \acrshort{VS} for the rest of this section, to evaluate the diversity of generative models of molecules, an application where diversity plays an important role in enabling discovery.
We compare \acrshort{VS} to IntDiv, a function of the average similarity:
\begin{align*}
  \mathrm{IntDiv}(x_1, \ldots, x_n) = 1 - \frac{1}{n^2} \sum_{i, j} k(x_i, x_j).
\end{align*}
We found that unlike IntDiv,
\acrshort{VS} is effective at capturing diversity.
We also applied \acrshort{VS} to generative models of images, and decoding algorithms of text, where we found it confirms what we know about diversity in those applications.
We also used \acrshort{VS} to measure mode collapse in \glspl{GAN} and datasets. Unlike the usual metrics for measuring mode collapse, \acrshort{VS} makes it possible to evaluate mode collapse without needing labels.
Furthermore, \acrshort{VS} revealed that current metrics for measuring mode collapse are insufficient in that, unlike \acrshort{VS}, they do not capture fine-grained notions of diversity.
Finally, we used \acrshort{VS} to analyze the diversity of several image, text, and molecule datasets, gaining insights into the diversity profile of those datasets.
(Implementation details are provided in Appendix~\ref{app:implementation_details}.)

\subsection{Evaluating molecular generative models for diversity}
First, we evaluate the diversity of samples from generative models of molecules. For generative models to be
useful for the discovery of novel molecules, they ought to be diverse. The standard diversity metric in this setting is IntDiv.
We evaluate samples from generative models provided in the \acrshort{MOSES} benchmark~\citep{polykovskiy2020moses}, using the first $2500$ valid molecules in each sample.
Following prior work, our similarity function is the Morgan fingerprint similarity (radius 2), implemented in RDKit.\footnote{RDKit: Open-source Cheminformatics. https://www.rdkit.org.}

In Table~\ref{tab:molecule_metrics}, we highlight an instance where IntDiv and \acrshort{VS} disagree: IntDiv ranks the \acrshort{HMM} among the most diverse models, while \acrshort{VS} ranks it as the least diverse.
The reason is illustrated in Figure~\ref{fig:molecule_kernels}: the \acrshort{HMM} has a high IntDiv score because, on average, the \acrshort{HMM} molecules have low pairwise similarity scores,
but there are a number of clusters of identical or nearly identical molecules.
\begin{figure}[t]
  \centering
  \includegraphics[width=\textwidth]{.//molecule_kernels}
  \caption{The kernel matrices for $250$ molecules sampled
    from the \acrshort{HMM}, \acrshort{AAE}, and the original dataset, sorted
    lexicographically by \acrshort{SMILES} string representation.
    The samples have similar IntDiv scores,
    but the \acrshort{HMM} samples score much lower on \acrshort{VS}.
    The figure shows that the \acrshort{HMM} generates a number of exact duplicates.
    \acrshort{VS} is able to capture the \acrshort{HMM}'s lack of diversity while IntDiv cannot.}
\label{fig:molecule_kernels}
\end{figure}

\subsection{Assessing mode collapse in \glspl{GAN} and datasets}
\begin{figure}[h]
  \centering
\includegraphics[width=0.49\textwidth]{.//mnist_modes}\includegraphics[width=0.49\textwidth]{.//mnli_modes}\caption{Detecting mode dropping in image and text datasets.
    We evaluate \acrshort{VS} and IntDiv on datasets containing $500$ examples drawn uniformly from between one and ten classes:
    digits in \acrshort{MNIST} and sentences genres in Multi\acrshort{NLI}.
Compared to IntDiv, \acrshort{VS} increases more consistently with the number of classes.
} \label{fig:mode_dropping}
\end{figure}

 First we examine whether \acrshort{VS} captures mode collapse in a controlled setting, where we have information about the ground truth class distribution.
We simulate mode collapse by sampling equally-sized subsets of two classification datasets, with each subset ${\mathcal{S}}_i$ containing examples sampled uniformly from the first $i$ categories.
We perform this experiment on one image dataset (\acrshort{MNIST}) and one text dataset (Multi\acrshort{NLI};~\citealp{williams2018broad}), using simple similarity functions.

\acrshort{MNIST} consists of 28$\times$28-pixel images of hand-written digits, divided into ten classes.
The similarity score we use is the cosine similarity between pixel vectors: $k(x, x') = \langle x, x' \rangle / \|x\| \|x'\|$,
where $x, x'$ are $28^2$-dimensional vectors with entries specifying pixel intensities between 0 and 1.
Multi\acrshort{NLI}~\citep{williams2018broad} is a multi-genre sentence-pair classification dataset.
We use the premise sentences from the validation split (mismatched), which are drawn from one of ten genres.
We define similarity using the n-gram overlap kernel:
for a given $n$, the n-gram kernel $k_n$ can be expressed as the cosine similarity between feature vectors $\phi^n(x)$, where $\phi_i^n(x)$ is equal to the number of times n-gram $i$ appears in $x$.
We use the average of $k(x, x') = \frac{1}{4}\sum_{n=1}^4 k_n(x, x')$.

The results (Figure~\ref{fig:mode_dropping}) show that \acrshort{VS} generally increases with the number of classes, even using these simple similarity scores.
In \acrshort{MNIST} (left), \acrshort{VS} increases roughly linearly for the first six digits (0-5) and then fluctuates. This could occur if the new modes are similar to the other modes in the sample, or have low internal diversity.
In Multi\acrshort{NLI} (right), \acrshort{VS} increases monotonically with the number of genres represented in the sample.
In both cases, \acrshort{VS} has a stronger correlation with the number of modes compared to IntDiv.

Mode collapse is a failure mode of \glspl{GAN} that has received a lot of attention from the \gls{ML} community~\citep{Metz2017, dieng2019prescribed}.
The main metric for measuring mode collapse, called \acrlong{NOM}(\gls{NOM}), can only be used to assess mode collapse for \glspl{GAN} trained on a labelled dataset.
\gls{NOM} is computed by training a classifier on the labeled training data and counting the number of unique classes that are predicted by the trained classifier for the generated samples.
In Table~\ref{tab:mode_dropping}, we evaluate two models that were trained on the Stacked\acrshort{MNIST} dataset, a standard setting for evaluating mode collapse in \glspl{GAN}.
Stacked\acrshort{MNIST} is created by stacking three \acrshort{MNIST} images along the color channel, creating $1000$ classes corresponding to $1000$ number of modes.

\begin{table*}[ht]
\small
\centering
\begin{tabular}{l c c c c}
  \toprule
Model & \acrshort{NOM} & Mode Div. & \acrshort{VS}\\
\midrule
Self-cond. \gls{GAN} & 1000 & 921.0 & 746.7 \\
Pres\gls{GAN} & 1000 & 948.7 & 866.6 \\
\midrule
Original & 1000 & 950.8 & 943.7 \\
\bottomrule
\end{tabular}
\caption{
\label{tab:mode_dropping}
\acrshort{VS} captures a more fine-grained notion of diversity than \acrlong{NOM}(\acrshort{NOM}).
Although Pres\gls{GAN} and Self-cond.\gls{GAN} both capture all the $1000$ modes, \acrshort{VS} reveals that Pres\gls{GAN} is more diverse than Self-cond.\gls{GAN}
and that they both are less diverse than the original dataset. }
\end{table*}

In prior work, mode collapse is evaluated by training an \acrshort{MNIST} classifier
and counting the number of unique classes that are predicted for the generated samples.
We adapt this approach and
we calculate \acrshort{VS} using the probability product kernel~\citep{jebara2004probability}: $k(x, x') = \sum_{y} p(y \mid x)^{\frac{1}{2}} p(y \mid x')^{\frac{1}{2}}$, where the class likelihoods are given by the classifier.
We compare Pres\acrshort{GAN}~\citep{dieng2019prescribed} and Self-conditioned \gls{GAN}~\citep{liu2020selfconditioned}, two \glspl{GAN} that are known to capture all the modes.
Table~\ref{tab:mode_dropping} shows that Pres\acrshort{GAN} and Self-conditioned \gls{GAN} have the same diversity according to number of modes, they capture all $1000$ modes.
However, \acrshort{VS} reveals a more fine-grained notion of diversity, indicating that Pres\acrshort{GAN} is more diverse than Self-conditioned \gls{GAN} and that both are less diverse than the original dataset.
One possibility is that \acrshort{VS} is capturing imbalances in the mode distribution.
To see whether this is the case, we also calculate what we call \emph{Mode Diversity}, the exponential entropy of the predicted mode distribution: $\exp H(\hat{p}(y))$, where $\hat{p}(y) = \frac{1}{n} \sum_{i=1}^n p(y \mid x_i)$. The generative models score lower on \acrshort{VS} than Mode Diversity, indicating that low scores cannot be entirely attributed to imbalances in the mode distribution.
Therefore \acrshort{VS} captures more aspects of diversity, even when we are using the same representations as existing methods.

\subsection{Evaluating image generative models for diversity}
\label{sec:images}

\begin{table*}[!t]
  \small
  \setlength{\tabcolsep}{4pt}
    \begin{minipage}{0.49\linewidth}
      \centering
        \begin{tabular}{l r r r r r}
          \toprule
          Model & \acrshort{IS}$_\uparrow$ & \acrshort{FID}$_\downarrow$ & Prec$_\uparrow$ & Rec$_\uparrow$ & \acrshort{VS}$_\uparrow$ \\
          \\
          \multicolumn{6}{l}{\textbf{\acrshort{CIFAR}-10}} \\
          \midrule
          Original &  &  &  &  & 19.50 \\
          \acrshort{VDVAE} & 5.82 & 40.05 & 0.63 & 0.35 & 12.87 \\
          DenseFlow & 6.01 & 34.54 & 0.62 & 0.38 & 13.55 \\
          \acrshort{IDDPM} & 9.24 & 4.39 & 0.66 & 0.60 & 16.86 \\
          \\
           \multicolumn{5}{l}{\textbf{LSUN Cat 256$\times$256}} \\
          \midrule
          Original &  &  &  &  & 15.12 \\
          Style\acrshort{GAN}2 & 4.84 & 7.25 & 0.58 & 0.43 & 13.55 \\
          \acrshort{ADM} & 5.19 & 5.57 & 0.63 & 0.52 & 13.09 \\
          \acrshort{RQ-VT} & 5.76 & 10.69 & 0.53 & 0.48 & 14.91 \\
        \end{tabular}
\end{minipage}\hfill
    \begin{minipage}{.49\linewidth}
      \centering
        \begin{tabular}{ r r r r r}
          \toprule
         \acrshort{IS}$_\uparrow$ &\acrshort{FID}$_\downarrow$ & Prec$_\uparrow$ & Rec$_\uparrow$ & \acrshort{VS}$_\uparrow$ \\
          \\
          \multicolumn{5}{l}{\textbf{ImageNet 64$\times$64}} \\
          \midrule
            &  &  &  & 43.93 \\
           9.68 & 57.57 & 0.47 & 0.37 & 18.04 \\
          5.62 & 102.90 & 0.36 & 0.17 & 12.71 \\
         15.59 & 19.24 & 0.59 & 0.58 & 24.28 \\
          \\
           \multicolumn{5}{l}{\textbf{LSUN Bedroom 256$\times$256}} \\
          \midrule
            &  &  &  & 8.99 \\
          2.55 & 2.35 & 0.59 & 0.48 & 8.76 \\
           2.38 & 1.90 & 0.66 & 0.51 & 7.97 \\
           2.56 & 3.16 & 0.60 & 0.50 & 8.48 \\
        \end{tabular}
\end{minipage}
\caption{
\label{tab:comparing_image_models}
\acrshort{VS} generally agrees with the existing metrics. On low-resolution datasets (top left and top right) the diffusion model performs better on all of the metrics.
On the \acrshort{LSUN} datasets (bottom left and bottom right), the diffusion model gets the highest quality scores as measured by \acrshort{IS}, but scores lower on \acrshort{VS}.
No model matches the diversity score of the original dataset they were trained on.
}
\end{table*}

\begin{figure*}[t!]
     \centering
     \begin{subfigure}[c]{0.5\linewidth}
         \centering
         \includegraphics[width=\linewidth]{.//lsun_cat_components_inception}
     \end{subfigure}\hfill
     \begin{subfigure}[c]{0.5\linewidth}
         \centering
         \includegraphics[width=\linewidth]{.//lsun_cat_components_pixels}
     \end{subfigure}
\caption{
\label{fig:lsun_cat_comparing_kernels}
The choice of similarity function provides a way of specifying the notion of diversity that is relevant for a given application.
We project \acrshort{LSUN} Cat images along the top eigenvectors of the kernel matrix, using either Inception features or pixels to define similarity.
Inception similarity provides a form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel captures visual similarity.
For each eigenvector ${\bm{u}}$, we show the four images with the highest and lowest entries in ${\bm{u}}$.
For both kernels, every similarity score is positive, so all entries in the top eigenvector have the same sign; the images with the
highest weights in this component have the highest expected similarity scores.
The remaining eigenvectors partition the images along different dimensions of variation.
}
\end{figure*}
 We now evaluate several recent models for unconditional image generation, comparing the diversity scores with standard evaluation metrics, \gls{IS}~\citep{Salimans2016}, \gls{FID}~\citep{heusel2017gans}, Precision~\citep{sajjadi2018assessing}, and Recall~\citep{sajjadi2018assessing}.
The models we evaluate represent popular classes of generative models, including a variational autoencoder~\citep[\acrshort{VDVAE}; ][]{child2020very}, a flow model~\citep[DenseFlow; ][]{grcic2021densely}, diffusion models (\acrshort{IDDPM}, \citealp{nichol2021improved}; \acrshort{ADM}~\citealp{dhariwal2021diffusion}), \gls{GAN}-based models~\citep{karras2019style,karras2020analyzing}, and an auto-regressive model~\citep[\acrshort{RQ-VT}; ][]{lee2022autoregressive}.
The models are trained on \acrshort{CIFAR}-10~\citep{Krizhevsky2009}, ImageNet~\citep{russakovsky2015imagenet}, or two categories from the \acrshort{LSUN} dataset~\citep{yu2015lsun}.
We either select models that provide precomputed samples, or download publicly available model checkpoints and sample new images using the default hyperparameters. (More details are in Appendix~\ref{app:implementation_details}.)

The standard metrics in this setting use a pre-trained Inception ImageNet classifier to map images to real vectors.
Therefore, we calculate \acrshort{VS} using the cosine similarity between Inception embeddings, using the same 2048-dimensional representations used for evaluating \acrshort{FID} and Precision/Recall.
As a result, the highest possible similarity score is 2048.
The baseline metrics are reference-based, with the exception of \acrshort{IS}.
\acrshort{FID} and \acrshort{IS} capture diversity implicitly. Recall was introduced to capture diversity explicitly, with diversity defined as coverage of the reference distribution.

The results of this comparison are in Table~\ref{tab:comparing_image_models}.
On the lower resolution datasets (top left and top right), \acrshort{VS} generally agrees with the existing metrics.
On those datasets the diffusion model performs better on all of the metrics.
On the \acrshort{LSUN} datasets (bottom left and bottom right), the diffusion model gets the highest quality scores as measured by \acrshort{IS}, but scores lower on \acrshort{VS}.
No model matches the diversity score of the original dataset they were trained on.
In addition to comparing the diversity of the models, we can also compare the diversity scores between datasets:
as a function of Inception similarity, the most diverse dataset is ImageNet 64$\times$64, followed by \acrshort{CIFAR}-10,
followed by \acrshort{LSUN} Cat, and then \acrshort{LSUN} Bedroom.
This agrees with the intuition that the Inception network will induce the highest-rank embedding space for datasets that most resemble the training distribution.

\acrshort{VS} should be understood as the diversity with respect to a specific similarity function, in this case, the Inception ImageNet similarity.
We illustrate this point in Figure~\ref{fig:lsun_cat_comparing_kernels} by comparing the top eigenvalues of the kernel matrices corresponding to the
Inception similarity and the pixel similarity, which we calculate by resizing the images to 32$\times$32 pixels and taking the cosine similarity between pixel vectors.
Inception similarity provides a form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel provides a
simple form of visual similarity, with components corresponding to broad differences in lightness, darkness, and color.

\subsection{Evaluating decoding algorithms for text for diversity}
\label{sec:text}
We evaluate diversity on the \acrshort{MS COCO} image-captioning dataset~\citep{lin2014microsoft}, following prior work on diverse text generation~\citep{vijayakumar2018diverse}.
In this setting, the subjects of evaluation are diverse decoding algorithms rather than parametric models.
Given a fixed conditional model of text $p(x \mid c)$, where $c$ is some conditioning context, the aim is to identify a ``Diverse N-Bet List'', a list of sentences that have high likelihood but are mutually distinct.
The baseline metric we compare to is n-gram diversity~\citep{li2016diversity},
which is the proportion of unique n-grams divided by the total number of n-grams.
We define similarity using the n-gram overlap kernel:
for a given $n$, the n-gram kernel $k_n$ is the cosine similarity between bag-of-n-gram feature vectors.
We use the average of $k_1, \ldots, k_4$.
This ensures that \acrshort{VS} and n-gram diversity are calculated using the same feature representation.

Each image in the validation split has five captions written by different human annotators, and we compare these with captions generated by a
publicly available captioning model trained on this dataset~\footnote{https://huggingface.co/ydshieh/vit-gpt2-coco-en-ckpts}.
For each image, we generate five captions using either beam search or \gls{DBS}~\citep{vijayakumar2018diverse}.
\gls{DBS} takes a parameter, $\gamma$, called the diversity penalty, and we vary this between $0.2$, $0.6$, and $0.8$.

Table~\ref{tab:caption_diversity} shows that all diversity metrics increase as expected, ranking beam search the lowest, the human
captions the highest, and \gls{DBS} in between, increasing with the diversity penalty.
The human diversity score of $4.88$ can be interpreted as meaning that, on average, all five human-written captions
are almost completely dissimilar from each other, while beam search effectively returns only three distinct responses for every five that it generates.

\begin{table}[t]
  \centering
\begin{tabular}{l c c c}
  \toprule
Source & \acrshort{BLEU}-4 & N-gram div. & \acrshort{VS} \\
\midrule
Human &  & 0.82 & 4.88\\
  \midrule
Beam Search & 0.27 & 0.42 & 3.00\\
\acrshort{DBS} $\gamma=0.2$ & 0.25 & 0.49 & 3.16\\
\acrshort{DBS}  $\gamma=0.5$ & 0.22 & 0.63 & 4.14\\
\acrshort{DBS}  $\gamma=0.8$ & 0.21 & 0.68 & 4.37\\
\bottomrule
\end{tabular}
\captionof{table}{
\label{tab:caption_diversity}
Quality and diversity scores for an image captioning model using Beam Search or \acrlong{DBS}(\acrshort{DBS}), varying the diversity penalty $\gamma$.
\acrshort{BLEU}-4 measures n-gram overlap with the human-written reference captions, a proxy for quality.
Increasing $\gamma$ leads to higher diversity scores, as expected and as captured by both N-gram div. and \acrshort{VS}, but leads to a lower quality score.
}
\end{table}

\begin{figure}[t]
\centering
\resizebox{0.5\linewidth}{!}{\includegraphics{.//caption_diversity.pdf}
}
\caption{
\label{fig:caption_diversity}
\acrshort{VS} is correlated with N-gram diversity. Each point represents a group of five captions for a particular image.
}
\end{figure}

 \begin{figure}[h!]
  \small
\begin{minipage}[t]{.48\textwidth}
  High Vendi Score, low n-gram diversity:
  \it{}
  \begin{itemize}[leftmargin=*]
\item two men in bow ties standing next to steel rafter.
\item several men in suits talking together in a room.
\item an older \textbf{man in a tuxedo} standing next to a younger \textbf{man in a tuxedo} wearing glasses.
\item two men wearing tuxedos glance at each other.
\item older \textbf{man in tuxedo} sitting next to another younger \textbf{man in tuxedo}.
  \end{itemize}
\end{minipage}\hfill
\begin{minipage}[t]{.48\textwidth}
  Low Vendi Score, high n-gram diversity:
  \it{}
  \begin{itemize}[leftmargin=*]
\item a man and woman cutting a slice of cake by trees.
\item a couple of people standing cutting a cake.
\item \textbf{the dork with the earring stands next to the asian beauty who is way out of his league.}
\item a newly married couple cutting a cake in a park.
\item a bride and groom are cutting a cake as they smile.
\end{itemize}
\end{minipage}
\caption{
\label{fig:caption_diversity_example}
Two sets of captions that receive different ranks according to \acrshort{VS} and n-gram diversity.
We manually highlight some features contributing to the different scores.
On the left, a sentence contains repeated n-grams, which is penalized by n-gram diversity even though the sentences in the bunch are diverse.
On the right, one long outlier sentence contributes most of the n-grams for this group of sentences, greatly increasing the n-gram diversity even though the
other sentences in the group are not diverse. \acrshort{VS} doesn't have those limitations.
}
\end{figure}

\begin{figure}[!hbpt]
\begin{minipage}{\linewidth}
     \centering
      \includegraphics{.//goodscents_categories}
\captionof{figure}{
\label{fig:goodscents_categories}
The Vendi Scores of samples containing $500$ molecules with different scent labels, calculating diversity using two similarity functions:
Morgan molecular fingerprint similarity, and the similarity between odor descriptors. Each molecule is associated
with one or more human-written tags (e.g. ``floral, fruity, green, sweet''), and the odor-descriptor similarity is the cosine similarity between binary tag indicator vectors.
}
\end{minipage}
\end{figure}

\begin{figure}[ht!]
\begin{minipage}{\linewidth}
     \centering
      \includegraphics[width=\linewidth]{.//molecule_categories}
\captionof{figure}{
\label{fig:molecule_categories}
The scent categories in Goodscents dataset with the highest (top) and lowest (bottom) \gls{VS}, using the molecular fingerprint similarity.
We show $100$ examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores.
}
\end{minipage}
\end{figure}

\begin{figure}[!hbpt]
     \centering
\includegraphics[width=\linewidth]{.//celeba_categories_attr}
\caption{
\label{fig:celeba_categories}
The attributes in \acrshort{CelebA} with the lowest (left) and highest (right) \acrshort{VS}, defining similarity as the cosine similarity
between either Inception embeddings (top), pixel vectors (middle), or binary attribute vectors (bottom).
We show $100$ examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores according to the corresponding kernel.
These examples illustrate the importance of the choice of similarity function for defining the notion of diversity that is relevant for a given application. However, almost all choices of similarity functions show that the \acrshort{CelebA} dataset is more diverse for men than for women.
}
\end{figure}
 In Figure~\ref{fig:caption_diversity}, we plot the relationship between \acrshort{VS} and n-gram diversity using the \acrshort{MS COCO} captioning data
and the n-gram overlap kernel described in Section~\ref{sec:text}.
The figure shows that \acrshort{VS} is highly correlated with n-gram diversity, which is expected given that our similarity function is based on n-gram overlap.
Nonetheless, there are some data points that the metrics rank differently.
This is because n-gram diversity conflates two properties: the diversity of n-grams within a single sentences and the n-gram overlap between sentences.
We highlight two examples in Figure~\ref{fig:caption_diversity_example}.
In general, the instances that n-gram diversity ranks lower compared to \acrshort{VS} contain individual sentences that repeat phrases.
On the other hand, n-gram diversity can be inflated in cases when one sentence in the sample is much longer than the others, even if the other sentences are not diverse.

\subsection{Diagnosing datasets for diversity}
\label{sec:diagnosing_datasets}
\paragraph{Molecules.}
We evaluate the diversity of molecules in the GoodScents database of perfume materials\footnote{http://www.thegoodscentscompany.com/}, which
has been used in prior \gls{ML} research on odor modeling~\citep{sanchez2019machine}.
We use the standardized version of the data provided by the Pyrfume library~\footnote{https://pyrfume.org/}.
Each molecule in the dataset is labeled with one or more odor descriptors (for example, ``clean, oily, waxy'' or ``floral, fruity, green'').
We form groups of molecules corresponding to the seven most common odor descriptors, with each group consisting of $500$ randomly sampled molecules.
We evaluate \acrshort{VS} using two similarity functions: the Morgan fingerprint similarity (radius $2$), and the
similarity between odor descriptors, defined as the cosine similarity between descriptor indicator vectors $\phi(x)$, where $\phi_i(x)$ is equal to one
if descriptor $i$ is associated with molecule $x$ and zero otherwise.
The diversity scores are plotted in Figure~\ref{fig:goodscents_categories}.
The molecular diversity score and the odor-descriptor diversity scores are correlated, meaning that words like ``woody'' and ``green'' are used to
describe molecules that vary in molecular structure and also elicit diverse odor descriptions,
while words like ``waxy'' and ``fatty'' are used for molecules that are similar to each other and elicit similar odor descriptions.
For example, the word ``green'' appears in tag sets such as
``aldehydic, citrus, cortex, green, herbal, tart'' and ``floral, green, terpenic, tropical, vegetable, woody'', whereas the word ``waxy'' tends to co-occur with the same tags
(``fresh, waxy'';
``fresh, green, melon rind, mushroom, tropical, waxy'';
``fruity, green, musty, waxy'').
Molecules from the categories with the highest and lowest scores are illustrated in Figure~\ref{fig:molecule_categories}.
\paragraph{Images.} We now assess both \acrshort{CIFAR-100} and \acrshort{CelebA} for diversity.
In Figure~\ref{fig:celeba_categories} we calculate \acrshort{VS} for samples from \acrshort{CelebA}.
Images in \acrshort{CelebA} are associated with $40$-dimensional binary attribute vectors.
We use these attributes as an additional similarity score, defining the attribute similarity as the cosine similarity between attribute vectors.
Figure~\ref{fig:celeba_categories} shows that the \acrshort{CelebA} dataset is more diverse for men than for women.
This result illustrates the potential usefulness of the Vendi Score in mitigating biases in datasets.

In Figure~\ref{fig:image_categories}, we calculate \acrshort{VS} for samples from different categories in \acrshort{CIFAR}-100,
using the cosine similarity between either Inception embeddings or pixel vectors.
The pixel diversity is highest for categories like ``aquarium fish'', which vary in color, brightness, and orientation, and lowest for
categories like ``cockroach'' in which images have similar regions of high pixel intensity (like white backgrounds).
The Inception diversity is less straightforward to interpret, but might correspond to some form of semantic diversity---for example, the
Inception diversity might be lower for classes like ``castle,'' that correspond to distinct ImageNet categories, and
higher for categories like ``clock'' and ``keyboard'' that are more difficult to classify.
In Appendix~\ref{app:diagnosing_datasets}, we show additional examples from text and other image datasets.

\begin{figure*}[t]
     \centering
      \includegraphics[width=\linewidth]{.//cifar100_categories}
\caption{
\label{fig:image_categories}
The categories in \acrshort{CIFAR}-100 with the lowest and highest \acrshort{VS}, defining similarity as the cosine similarity between either Inception embeddings or pixel vectors.
We show $100$ examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores according to the corresponding kernel.
}
\end{figure*}

 \glsresetall

\section{Discussion}
\label{sec:discussion}
We introduced the Vendi Score, a metric for evaluating diversity in \gls{ML}.
The Vendi Score is defined as a function of the pairwise similarity scores between elements of a sample and can be interpreted as the effective number of unique elements in the sample. The Vendi Score is interpretable, general, and applicable to any domain where similarity can be defined. It is unsupervised, in that it doesn't require labels or a reference probability distribution or dataset. Importantly, the Vendi Score allows its user to specify the form of diversity they want to measure via the similarity function. We showed the Vendi Score can be computed efficiently exactly and showcased its usefulness in several \gls{ML} applications, different datasets, and different domains. In future work, we'll leverage the Vendi Score to improve data augmentation, an important \gls{ML} approach in settings with limited data.

\subsection*{Acknowledgements}
Adji Bousso Dieng is supported by the National Science Foundation, Office of Advanced Cyberinfrastructure (OAC): \#2118201.

\clearpage
\bibliographystyle{apa}


\section{Norfolk Many-body Interactions}
\label{sec:norfolk}
The Norfolk interactions are obtained from a chiral effective field theory ($\chi$EFT) that uses pions, nucleons and $\Delta$'s as fundamental degrees of freedom, and consists of long-range parts mediated by one- and two-pion exchange, and contact terms specified by unknown low-energy constants (LECs). The LECs entering the $N\!N$ contact interactions are constrained to reproduce $N\!N$ scattering data
from the most recent and up-to-date database collected by the Granada group~\cite{Perez:2013jpa,Perez:2013oba,Perez:2014yla}.
The contact terms are regularized via a Gaussian cutoff function
with $R_S$ as the Gaussian parameter~\cite{Piarulli:2016vel,Piarulli:2014bda,Baroni:2018fdn}. The divergences at high-value of momentum transfer in the pion-range operators are removed via a special radial function characterized by the cutoff $R_L$~\cite{Piarulli:2016vel,Piarulli:2014bda,Baroni:2018fdn}.
There are two classes of NV2 potentials. Class I (II) has been fitted
to data up to $125$ MeV ($200$ MeV). For each class, two combinations
of short- and long-range regulators have been used, namely
($R_S$, $R_L$)=(0.8, 1.2) fm (models NV2-Ia and NV2-IIa) and
($R_S$, $R_L$)=(0.7, 1.0) fm (models NV2-Ib and NV2-IIb).
Class I (II) fits about 2700 (3700)
data points with a $\chi^2$/datum $\lesssim 1.1$
($\lesssim 1.4$)~\cite{Piarulli:2016vel,Piarulli:2014bda}. The short-range component of the $3N$ interactions is parametrized in terms of two LECs, $c_D$ and $c_E$. In the first generation of Norfolk potentials
(NV2+3-Ia/b and NV2+3-IIa/b), these LECs have been determined by simultaneously reproducing the experimental
trinucleon ground-state energies and $nd$ doublet scattering length~\cite{Piarulli:2017dwd}.
Within the $\chi$EFT framework, $c_D$ is related to the LEC entering
the axial two-body contact current~\cite{Gazit:2008ma,Marcucci:2011jm,Schiavilla:2017}.
This allows one to adopt a different strategy to constrain $c_D$ and $c_E$.
In particular, in Ref.~\cite{Baroni:2018fdn} they have been constrained
to reproduce the trinucleon binding energies and the empirical value of the Gamow-Teller matrix element in tritium $\beta$ decay. Norfolk models that use this fitting procedure
are designated with a `*' namely, NV2+3-Ia*/b* and NV2+3-IIa*/b*.

These interactions have been recently employed in the VMC and Green's function Monte Carlo (GFMC) approaches~\cite{Carlson:2014vla,Gandolfi:2020pbj} to calculate energies~\cite{Piarulli:2017dwd}, charge radii and electromagnetic form factors~\cite{Gandolfi:2020pbj}, beta-decay transitions~\cite{Baroni:2018fdn,King:2020pza,King:2020wmp}, neutrinoless double beta-decay~\cite{Cirigliano:2019vdj,Cirigliano:2018hja} of light nuclei, beta decay spectra~\cite{King:2022zkz}, muon-capture rates~\cite{King:2021jdb} and with the auxiliary field diffusion Monte Carlo (AFDMC)~\cite{Gandolfi:2020pbj} to study the equation of state of pure neutron matter~\cite{Piarulli:2019pfq,Lovato:2022apd}.

\section{density distributions}
\label{sec:den}
The one- and two-body densities are evaluated as simple $\delta$-function expectation values given by
\begin{eqnarray}
	\label{eq:rho}
	\rho_{N}(r) &=&\frac{1}{4\pi r^2}\big\langle\Psi\big|\sum_i    \mathcal P_{N_i}\delta(r-|\bf{r}_i-\bf{R}_{\mbox {cm}}|)\big|\Psi\big\rangle\,, \\
	\rho_{N\!N}(r) &=&\frac{1}{4\pi r^2}\big\langle\Psi\big|\sum_{i<j}    \mathcal P_{N_i}P_{N_j}\delta(r-|\bf{r}_i-\bf{r}_j|)\big|\Psi\big\rangle\,,
\end{eqnarray}
where $\mathcal P_{N_i}$ represents the projector operator onto protons ($+$) or neutrons ($-$)
defined as $\mathcal P_{N_i}=(1\pm\tau_{z_i})/2$, $\bf{r}_i$ is the position of nucleon $i$ and $\bf{R}_{\mbox {cm}}$ is the coordinate of the center of mass.

A detailed survey of one- and two-body densities have been calculated for a variety of nuclei in the range $A=2-12$ using variational Monte Carlo wave functions developed for the AV18+UX and the Norfolk local chiral interactions. The corresponding tables and figures are available online at \url{www.phy.anl.gov/theory/research/density/}, for the one-nucleon densities, and at \url{www.phy.anl.gov/theory/research/density2/}, for the two-nucleon densities.

\subsection{One-body density results}
\label{sec:1den}

In Fig.~\ref{1bdensity} we present the neutron and proton densities calculated for $^3$H, $^{3,4,8}$He, $^{6,7}$Li, $^9$Be, $^{10}$B, and $^{12}$C using the AV18+UX and the NV2+3-Ia,  NV2+3-Ia*, and NV2+3-IIb* local chiral interactions.
Additional densities for $^2$H, $^6$He, $^{8,9}$Li, $^{8,10,12}$Be, $^{11}$B and $^{10,11}$C may be found in the online tables, as well as results for the NV2+3-Ib* and NV2+3-IIa* interactions.
We also give neutron and proton rms radii there.

\begin{figure*}[!tbp]
 \includegraphics[width=\linewidth]{figures/1bdensities.pdf}
 \captionsetup{justification=raggedright,singlelinecheck=false}
 \caption{One-body neutron (left panel) and proton (right panel) densities are shown for $^3$H, $^{3,4,8}$He, $^{6,7}$Li, $^9$Be, $^{10}$B, and $^{12}$C using the phenomenological AV18+UX and the local chiral NV2+3-Ia,  NV2+3-Ia*, and NV2+3-IIb* interactions.}
 \label{1bdensity}
\end{figure*}

The VMC wave functions are treated as states of unique isospin $T$.  Thus for $N=Z$ nuclei, proton and neutron densities are the same and only proton densities are given in the online tables.
However, the wave functions for nuclei with $T>0$ can be different for different isospin projections $T_z$, so mirror nuclei are not isospin symmetric.  This allows the proton-rich nuclei to be slightly more diffuse than neutron-rich nuclei due to their greater repulsive Coulomb interaction.

Spin-up and spin-down densities are also provided in the online tables.  In $J=0$ nuclei, spin-up and spin-down densities are
identical, but not for $J>0$ nuclei.
If spin-up and spin-down projections are the same, as in $0^+$ states, we give only totals. The total number of spin-up/down protons and neutrons in $J > 0$ nuclei with $M_J = J$ are reported in Table~\ref{tab:sigma-tau}. Unless otherwise indicated by an error in parentheses, variation among the different interaction models is less than 0.01.
We note that for these nuclei, the subset with an odd number of neutrons has $(n\!\uparrow \!- n\!\downarrow) \approx$ 0.7-0.9, while those with an even number of neutrons have $(n\!\uparrow \!- n\!\downarrow) \approx$ -0.02. Similar results hold for nuclei with odd and even proton numbers.  The sole exception is $^9$Li which has an exceptionally large error bar.

We also note that the s-shell nuclei ($A \leq 4$) exhibit large peaks at small separation, while the p-shell nuclei ($A \geq 6$) are much reduced at small $r$ and more spread out.  This can be attributed to the cluster structure of these light p-shell nuclei, e.g., $\alpha d$ in $^6$Li, $\alpha t$ in $^7$Li, $\alpha \alpha n$ in $^9$Be, and $3 \alpha$ in $^{12}$C.  This puts the center of mass of these nuclei in between clusters and thus reduces the central density.

\begin{table}[t!]
\captionsetup{justification=raggedright,singlelinecheck=false}
\caption{Total number of spin-up/down protons and neutrons in $J > 0$ nuclei with $M_J = J$ for the local chiral Norfolk NV2+3 interactions.  Variation among the different interactions NV2+3-Ia, -Ia*, -Ib*, -IIa*, and -IIb* is less than 0.01
unless otherwise indicated by an error in parentheses.}
\begin{ruledtabular}
\begin{tabular}{ l d d d d }
Nucleus & \multicolumn{1}{c}{$N_{\uparrow p}$}
        & \multicolumn{1}{c}{$N_{\downarrow p}$}
        & \multicolumn{1}{c}{$N_{\uparrow n}$}
        & \multicolumn{1}{c}{$N_{\downarrow n}$} \\
\hline
$^2$H($1^+$)             & 0.96 & 0.04 & 0.96 & 0.04 \\
$^3$He($\frac{1}{2}^+$)  & 0.98 & 1.02 & 0.94 & 0.06 \\
$^6$Li($1^+$)            & 1.93 & 1.07 & 1.93 & 1.07 \\
$^7$Li($\frac{3}{2}^-$)  & 1.94 & 1.06 & 1.99 & 2.01 \\
$^8$Li($2^+$)            & 1.91 & 1.09 & 2.85(1) & 2.15(1) \\
$^9$Li($\frac{3}{2}^-$)  & 1.91 & 1.09 & 3.12(7) & 2.88(7) \\
$^9$Be($\frac{3}{2}^-$)  & 2.00 & 2.00 & 2.85(2) & 2.15(2) \\
$^{10}$B($3^+$)          & 2.90(1)& 2.10(1)& 2.90(1)& 2.10(1) \\
$^{11}$B($\frac{3}{2}^-$)& 2.87(2)& 2.13(2)& 2.99(1)& 3.01(1)
\label{tab:sigma-tau}
\end{tabular}
\end{ruledtabular}
\end{table}

\subsection{Two-body density results}
\label{sec:2den}

In Fig.~\ref{2bdensitiesnoscaled}, we present the relative-distance pair densities, with neutron-proton ($np$) in the left panel and proton-proton ($pp$) in the right panel, for $^3$H, $^{3,4,8}$He, $^{6,7}$Li, $^9$Be, $^{10}$B, and $^{12}$C using the phenomenological AV18+UX and the local chiral NV2+3-Ia, NV2+3-Ia*, and NV2+3-IIb* interactions. The online tables contain additional results for the NV2+3-Ib* and NV2+3-IIa* interactions.

\begin{figure*}[!tbp]
 \includegraphics[width=\linewidth]{figures/2bdensitiesnoscaled.pdf}
 \captionsetup{justification=raggedright,singlelinecheck=false}
 \caption{Relative-distance $np$ (left panel) and $pp$ (right panel) pair densities for several
nuclei. Distributions are shown for $^3$H, $^{3,4,8}$He, $^{6,7}$Li, $^9$Be, $^{10}$B, and $^{12}$C using the phenomenological AV18+UX and the local chiral NV2+3-Ia,  NV2+3-Ia*, and NV2+3-IIb* interactions.}  \label{2bdensitiesnoscaled}
\end{figure*}

\begin{figure*}[!tbp]
 \includegraphics[width=\linewidth]{figures/2bdensities.pdf}
 \captionsetup{justification=raggedright,singlelinecheck=false}
 \caption{Relative-distance $np$ (left panel) and $pp$ (right panel) pairs densities for several
nuclei. Distributions are shown for $^3$H, $^{3,4,8}$He, $^{6,7}$Li, $^9$Be, $^{10}$B, and $^{12}$C using the phenomenological AV18+UX and the local chiral NV2+3-Ia,  NV2+3-Ia*, and NV2+3-IIb* interactions. For each potential, all calculations are scaled to have the same value at $\sim 1$ fm. }  \label{2bdensities}
\end{figure*}

We can see that within a fixed interaction model, the two-nucleon densities at $r \lesssim 1.5$ fm for various nuclei exhibit a similar behavior, generated
by the cooperation of the short-range repulsion and the intermediate-range tensor attraction of the $N\!N$ interaction, with the tensor force governing the large overshoot at $r \sim 1.0$ fm between $np$ pairs.

As shown in Fig.~\ref{2bdensities}, where all calculations are scaled to have the same value at $\sim 1$ fm, the two-nucleon densities at short separations appears to be the same for all values of $A$, which leads to the nontrivial conclusion that at short ranges the two-nucleon motion is not affected by the presence of the other particles.
This is what has been called universality of SRCs~\cite{Feldmeier:2011qy}. Moreover, at large separation the asymptotic behavior of the two-nucleon densities for different nuclei differs due to the different surface effects.

While the short-distance behaviour is the same for all nuclei, it differs for each interaction. Indeed, the probability of finding two nucleons at short distances is finite for the "soft" NV2+3-Ia and NV2+3-Ia* chiral models, but approaches zero as we progress to the "hard" local chiral interaction NV2+3-IIb* and the "hardest" phenomenological AV18+UX.

Nucleon pair distributions in different combinations of $ST$ for different nuclei can also be found online. In Table~\ref{tab:ST}, we report the number of pairs $N_{ST}$ for $^{3,4,6,8}$He, $^{6,7,8,9}$Li, $^{8,9,10}$Be, $^{10,11}$B, and $^{12}$C using the NV2+3 potentials. We show both independent pair (IP) numbers for the highest spatial symmetry states and for the fully correlated (cor) wave functions.  Correlated pair counts for the AV18+UX interaction fall within these ranges for all but two cases.

\begin{table}[t!]
\captionsetup{justification=raggedright,singlelinecheck=false}
\caption{Total number of spin-isospin $ST$ pairs in different nuclei for the NV2+3 potentials, showing both independent pair (IP) for the highest spatial symmetry states and the fully correlated (cor) wave functions.  Correlated pair counts for AV18+UX are consistent within error bars for all but a few cases.}
\setlength{\tabcolsep}{0.01\tabcolsep
\begin{ruledtabular}
\begin{tabular}{ l c d d d d }
Nucleus & $\Psi$ & \multicolumn{1}{c}{$N_{01}$}
                 & \multicolumn{1}{c}{$N_{11}$}
                 & \multicolumn{1}{c}{$N_{10}$}
                 & \multicolumn{1}{c}{$N_{00}$} \\
\hline
$^3$He($\frac{1}{2}^+$)  & IP  & 1.5     & 0.0     & 1.5   & 0.0   \\
                         & cor & 1.37    & 0.13    & 1.49  & 0.01  \\
$^4$He($0^+$)            & IP  & 3.0     & 0.0     & 3.0   & 0.0   \\
                         & cor & 2.57(1) & 0.43(1) & 2.99  & 0.01  \\
$^6$He($0^+$)            & IP  & 5.5     & 4.5     & 4.5   & 0.5   \\
                         & cor & 4.95(1) & 5.05(1) & 4.49  & 0.51  \\
$^6$Li($1^+$)            & IP  & 4.5     & 4.5     & 5.5   & 0.5   \\
                         & cor & 4.07(2) & 4.93(2) & 5.47  & 0.53  \\
$^7$Li($\frac{3}{2}^-$)  & IP  & 6.75    & 6.75    & 6.75  & 0.75  \\
                         & cor & 6.13(2) & 7.37(2) & 6.73  & 0.77  \\
$^8$He($0^+$)            & IP  & 9.0     & 12.0    & 6.0   & 1.0   \\
                         & cor & 8.16(2) & 12.84(2)& 6.00  & 1.00  \\
$^8$Li($2^+$)            & IP  & 8.0     & 11.0    & 8.0   & 1.0   \\
                         & cor & 7.42(3) & 11.58(3)& 7.94  & 1.06  \\
$^8$Be($0^+$)            & IP  & 9.0     & 9.0     & 9.0   & 1.0   \\
                         & cor & 8.09(3) & 9.91(3) & 8.97  & 1.03  \\
$^9$Li($\frac{3}{2}^-$)  & IP  & 10.5    & 15.0    & 9.0   & 1.5   \\
                         & cor & 9.46(12)& 16.04(11)& 8.99(1)& 1.51(1)  \\
$^9$Be($\frac{3}{2}^-$)  & IP  & 10.5    & 13.5    & 10.5  & 1.5   \\
                         & cor & 9.57(4) & 14.43(4)& 10.46 & 1.54  \\
$^{10}$Be($0^+$)         & IP  & 13.0    & 18.0    & 12.0  & 2.0   \\
                         & cor & 11.73(4)& 19.27(4)& 11.98 & 2.02  \\
$^{10}$B($3^+$)          & IP  & 12.0    & 18.0    & 13.0  & 2.0   \\
                         & cor & 11.01(7)& 18.99(7)& 12.94(1) & 2.06(1)  \\
$^{11}$B($\frac{3}{2}^-$)& IP  & 15.0    & 22.5    & 15.0  & 2.5   \\
                         & cor & 13.84(2)& 23.66(3)& 14.91(3) & 2.59(3)  \\
$^{12}$C($0^+$)          & IP  & 18.0    & 27.0    & 18.0  & 3.0  \\
                         & cor & 16.54(6)& 28.46(6)& 17.91(1) & 3.09(1)
\label{tab:ST}
\end{tabular}
\end{ruledtabular}
\end{table}

A common feature in the $ST$ pair counts is that there is a moderate 10-15\% depletion of the $ST=01$ pairs going from IP to correlated wave functions, with a corresponding increase in the number of $ST=11$ pairs.  This is attributable to the many-body tensor correlations, which can flip spins (in exchange for orbital angular momentum) but not change isospin.  Because the $ST=01$ interactions are more attractive than $ST=11$, this depletion mechanism is a source of saturation of the nuclear binding.  The $ST=10$ pairs also show a depletion going from IP to correlated wave functions, with an increase of $ST=00$ pairs, but the effect is much smaller, probably because $ST=00$ interaction is generally much more repulsive than $ST=11$.

The probability of finding two nucleons with relative separation $r$ and center-of-mass distance $R$ is described by the calculation of the full probability density $\rho_{N\!N}(r,R)$. These densities are computationally demanding and are not available for all nuclei and interactions, but they can be generated upon request.

In Fig.~\ref{4He_12C_rhorR.av18ux.np.pp}, we present the the $np$ and $pp$ densities, multiplied by $r^2 R^2$, as a function of $r$ and $R$ for $^4$He and $^{12}$C using the phenomenological AV18+UX interaction. The curves are normalized to obtain the corresponding $np$ and $pp$ pairs, 4 $np$ and 1 $pp$ pairs in $^4$He,  36 $np$ and 15 $pp$ pairs in $^{12}$C.

\begin{figure*}[b]
 \includegraphics[width=\linewidth]{figures/4He_12C_rhorR.av18ux.np.pp.cropped.pdf}
 \captionsetup{justification=raggedright,singlelinecheck=false}
 \caption{The surface plots show the $np$ and $pp$ densities as function of the relative distance $r$ and center-of-mass $R$ for $^4$He (left panel) and $^{12}$C (right panel) using the phenomenological AV18+UX. The curves are normalized to obtain the corresponding $np$ and $pp$ pairs. }  \label{4He_12C_rhorR.av18ux.np.pp}
\end{figure*}

\begin{figure*}[h]
 \includegraphics[width=\linewidth]{figures/1bmomentum.pdf}
 \captionsetup{justification=raggedright,singlelinecheck=false}
 \caption{Total one-body neutron (left panel) and proton (right panel) momentum distributions for $^3$H, $^3$He, $^4$He, $^6$Li, $^7$Li, $^8$He, $^9$Be, $^{10}$B, and $^{12}$C using the AV18+UIX phenomenological potentials, and the NV2+3-Ia,  NV2+3-Ia*, and NV2+3-IIb* local chiral interactions.}\label{1bmomentum}
\end{figure*}

\section{Momentum distributions}
\label{sec:mom}

\subsection {One-body momentum distributions}
\label{sec:single}
The probability of finding a nucleon with momentum $k$ and spin-isospin projection $\sigma$,$\tau$ in a given nuclear state is obtained by the Fourier transform of the one-nucleon nondiagonal density matrix

\begin{eqnarray}
\label{eq:rhok}
\rho_{\sigma\tau}({\bf k})\!\!&=&\!\!
\int d{\bf r}^\prime_1\, d{\bf r}_1\, d{\bf r}_2 \cdots d{\bf r}_A\,
\psi^\dagger_{JM_J}({\bf r}_1^\prime,{\bf r}_2, \dots,{\bf r}_A)\,  \nonumber \\
& \times &\, e^{-i{\bf k}\cdot ({\bf r}_1-{\bf r}^\prime_1)}
\, P_{\sigma\tau}(1) \,
\psi_{JM_J} ({\bf r}_1,{\bf r}_2, \dots,{\bf r}_A) \, .
\end{eqnarray}

where $P_{\sigma\tau}(i)$ is the spin-isospin projection operator for nucleon $i$, and  $\psi_{JM_J}$ is the nuclear wave function with total spin $J$ and
spin projection $M_J$.  The normalization is
\begin{equation}
\label{eq:nst}
  N_{\sigma \tau} =
    \int \frac{d{\bf k}}{(2\pi)^3}\,\,\rho_{\sigma\tau}({\bf k}) \ ,
\end{equation}
where $N_{\sigma \tau}$ is the number of spin-up or spin-down protons
or neutrons.

Monte Carlo (MC) integration is used to construct the Fourier transform in Eq.~(\ref{eq:rhok}).  A conventional Metropolis walk, guided by
$|\psi_{JM_J} ({\bf r}_1,\dots,{\bf r}_i,\dots,{\bf r}_A)|^2$, is used
to sample configurations~\cite{Pudliner:1997ck}.
We average across all particles $i$ in each configuration, and for each
particle, the Fourier transform is computed using a grid of Gauss-Legendre points $\bf x_i$.
Instead of just moving the position ${\bf r}_i^\prime$ in the left-hand
wave function away from a fixed position ${\bf r}_i$ in the right-hand
wave function, both positions are moved
symmetrically away from ${\bf r}_i$, so Eq.~(\ref{eq:rhok}) becomes
\begin{eqnarray}
\label{eq:actual}
  \rho_{\sigma\tau}({\bf k}) &=& \frac{1}{A}\sum_i
  \int d{\bf r}_1 \cdots d{\bf r}_i \cdots d{\bf r}_A \int d{\bf \Omega}_x
  \int_0^{x_{\rm max}} x^2 dx \nonumber \\
  & & \psi^\dagger_{JM_J}
  ({\bf r}_1,\dots,{\bf r}_i+{\bf x}/2,\dots,{\bf r}_A) \,
  e^{-i{\bf k}\cdot {\bf x} }  \\
  &\times& \,  P_{\sigma\tau}(i) \,
  \psi_{JM_J}
  ({\bf r}_1,\dots,{\bf r}_i-{\bf x}/2,\dots,{\bf r}_A) \nonumber \, .
\end{eqnarray}
Here the polar angle $d{\bf \Omega}_x$ is also sampled by MC integration, with
a randomly chosen direction for each particle in each MC configuration.
This approach is analogous to that used in studies of the nucleon-pair momentum distribution, see Refs.~\cite{Schiavilla:2006xx,Wiringa:2013ala}, and has the benefit of significantly decreasing statistical errors caused by the rapidly oscillating nature of the integrand for large values of $k$.
To reach momenta $k \sim 10$ fm$^{-1}$ in $^4$He with good statistics
requires integrating to $x_{\rm max}$=20 fm using 200 Gauss-Legendre points.\\

The results for a variety of nuclei in the range $A=2-12$ are available on the web page at \url{www.phy.anl.gov/theory/research/momenta/}. They are generated as distributions for neutron spin-down $\rho_{n\downarrow}(k)$, neutron spin-up $\rho_{n\uparrow}(k)$, proton spin-down $\rho_{p\downarrow}(k)$, and proton spin-up $\rho_{p\uparrow}(k)$, for the $M_J=J$ state. Where proton and neutron momentum distributions are the same, as in $T=0$ nuclei, only one set is given, and similarly, if spin-up and spin-down projections are the same, as in $0^{+}$ states, we give totals only.

In Fig.~\ref{1bmomentum} we show the total one-body neutron (left panel), $\rho_{n\downarrow}(k)+\rho_{n\uparrow}(k)$, and proton (right panel), $\rho_{p\downarrow}(k)+\rho_{p\uparrow}(k)$, momentum distributions for $^3$H, $^{3,4}$He, $^{6,7}$Li, $^8$He, $^9$Be, $^{10}$B, and $^{12}$C using the phenomenological AV18+UX, and the local chiral NV2+3-Ia,  NV2+3-Ia*, and NV2+3-IIb* interactions. Additional results for $^2$H, $^6$He, $^{8,9}$Li, $^{8,10}$Be, and $^{11}$B are shown in the online tables.

All models show the progressive high-momentum behavior in $k$ as the number of nucleons increases. Adding nucleons to the p-shell widens the distribution at low momenta and creates a peak at a finite $k$. All of these nuclei have a dramatic shift in slope at $k = 2$ fm$^{-1}$ to a broad shoulder, which is attributed to the large tensor correlation caused by the pion-exchange component of the nuclear force. As expected, the differences between the models are most noticeable in the high-momentum tails, which decay more rapidly with increasing $k$ for the "soft" NV2+3-Ia and NV2+3-Ia* interactions than the "hard" NV2+3-IIb* and AV18+UX potentials. The difference in momentum distributions observed for the NV2+3-Ia and NV2+3-Ia* models, which have the same two-body interaction but different the three-nucleon force parametrization, is small. For NV2+3-Ia, there is a minor overall shift in $\rho_n(k)$ and $\rho_p(k)$ toward bigger $k$.

\subsection {Two-body momentum distributions}
\label{sec:double}

The probability of finding two nucleons in a nucleus with relative momentum
${\bf q}=({\bf k}_1-{\bf k}_2)/2$ and total center-of-mass momentum
${\bf Q}={\bf k}_1+{\bf k}_2$ in a given spin-isospin state is given by:
\begin{eqnarray}
\label{eq:rhoqQ}
\rho_{ST}({\bf q},{\bf Q})
&=&\int d{\bf r}^\prime_1 d{\bf r}_1 d{\bf r}^\prime_2d{\bf r}_2 d{\bf r}_3 \cdots d{\bf r}_A \nonumber \\
&& \psi^\dagger_{JM_J}({\bf r}_1^\prime,{\bf r}^\prime_2,{\bf r}_3,\dots,{\bf r}_A)  \\
&& e^{-i{\bf q}\cdot({\bf r}-{\bf r}^\prime)}
e^{-i{\bf Q}\cdot({\bf R}-{\bf R}^\prime)} \nonumber \\
&&  P_{ST}(12) \psi_{JM_J}({\bf r}_1,{\bf r}_2,{\bf r}_3,\dots,{\bf r}_A) \, ,
\nonumber
\end{eqnarray}
where ${\bf r} = {\bf r}_1 - {\bf r}_2$,
${\bf R} = ({\bf r}_1 + {\bf r}_2)/2$,
and $P_{ST}(12)$ is a projector onto pair spin $S=0$ or 1, and isospin
$T=0$ or 1.
The total normalization is:
\begin{equation}
N_{ST} = \int \frac{d{\bf q}}{(2\pi)^3} \frac{d{\bf Q}}{(2\pi)^3}
         \rho_{ST}({\bf q},{\bf Q}) \, ,
\end{equation}
where $N_{ST}$ is the total number of nucleon pairs with given spin-isospin.
Alternate projectors can also be used, e.g., for $N\!N$ pairs $pp$, $np$,
and $nn$ (and each of these with spin S) with corresponding normalizations.

The nucleon-pair momentum distributions can be examined in a number of different ways.
One way is to integrate over all values of ${\bf Q}$ and reduce the total pair
density to a function $\rho_{ N\!N}(q)$ of the relative momentum $q$ only.
In this case, Eq.(\ref{eq:rhoqQ}) reduces to a form similar to
Eq.(\ref{eq:actual}), with a sum over all configurations in the Monte
Carlo walk controlled by $|\Psi_{JM_J}|^2$, and a Gauss-Legendre integration
over the relative separation ${\bf x} = {\bf r}-{\bf r}^\prime$.
Again, the polar angle ${\bf \Omega}_x$ is sampled by randomly choosing the
direction of ${\bf x}$ in space, and an average over all pairs in every
MC configuration is made.

Many results for $\rho_{ST}(q)$ and $\rho_{N\!N}(q)$ obtained for various light nuclei in the range $A=3-12$ are recorded in the online tables at \url{www.phy.anl.gov/theory/research/momenta2/}. These are from VMC calculations using different Norfolk NV2+3 potentials, including -Ia, -Ia*, -Ib*, -IIa*, and -IIb*, as well as results obtained with the AV18+UX.
The nuclei covered include $^3$H, $^{3,4,6,8}$He, $^{6,7,8,9}$Li, $^{8,9}$Be, $^{10}$B, and $^{12}$C.

In Fig.~\ref{2bmomentum}, we display the $np$ and $pp$ momentum distributions for selected nuclei using the AV18+UX phenomenological potentials, and the NV2+3-Ia,  NV2+3-Ia*, and NV2+3-IIb* local chiral interactions. They have been calculated for relative momentum $q$ from 0 to 10\, fm$^{-1}$ and integrated over all values of ${\bf Q}$. All the four Hamiltonians show the high-momentum tail in $q$, but it decays more rapidly for the soft NV2+3-Ia and NV2+3-Ia*.

\begin{figure*}[b]
 \includegraphics[width=\linewidth]{figures/2bmomentum.pdf}
 \captionsetup{justification=raggedright,singlelinecheck=false}
 \caption{Total $np$ (left panel) and $pp$ (right panel) momentum distributions as function of the relative momentum $q$ for $^3$H, $^3$He, $^4$He, $^6$Li, $^7$Li, $^8$He, $^9$Be, $^{10}$B, and $^{12}$C using the AV18+UIX phenomenological potentials, and the NV2+3-Ia,  NV2+3-Ia*, and NV2+3-IIb* local chiral interactions.}  \label{2bmomentum}
\end{figure*}

In Fig.~\ref{ratio}, we show the ratio of the $np$ and $pp$ momentum distributions for several nuclei relative to the total $np/pp$ ratio.  For $q\leq 2$ fm$^{-1}$ the ratios are virtually identical for the different interactions and close to the total number of $np/pp$ pairs.  Beyond that point, the $np/pp$ ratio gets larger, with the soft interactions showing a larger peak at smaller $q$, while the hard interactions interactions show a lower but broader peak at larger $q$.  This behavior is probably due to the strong tensor correlations in the $np$ channel.
Note these ratios are integrated over all $Q$, while the much larger $np/pp$ ratios, mentioned above from Refs.~\cite{Hen:2014nza,LabHallA:2014wqo,CLAS:2018xvc} are for small $Q$.

\begin{figure*}[t]
 \includegraphics[width=\linewidth]{figures/super.pdf}
 \caption{Ratio of $np$ to $pp$ pairs as function of relative momentum $q$
 for various nuclei; the constant line in each panel is the total pair ratio.}  \label{ratio}
\end{figure*}

\begin{figure*}[h]
 \includegraphics[width=\linewidth]{figures/4He.rho_qQ_tot.pdf}
 \captionsetup{justification=raggedright,singlelinecheck=false}
 \caption{The surface plots show the the $np$ (left panels) and $pp$ (right panels) momentum distributions as functions of the relative momentum $q$ and center-of-mass momentum $Q$ for the alpha particle obtained with the NV2+3-Ia* and AV18+UX interactions.}    \label{He.rho(q,Q)_tot}
\end{figure*}

\begin{figure*}[h]
 \includegraphics[width=\linewidth]{figures/12C.rho_qQ_av18ux.tot.cropped}
 \captionsetup{justification=raggedright,singlelinecheck=false}
 \caption{Same as Fig.~\ref{He.rho(q,Q)_tot} but for $^{12}$C using the AV18+UX. }  \label{C.rho(q,Q)_tot}
\end{figure*}

We can also integrate Eq.(\ref{eq:rhoqQ}) over all ${\bf q}$, leaving a function $\rho_{N\!N}(Q)$ of the total pair momentum $Q$ only.  In general, the $\rho_{N\!N}(Q)$ for a given nucleus has a smaller falloff at large momenta than the $\rho_{N\!N}(q)$ and the ratios of different $N\!N$ components vary less over the range of $Q$. These distributions are generally not as interesting, but they can be generated on request.

Calculations of the full $\rho_{ N\!N}({\bf q},{\bf Q})$ are more challenging as they require a double Gauss-Legendre integral over two randomly chosen directions for each pair in each MC sample: ${\bf x}$ for ${\bf r}$ and ${\bf X}$ for ${\bf R}$. In Fig.~\ref{He.rho(q,Q)_tot}, we display the surface plots of $\rho_{np}({\bf q},{\bf Q})$ and $\rho_{pp}({\bf q},{\bf Q})$ as functions of the relative momentum $q$ and center-of-mass momentum $Q$ for $^4$He using the NV2+3-Ia* and AV18+UX interactions.  In Fig.~\ref{C.rho(q,Q)_tot} we show similar plots for $^{12}$C but only for the AV18+UX interaction.  In the online tables we present $\rho_{ N\!N}({\bf q},{\bf Q})$ results for $^{3,4}$He, $^6$Li, $^{12}$Be, and $^{12}$C for the AV18+UX and one or both of the NV2+3-Ia* and NV2+3-IIb* interactions.

In addition, we can differentiate between short-range (SR) and long-range (LR) pair contributions by simply sorting our MC samples into two sets, where $r<a$ for SR pairs and $r>a$ for LR pairs. For example, in $^4$He, a boundary of $a=2$ fm divides the six $N\!N$ pairs into approximately two equal groups. Fig.~\ref{He.rho(q,Q)_2.0} shows the $np$ (left panels) and $pp$ (right panels) SR and LR pair distributions for the alpha particle using the NV2+3-Ia* and AV18+UX interactions. Taking the integral of these distributions we find that in the case of the NV2+3-Ia* the number of $np$ SR and LR pairs are 2.04 and 1.96, respectively, while the number of $pp$ SR and LR pairs are 0.53 and 0.47. For the case of the AV18+UX the of $np$ SR and LR pairs are 2.06 and 1.94 while the number of $pp$ SR and LR pairs are 0.48 and 0.52.

\begin{figure*}[h]
 \includegraphics[width=\linewidth]{figures/4He.rho_qQ_2.0.pdf}
 \caption{ $np$ (left panel) and $pp$ (right panel) momentum distributions in $^4$He from pairs with separation $r \leq 2.0$ fm of each other and from pairs with $r \geq 2.0$ fm. }  \label{He.rho(q,Q)_2.0}
\end{figure*}

Similarly, The $np$ and $pp$ SR and LR pair distributions for $^{12}$C employing the AV18+UX interactions are shown in Fig.~\ref{12C.rho(q,Q)_av18ux.2.5} for $a=2.5$ fm. By integrating these distributions, we find that the number of $np$ SR and LR pairings is 12.9 and 23.1 respectively, while the number of $pp$ SR and LR pairs is 4.3 and 10.7, respectively.
In the online figures and tables we also provide the breakdown for $^{3,4}$He and $^6$Li for the AV18+UX, NV2+3-Ia*, and NV2+3-IIb* with the break point $a=2.5$ fm.
These figures clearly show that the LR pairs dominate at low pair momenta but fall off rapidly beyond $q \approx 1.5$ fm$^{-1}$, while the SR pairs provide the high-momentum tail.  For the pair center of mass momentum, the total number of pairs declines significantly beyond $Q \approx 2$ fm$^{-1}$ but there continues to be a high-momentum tail in $q$.

\begin{figure*}[h]
 \includegraphics[width=\linewidth]{figures/12C.rho_qQ_av18ux.2.5.cropped}
 \caption{ Same as Fig.~\ref{He.rho(q,Q)_2.0} but for $^{12}$C using the AV18+UX with separation boundary $a = 2.5$ fm. }  \label{12C.rho(q,Q)_av18ux.2.5}
\end{figure*}

\section {Conclusions}
\label{sec:conclude}
We have performed VMC calculations of one- and two-body density distributions and one- and two-body momentum distributions for a wide variety of nuclei from $^2$H up to $^{12}$C, using the $\Delta$-full Norfolk interactions obtained from $\chi$EFT.
Results are compared to those obtained with the conventional AV18+UX interaction, some of which were previously reported in Ref.~\cite{Wiringa:2013ala}.
New features in the present work include i) calculations of the pair density $\rho_{N\!N}(r,R)$ as a function of both the pair separation $r$ and pair center-of-mass $R$; and ii) calculations of the two-body momentum distribution $\rho_{N\!N}(q,Q,a)$ coming from short- and long-range pairs differentiated by a pair separation boundary $a$.

Comparing results among the different NV2+3 and AV18+UX interactions, we find the one-body densities $\rho_N(r)$ for a given nucleus are very similar for all cases.  Also, the total number of spin-up/down protons and neutrons is remarkably constant.
In contrast, the two-body densities $\rho_{N\!N}(r)$ vary significantly at short distances, depending on whether the interaction is `soft' like NV2+3-Ia or `hard' like NV2+3-IIb*.  However, the total number of spin-isospin $ST$ pairs for a given nucleus shows little variation among the different interactions.

One-body momentum distributions $\rho(k)$ all share the same characteristics, with a maximum at $k=0$ fm$^{-1}$, a rapid fall off to $k\approx 2$ fm$^{-1}$, followed by a high-momentum tail that is more prominent for `hard' interactions, and less so for `soft' interactions.  For a given interaction, the low-$k$ behavior varies with the nucleus, but the high-$k$ tails are essentially parallel for all $A=2$-12 nuclei.  Two-body momentum distributions $\rho_{N\!N}(q)$ are similar, but tend to change slope near $q\approx 1.5$ fm$^{-1}$.  Again, the high-$q$ tail is larger for `hard' interactions.  We also note that the ratio $\rho_{np}(q)/\rho_{pp}(q)$ is relatively flat and proportional to the number of pairs of each type at lower $q$, but much larger at higher $q$, reflecting the importance of the stronger tensor correlations in $np$ versus $pp$ pairs.  Finally, our studies of $\rho_{N\!N}(q,Q,a)$, which separates the contributions of short- from long-range pairs, clearly indicates the high-$q$ tails are due to SRCs.

Concurrent to these studies, are QMC studies of nuclear electroweak response densities and response functions~\cite{Pastore:2019urn,Barrow:2020mfy,Andreoli:2021cxo} where the interaction of the external probes (both electron and neutrinos) is accounted for at one- and two-body level along with SRCs. In particular, within the Short-Time-Approximation~\cite{Pastore:2019urn} it is possible to analyze electroweak nuclear responses in terms of the kinematic variables, that is relative and center of mass momenta,  associated with a pair of correlated nucleons struck by the probe. Many-body effects in the coupling of electroweak probes with correlated nucleons are being vigorously investigated due to their relevance to both electron and neutrino scattering processes.

While this paper provides examples of the densities and momentum distributions, the full set of results is accessible in graphical and tabular forms online at \url{www.phy.anl.gov/theory/research/QMCresults.html}.
We expect to continue expanding and updating these results in future.

\section*{Acknowledgments}
This work is supported by the U.S.~Department of Energy under contract DE-SC0021027 (S.~P.), DE-AC02-06CH11357 (R.~B.~W.), a 2021 Early Career Award number DE-SC0022002 (M.~P.), the FRIB Theory Alliance award DE-SC0013617 (S.~P. and M.~P.), and the NUCLEI SciDAC program (R.~B.~W.).

The many-body calculations were performed on the parallel computers of the Laboratory Computing Resource Center, Argonne National Laboratory, and the computers of
the Argonne Leadership Computing Facility via the INCITE grant ``Ab-initio nuclear structure and nuclear reactions'', the 2019/2020 ALCC grant ``Low Energy Neutrino-Nucleus interactions'' for the project NNInteractions,
the 2020/2021 ALCC grant ``Chiral Nuclear Interactions from Nuclei to Nucleonic Matter'' for the project ChiralNuc, and by the 2021/2022 ALCC grant ``Quantum Monte Carlo Calculations of Nuclei up to $^{16}{\rm O}$ and Neutron Matter" for the project QMCNuc.

\bibliographystyle{apsrev.bst}