\section{Introduction}

    Self-supervised learning (SSL) methods usually rely on a supervision obtained from the data itself through solving specific pretext tasks leveraging the underlying structure of the considered data \cite{doersch2016unsupervised, Arandjelovic_2018_ECCV}. This technique is used in various domains, including image processing \cite{misra2020self, jing2020self, grill2020bootstrap}, natural language understanding \cite{lan2019albert,chen2020big, du} or speech and audio processing \cite{baevski2020wav2vec,Liu_2020,jiang2020speech}. It offers numerous advantages, such as the independence from labeled data, stronger performance on downstream tasks, more robust models and an easier transfer to low-resource setups (\textit{e.g.}, low-resource languages) \cite{baevski2020wav2vec, jing2020self}.

Various and numerous approaches for speech SSL have been proposed which can be classified according to the nature of the pretext task they try to solve. For example, common methods include predictive coding \cite{baevski2020wav2vec,Liu_2020,song2020speechxlnet,Zhang2020, hubert}, pseudo-label learning \cite{pascual2019learning,ravanelli2020multitask}, auto-encoding \cite{Renshaw2015ACO,https://doi.org/10.48550/arxiv.2007.13542}, triplet-loss learning \cite{shor2020towards,peplinski2020frill}, generative modelling \cite{khurana2020convolutional} or contrastive learning \cite{Saeed2020,jiang2020speech}. In particular,  pretext tasks can be defined as the prediction of pretext labels, hereafter referred to as \textit{pseudo-labels}. The automatic extraction of pseudo-labels for SSL (\textit{i.e.} from the unlabeled data itself) is common in many application fields, such as computer vision \cite{noroozi2017unsupervised, gidaris} or audio and speech processing \cite{hung2019multitask, pascual2019learning, shukla, wu2021multitask}, and is commonly referred to as \textit{multitask self supervised learning}. Regarding the specific context of audio and speech processing, generating pseudo-labels may benefit from decades of research in signal processing. For instance, classic signal features such as pitch estimators, energy-based features or voicing states can be used as pseudo-label candidates.

As shown by \cite{pascual2019learning}, multi-task learning of speech representations obtains solutions which lead to improvements in distinct downstream tasks, through the combination of different pseudo-labels that seem 'intuitively' useful for these tasks. Unfortunately, there is no clear understanding of how these pseudo-labels may interact when optimised together, and therefore there is no common practice on how to select groups of pseudo-labels to achieve better performance on a known downstream task. In fact, this design process has been mainly relying on empirical validation. Such an empirical approach raises the issue of intractability with modern SSL architectures containing hundreds of millions or billions of parameters trained over thousands of hours of speech. For example, the self-supervised training of a single peak wav2vec 2.0 model over $53.2k$ hours of speech requires $128$ GPU for $5.2$ days.

The goal of this work is to provide an efficient and theoretically motivated way to select and weight a group of pretext tasks among a set of candidates using conditional independence (CI). The model outputs, before the training, the multitask self-supervised loss most suited to the final downstream of interest. This allows one to save time and compute by avoiding the empirical pseudo-label search and ablations. Hence, the contributions of this work are fourfold:

\begin{enumerate}
    \item Present a theoretically motivated and computationally efficient (Appendix \ref{efficiency}) method for the group selection of pseudo-label \textit{groups} among a candidates' pool towards solving a considered downstream task (Sections \ref{sec:CI} and \ref{sec:group}).
    \item Validate empirically the group selection approach with a multi-task  SSL model using different weighted sets of pseudo-labels adapted for three considered speech tasks : automatic speech recognition, speaker verification and emotion recognition. (Sections \ref{sec:exps}).
    \item Test the extension of a reference SSL approach, namely wav2vec 2.0, with carefully selected signal-related pretext tasks, enhancing its performance on three downstream tasks. (Section \ref{sec:wav2vec}).
    \item Release the code base developed with SpeechBrain \cite{speechbrain} for replication and to help further research on the topic.\footnote{\url{https://github.com/salah-zaiem/Multitask-pretext-task-selection}}
\end{enumerate}

With this motivated pretext task group selection, the experiments conducted show that the multitasked model proposed performs better than the baseline, obtaining $31.6\%$, $27.4\%$ and $7.8\%$ lower error rates on, respectively, automatic speech recognition, speaker verification and emotion recognition, without any need for empirical search.

\section{Related works and motivations}
\label{sec:related}

 \begin{figure*}[t!]
  \centering
  \includegraphics[width=1\linewidth, scale=0.17]{fixedblocks.pdf}
  \vspace{-0.5cm}
  \caption{Illustration of the training pipeline. The three steps are depicted: 1. Selecting the group of pseudo-labels and their corresponding weights; 2. SSL training with the selected pretext task; 3. Training on the downstream task with the pretrained SSL model. }
  \label{diagram}
  \vspace{-0.5cm}
\end{figure*}

SSL has recently been crucial in achieving state of the art results, especially in low-resource setups, either in speech \cite{baevski2020wav2vec, conneau}, natural language processing \cite{lan2019albert, chen2020big} or computer vision \cite{gidaris2019boosting, misra2020self, jing2020self}. As no labeled data is needed, SSL leverages large amounts of data to train large deep neural networks, often over long periods of time. This makes the proper understanding of what makes a good SSL model crucial to lower the amount of empirical testing, and thus, time and compute needed to obtain the best downstream performance.

\textbf{SSL for Speech.}  Learning self-supervised speech representations has recently enabled speech researchers to reach state-of-the-art results on various speech processing tasks \cite{fan2021exploring}. Future prediction and contrastive learning are the main paradigms used in the most successful works  \cite{chung2019unsupervised, baevski2020wav2vec,  hubert,  shor2021universal} lowering the error rates of several tasks even in low-resource settings. These self-supervised representations are thoroughly compared in extended benchmarks covering different languages \cite{superb, evain2021lebenchmark}. However, in contrast to our proposition, there were no attempts in  these works to motivate beforehand the different choices made in the design of the self-supervision task.

\textbf{Understanding SSL.} While the self-supervised learning field has been mainly empirical for the moment, a few works have tried to theoretically justify the success of the main techniques and choices. Exploring the different paradigms of self-supervision, various approaches have been proposed to understand what leads to a good self-supervised representation \cite{Arora2019, lee2020predicting, wei}. On the one hand, contrastive learning \cite{oord2018representation, chen2020simple}, \textit{i.e.} learning representations through enforcing the proximity of the embeddings of similar data points, has been studied both theoretically and empirically in regards to the mutual information shared by alternative views of the data, highlighting the goal of sharing just the information related to the class-specific content \cite{bachman, tschannen2019mutual, tian}. In a recent work from \cite{li2021selfsupervised}, independence testing has been used to produce better transformations in contrastive learning settings for image representations. In predictive learning settings, on the other hand, the representations are learned through the prediction of masked elements in sequential data. Recent works on language modeling have theoretically shown how this technique is useful for downstream tasks that can be reduced to a masking problem \cite{saunchi}. However, all these works have been focusing on computer vision or text-related applications, and none of them addressed the problem of multitasked self-supervision.

\textbf{Multi-task self-supervised learning.} The literature on multi-tasking has been mainly tackling classic supervised learning settings, and is still limited for the self-supervision cases. Regarding the former setting, it has been shown that similar enough tasks can take advantage of being solved with a common encoder \cite{Chen2015, zamir2018, Dwivedi2019,shafey}. Concerning multitasked self-supervision, the combination of pretext tasks has been mainly explored in computer vision and speech \cite{pascual2019learning, ravanelli2020multitask}. In vision, pretext tasks such as colourisation, jigsaw \cite{doersch2016unsupervised} and rotation \cite{gidaris} have improved downstream performance when combined \cite{jigsawpluss, ShinyaYamaguchi}. The two closest works to our line of research are from \citet{lee2020predicting} and \citet{Doersch}. The former develops a theoretical link  between conditional independence and the downstream task performance, while the latter adapted the depth in the selection of the embeddings from a multitask self-supervised encoder according to the pretext task to be solved. However, in both papers, no practical and theoretical solutions for the group selection of pretext tasks in a multitask SSL model has been proposed.

\textbf{Group feature selection.} Feature selection, and more precisely group feature selection is one of the closest problems to the considered one, and has been a source of inspiration as we will further consider these lines of works for concurrent baselines. The interactions between features, and the best way to combine them have been largely studied in the classical supervised learning literature \cite{guyon}, especially for linear classifiers. These investigations led to various solutions to the feature group selection problem, including regularization-related ones like the LASSO-based techniques \cite{grouplasso}, or multiple kernel formulations \cite{sonnenburg, bach}. However, these works do not consider any notion of self-supervision, and nothing proves the link between feature selection and self-supervision pretext-task selection.

The aim of this work is to shorten the process of SSL model design and at the same time provide insight into the importance of pseudo-labels and the underlying mechanisms between the self supervised training and downstream tasks. We chose to experiment with speech because of the paucity of literature in this area on multi-task SSL and the availability of a variety of pseudo-labels based on decades of signal processing research. The entire pipeline from acoustic feature extraction to downstream tasks performance consists of three main steps, shown in Figure \ref{diagram}. Firstly, for all downstream tasks, the method generates a selection and weighting of the pretext tasks. Second, the SSL model is trained. Finally, its encoder is used as a feature extractor front-end for one or more downstream tasks.

\section{Conditional independence for utility estimation}
\label{sec:CI}

As a first step, in order to improve the performance of the downstream task, we need a function that estimates the utility of learning to solve a considered pretext task. Here, we use the estimation of the conditional independence between the pseudo-label values and the downstream data samples, given the final downstream labels. In the following, we explain the theoretical motivation and describe the computational procedure.

\subsection{Problem definition and intuition}

 Let $X$, $Y$ and $Z$ be, respectively, the downstream data points, their downstream labels and their pseudo-labels. Let also $\mathcal{C}$ be the set of possible downstream classes. As an example, if one considers emotion recognition as a downstream task, $X$ would be the speech samples, $Y$ the emotions corresponding to the samples, $\mathcal{C}$ the set of possible emotions, and $Z$ an automatically generated signal feature, such as the fundamental frequency.

 As stated in Section~\ref{sec:related}, \citet{lee2020predicting} linked the utility of a pseudo-label ($Z$) to the conditional independence (CI) between $Z$ and $X$ given $Y$. The approach asserts that, given the labels $Y$, one may seek to \textit{quantify how much it is possible to predict the pseudo-labels $Z$ without much information about $X$}. The authors bounded, under constraining assumptions mainly concerning the nature of the classifiers, the downstream classifier error with a function depending of the downstream training dataset size, and an estimate of the CI. More precisely, the main theorem shows that the bounding function decreases linearly with the training set size of the downstream task ($M$) and quadratically with CI, indicating that it is a promising estimator of the pseudo-label utility.

 The proposed estimator is a function of the final downstream task to be solved. This is motivated by two main reasons. First, the extensive literature on feature selection for different speech and computer vision tasks \cite{schuller,serizel,Wang_2019_CVPR, Liu_2020} shows that different tasks require different aspects of the data to be described. This suggests that different downstream tasks may perform better after different pre-trainings. A second argument is the difficulty to evaluate representations quality intrinsically, as speech representations perform differently from one task to another. A few intrinsic quality metrics and tests \cite{schatz, carlin, lakhotia2021generative} have been proposed for speech, but the correlation between these and the final downstream-task performance is not systematic \cite{https://doi.org/10.48550/arxiv.2007.13542,Gump2020}. Finally, recent experiments adapting the self-supervised representation to the speaker identification task have shown substantial improvements compared to task-agnostic representations \cite{chen2021unispeechsat}, validating our intuition that downstream task oriented SSL is an interesting trend towards better downstream performances.

It is notoriously difficult to correctly estimate the extent to which two variables are independent conditionally on a third variable on realistic data \cite{hardness}. In our previous work we proposed a simple method for estimating conditional independence for speech data \cite{Zaiem_2021}. This method proved to be effective for the selection of individual pretext tasks, as  the utility estimator was highly correlated with the final downstream performance. This approach relies on performing a classical independence test on data sliced according to the downstream class.

\subsection{Conditional independence estimator computation}

This section details the computation of the conditional independence estimate that is  used as a measure of pseudo-label utility. Let  $X =\{x_i\}_{i \in \{0, \ldots, M\}}$ with $M$ being the cardinal of $X$ and $x_i$ data samples (\textit{e.g.} Mel-band spectra for speech). Every sample $x_i$ has a corresponding downstream label $y_i$ and an automatically generated pseudo-label $z_i$. We assume that $y_i$ is discrete reducing the task to a classification problem such as with emotion class for emotion recognition. We also assume that for every pretext-task $Z$, a single $z_i$ value corresponds to each $x_i$. In our case, $z_i$ values are the mean of the frame-level pseudo label values.

For independence testing, we decided to rely on the kernel-based Hilbert Schmidt Independence Criterion (HSIC) \cite{gretton} for two reasons. First, HSIC has already proven successful  in testing statistical dependence between translated sentences \cite{gretton} on sequential data like text. Secondly, the kernel-based technique facilitates the handling of multivariate and variable-length data such as speech, as the estimation then boils down to the calculation of similarity measures between the considered variables.

\textbf{Computation steps.} Estimating the CI of a pseudo-label $Z$ for a downstream task $(X,Y)$ consists of three steps. First, we start by partitioning the data sample $X$ according to downstream (discrete) classes. Next, for each downstream class $c$, we compute the kernel matrices $K_c$ and $L_c$, which, respectively, represent the similarity measures between the data samples and the pseudo-labels.
Finally, we perform the independence test for every slice using $K_c$ and $L_c$ and aggregate the estimates with a weighted mean that takes into account the number of samples per downstream class. Thus, for two speech points $x_i$ and $x_j$, with their two pseudo-label values $z_i$ and $z_j$, the coefficients of the similarity matrices $K_c$ and $L_c$ are calculated as follows:
\begin{align}
K_{ij} &= K(x_i,x_j) = \cos(GD(x_i) ,GD(x_j) ). \\
L_{ij} &= RBF(z_i, z_j),
\end{align}
with $GD(.)$ the Gaussian Downsampling function (more details about this embedding method in the appendix \ref{kernels}) , $\cos(.,.)$ the cosine similarity, and $RBF(., .)$ the Radial Basis Function kernel, defined hereafter:
\begin{align}
    \cos(x,x') &= \frac{trace(x^T x')}{||x|| . ||x'||} . \\
    RBF(z,z') &= \exp (-\frac{|| z-z'||^2}{2 \sigma^2}),
\end{align}
where $\sigma$ is the width of the RBF kernel and $trace(.)$ is the function rendering the sum of elements of the main diagonal of the matrix. Note that we compute the matrices $K_c$ and $L_c$, for each group of samples sharing the same downstream class $c \in C$. Hence, $K_c$ and $L_c$ correspond to the definitions above, but only restricted to the points with $c$ as a downstream label. For each downstream class $c$, and as in \cite{gretton}, the HSIC value is obtained by computing:
\begin{equation}
HSIC_c(X, Z) = \frac{1}{n_c^2} trace(K_c H_c L_c H_c),
\end{equation}
with $H_c= I_{n_c} - \frac{1}{n_c}1_{n_c} 1_{n_c}^T $, $n_c$ being the number of points holding downstream label the $c$, and $1_{n_c}$ a vector of ones of size $n_c \times 1$.

The HSIC value is non-negative and corresponds to the Hilbert norm of their cross-covariance matrix. It is used to characterize the independence of the two considered variables. Intuitively, the HSIC value is high if samples similar in $K_c$ are similar in $L_c$. As a result, the lower this value is, the more conditionally independent the two arguments of HSIC are and the better the pseudo-label should be for self-supervision before fine-tuning on the downstream class. The final value for a given pseudo-label and a downstream task is expressed as:
\begin{equation}
HSIC(X,Z |Y) = \frac{1}{M}\sum_{c \in \mathcal{C}} HSIC_c(X, Z) \times n_c.
\end{equation}

\section{Pretext task group selection and weighting}
\label{sec:group}
The first part explained how to select individual pretext tasks among a set of candidates according to the downstream task to be solved. The next is now to define a way to select a group of pretext tasks and combine them optimally within the same pre-training process. Hence, we introduce a method to select a group of pseudo-labels and weight their respective losses to increasing or decreasing their contribution in the self-supervised representation. Our method aims to minimise the conditional dependence between the resulting weighted group pretext task, which consists in the prediction of a selected pseudo-label group,  and the downstream points given the downstream labels.

Given a set of $k$ possible pseudo-labels $(Z_i)_{i \in [0,k]}$, we want to estimate a set of parameters $(\lambda_i)_{i \in [0,k]}$ weighting the loss of every pseudo-label $Z_i$ during the pre-training phase. Hence, we define the grouping pseudo-label $Z_\lambda$ as an orthogonal concatenation of $(Z_i)_{i \in [0,k]}$ weighted with $(\lambda_i)_{i \in [0,k]}$  : $Z_\lambda = (\lambda_1 Z_1, ..., \lambda_k Z_k).$

As shown in appendix \ref{diffproof}, the custom conditional HSIC computation pipeline described above is fully differentiable with respect to $(\lambda_i)_{i \in [0,k]}$. During the HSIC computation, the data similarity matrices $\{K_c\}_{c \in C}$ are independent of $Z$ and therefore of $\lambda$, as they only depend on the speech samples. Only the pseudo-label similarity matrices $\{L_c\}_{c \in C}$ are changed. For every downstream class $c$, $L_c$ is defined as follows:
\begin{align}
[L_c]_{i,j} &= RBF((Z_\lambda)_i, (Z_\lambda)_j), \\
&= \exp( \frac{-1}{2\sigma^2} \sum\limits_{h=1}^{k} \lambda_h || z_{h,i} - z_{h,j} ||^2_2),\nonumber
\end{align}
with $z_{h,i}$ denotes the mean value of the $h$-th pseudo-label for the $i$-th data sample in the dataset.

 \subsection{Constraints on the weights}

 The conditional-independence based utility estimator is optimized with respect to the parameters $(\lambda_i)_{i \in [0,k]}$ weighting the losses of the pretext tasks in the pretraining. It should respect three constraints explained below.

 First, the parameters $(\lambda_i)_{i \in [0,k]}$ must be positive, because they are intended to be used as weights for the losses corresponding to their respective pretext tasks. A negative weighting loss would lack interpretability as it could imply that the self-supervised model should ``unlearn'' to predict the corresponding pseudo-label. While this is common in adversarial training methods, the current work does not explore this case.

 Second, the sum of the values of the weights must be high enough. Indeed, the presented method for estimating the conditional independence assumes that the considered pseudo-label $Z$ is not independent of $X$. It is fortunately true for speech features, as $Z$ is a function of $X$, but it may become false when altering highly the $Z$ values. For example, with $(\lambda_i)_{i \in [0,k]} = 0 \forall i\in [0,k] $, the utility estimator would be null and thus the lowest (\textit{i.e.} the best), but we would lose the assumption of non independence between $Z$ and $X$. Furthermore, with the $HSIC$ value decreasing with positive decreasing values of $(\lambda_i)_{i \in [0,k]}$ we need to ensure that the sum of the weights is significantly higher than zero, or it would mean that the losses of the pseudo-labels would be barely considered as their total contribution to the training loss is reduced.

 Finally, for a fair comparison between the weighting choices during the optimization of the conditional independence based estimator, the sum of the weights should remain constant. From now one, and to solve this issues and the one presented in the previous paragraph, the sum of the $(\lambda_i)_{i \in [0,k]}$ is fixed to one and the problem is summarized as:
\begin{align}
\label{eq:problem}
&\min_{\lambda \in \mathbb{R}^k}\quad  HSIC( Z_\lambda, X, Y ), \\
&\textrm{s.t.} \;  Z_\lambda = (\lambda_1 Z_1, ..., \lambda_k Z_k ),\nonumber \\
  &\lambda_i \geq 0, \:  \forall \: i \in [0,k], \; \nonumber
  \sum_i \lambda_i = 1.
\end{align}
To minimize the estimator quantity while respecting the constraints, the weights used in the computation of the CI value are the softmax output of free learnable parameters $(W_i)_{i \in [0,k]}$. The Softmax function enforces the conditions while being differentiable and the optimization problem becomes:
\begin{align}
\label{eq:secondproblem}
&\min_{W \in \mathbb{R}^k}\quad HSIC( Z_\lambda, X, Y ), \; \\
&\textrm{s.t.} \;  \lambda = Softmax(W) , \; \nonumber \\
 & Z_\lambda = (\lambda_1 Z_1, ..., \lambda_k Z_k ).\nonumber
\end{align}

 \subsection{Weights sparsity}
Sparsity is another characteristic that is desirable for the weighting vector in our case. Indeed, if a few pseudo-labels are superfluous or harming for the given downstream task, we would rather discard them completely than allocate them a low weight. Two reasons motivate this. First, by removing pretext tasks completely from the training, computation time would be saved as we would not need to extract the pseudo-labels in the first time, and in a second time, we would remove the predictor during the self-supervision training. Second, it would help with interpretability and clarity to understand which features are included or not in the latent space representation. The sparsity of the weighting vector is ensured using the Sparsemax function \cite{sparsemax} in place of the softmax in eq. \ref{eq:secondproblem}. Sparsemax is a differentiable function that outputs a sparse positive vector that sums to 1 while maintaining the property of differentiability of the whole computation.

\section{Experimental study}
\label{sec:exps}

This section describes the experiments validating the proposed estimator. It details the group selection and weighting processes (Section \ref{subsec:group}), the self-supervised learning models (Section \ref{subsec:models}), the downstream tasks considered (Section \ref{subsec:tasks}), and the obtained results (Section \ref{sec:res}).

\subsection{Group selection and weighting}
\label{subsec:group}

The considered pseudo-labels  are extracted using the OpenSmile library \cite{opensmile}.  The selected are mostly related to prosody and spectral descriptors as the signal feature processing literature commonly associates them to the three downstream tasks considered in our experiments (\textit{i.e.} speech, speaker verification and emotion recognition). They include:  \textit{Zero Crossing Rate, L1 Norm of Rasta Spectrum} \cite{rasta}, \textit{Loudness, F0, Voicing, $\alpha$ Ratio} \cite{alpha}, \textit{log of Harmonicity to Noise Ratio} \cite{hnr}. Then, and as pictured in Figure \ref{diagram} (\textit{step 1}), we combine these pseudo-labels in a single pretext task through weighting them with the output of equation \ref{eq:secondproblem}.

Comparative baselines follow common feature group selection strategies or simple ones based on natural intuition. The first one simply bundles all the pseudo-labels together with a constant weighting (\textit{i.e.} $\lambda=1$ for all pseudo-labels) as proposed in PASE \cite{pascual2019learning}. As SSL group pretext-task selection is yet to be fully explored, and as stated in the related works part, the two other considered baselines are inspired from the feature selection literature as it represents the closest research tracks with well established techniques. Our CI-based pretext task selection method is compared to Maximum Relevance Minimum Redundancy (MRMR, \cite{mrmr}) and Recursive Feature Elimination (RFE, \cite{RFE}). Appendix \ref{featureselection} gives more details about these concurrent baselines, while Appendix \ref{feature_weighting} shows the workers selected and their corresponding weights for every experiment. For instance, Noise (HNR) seems to be the most important feature to learn to predict for speaker recognition while fundamental frequency is useful for Emotion Recognition.
\begin{table*}[!t]
\centering
\caption{Results observed with the proposed group selection and weighting strategies on the two considered downstream tasks. Word Error Rate (WER) Equal Error Rate (EER), and Accuracy (Acc) are expressed in percentage and used for LibriSpeech$100$, VoxCeleb 1 and IEMOCAP respectively (\textit{i.e.} lower is better). ASR results are given with and without Language Modeling (LM). All SSL models contain $16.3M$ neural parameters. }
\scalebox{0.75}{
\begin{tabular}{@{}lcccccc@{}}
\toprule
\multicolumn{1}{c}{\textbf{Models}} & \multicolumn{2}{c}{\textbf{LibriSpeech} \textit{(WER \% $\downarrow$)}}         & \multicolumn{1}{c}{\textbf{VoxCeleb1} \textit{(EER \% $\downarrow$)}}
& \multicolumn{1}{c}{\textbf{IEMOCAP} \textit{(Acc \% $\uparrow$)}}
\\

\\
 & \textit{No LM} & \textit{ LM}  & &
 \\ \hline \\
PASE+ \cite{ravanelli2020multitask}                      & 25.11        & 16.62              & 11.61      &     57.86     \\
vq-wav2vec \cite{baevski2020vqwav2vec} & 17.71                 & 12.80              & 10.38     &   58.24       \\
\midrule
Selections                               & &              &    &       \\

All                             & 21.98   $\pm$ 0.36                 & 11.70  $\pm$ 0.27             & 11.90$\pm$ 0.32       & 56.4 $\pm$  1.3
\\

MRMR                                   & 18.94 $\pm$ 0.34                  & 10.36 $\pm$ 0.26                 & 10.56 $\pm$ 0.31    &     59.6 $\pm$1.29  \\
Remaining                                 & 19.81 $\pm$ 0.34                 & 11.65 $\pm$ 0.27                & 11.67 $\pm$ 0.32  & 58.8   $\pm$   1.29 \\

RFE                               & 20.02  $\pm$ 0.34                 & 11.42  $\pm$ 0.27              & 11.91   $\pm$ 0.33      & 55.8 $\pm$ 1.3  \\

Softmax                                 & \textbf{13.17$\pm$ 0.28}                    & \textbf{8.00 $\pm$ 0.23}               & 9.24  $\pm$ 0.29   &   60.6  $\pm$ 1.27
\\
Sparsemax                               & 17.18 $\pm$ 0.32                   & 10.41  $\pm$ 0.26              & \textbf{8.63 $\pm$ 0.27}      &   \textbf{60.8 $\pm$ 1.28 }    \\

\bottomrule

\end{tabular}
}
\label{results}
\end{table*}

\subsection{Self-supervised training}
\label{subsec:models}

The second step that appears in Figure \ref{diagram} is now described. The SSL model learns representations through the predictions of the selected pseudo-labels. For every considered one, the loss is multiplied by the corresponding  weight proposed by our method. Inspired from conclusions of multi-task speech SSL works \cite{ravanelli2020multitask, jiang2020speech}, apart from the considered pretext task, the network learns to reconstruct the input Mel spectrograms, and to compute $40$-dimensional Mel-Frequency Cepstral Coefficients (MFCC). These two predictions prevent information loss harming heavily downstream performance and are constant in all our experiments. Therefore, for a given weighting vector $(\lambda_i)_{i \in [0,k]}$, the self-supervised loss is defined as follows:
\begin{equation}
    L_{SSL} = MSE_{mel} + MSE_{mfcc} + \sum\nolimits_{i=1}^{k} \lambda_i \ell_1(Z_i),
\end{equation}
with $MSE$ the mean squared error computed for Mel spectrogram ($MSE_{mel}$) and MFCC frame vectors ($MSE_{mfcc}$), and $\ell_1(Z)$ the $\ell_1$-loss of the pretext task consisting in predicting the pseudo-label $Z$.

Prior to extending our method to state-of-the-art architectures such as wav2vec 2.0 that are particularly costly to train, we propose to first employ a PASE-like model to empirically validate the approach.  The encoder is composed of three distinct parts: a VGG-like features extractor \cite{simonyan2015deep}, a bidirectional LSTM, and a two-layered dense neural network with leakyRelu activations. All the details of the architecture are given in the appendix \ref{ap:details}. Then, and inspired by \cite{ravanelli2020multitask}, the decoders consist in simple one-layered MLP predictors. The predictors are voluntarily limited in capacity as the downstream architectures in speech SSL benchmarks.

\textbf{SSL dataset.} The SSL model is learned on the training set of the English Common Voice dataset (version $5.1$, $700$ hours of training, \cite{ardila2020common}). Common Voice is a collection of audio speech recordings made by users around the world from their own devices. This makes it suitable for self-supervised learning due to its closeness to natural settings.: $700$ hours of voice is a relatively small amount compared to what is typically used for state-of-the-art SSL models. However, as it is larger than the amount typically available for SSL use cases such as low-resource languages, we believe it is a sound choice. as the LibriSpeech dataset is part of our downstream evaluation protocol, we decided not to use it for pre-training, thus reducing the strong bias.

\subsection{Downstream tasks}
\label{subsec:tasks}

For every considered downstream task, we compute the corresponding group of selected pseudo-labels and their respective weights and compare the downtream performance with the concurrent baselines. Experiments are run on three different downstream tasks leading to different groups of pseudo-labels: automatic speech recognition (ASR, with the $100$ hours train from LibriSpeech) speaker recognition (SR, with VoxCeleb 1), and emotion recognition (ER with the audio data from IEMOCAP). Datasets and downstream architectures are inspired from the SUPERB benchmark \cite{superb} for SSL representations and described in details in Appendix \ref{superbsettings}. Prior to downstream training, the SSL encoder is frozen as it is used as a feature extractor for the task-dependent downstream pipeline. We do not use any data augmentation during the SSL training and downstream finetuning, for a  more pristine comparison of the learned models.

\vspace*{-3mm}

\subsection{Results}
\label{sec:res}

\begin{table*}[t!]
\centering
\caption{Results observed retraining the Wav2vec 2.0 model with and without weighted pretext tasks using the Sparsemax function method. An experiment adding the naive selection of pretext tasks is also performed. ``Fr." and ``Fine." also respectively refer to Frozen and Finetuned settings. Adding selected pretext tasks improves the donwstream performance on all three considered tasks. All models contain $100M$ neural parameters. }
\vspace*{3mm}
\scalebox{0.75}{
\begin{tabular}{@{}lcccccc@{}}
\hline

\multicolumn{1}{c}{\textbf{Selections}} & \multicolumn{2}{c}{\textbf{LibriSpeech} \textit{(WER \% $\downarrow$)}}         & \multicolumn{2}{c}{\textbf{VoxCeleb1} \textit{(EER \% $\downarrow$)}} & \multicolumn{2}{c}{\textbf{IEMOCAP} \textit{(Acc \% $\uparrow$)}}
\\ \hline \\

                                        & Fr.          & Fine.          &
                                       Fr. & Fine.  &
                                       Fr. & Fine.  \\
 \midrule \\
wav2vec 2.0 \textit{BASE}                             &  17.93  $\pm$ 0.33                  & 10.21 $\pm$ 0.25      &7.20 $\pm$ 0.26 &5.35  $\pm$ 0.22          & 56.6 $\pm$ 1.2     &\textbf{74.0 $\pm$ 1.16}\\
wav2vec 2.0 \textit{BASE}  + Naive selection                          &  17.23  $\pm$0.32                  & {10.10} $\pm$ 0.25      &6.80 $\pm$ 0.25 &\textbf{5.05  $\pm$ 0.21  }        & 57.4 $\pm$ 1.3     &73.7 $\pm$ 1.16\\
wav2vec 2.0 \textit{BASE} -Sparsemax                             & \textbf{16.70 $\pm$ 0.31}                   &  \textbf{9.18 $\pm$ 0.24}      &\textbf{6.57 $\pm$ 0.25}& 5.30 $\pm$ 0.22           & \textbf{59.5 $\pm$ 1.29}     &\textbf{74.0 $\pm$ 1.16}\\ \hline

\end{tabular}
}
\label{wav2vecresults}
\end{table*}
Baselines detailed in Section \ref{sec:group} and described in length in Appendix \ref{featureselection} are respectively referred to as ``\textit{All}'', ``\textit{RFE}'' and ``\textit{MRMR}''.  The details about the tasks selected and their weights can be found in Appendix \ref{feature_weighting}. First, the results reported in Table \ref{results} show that, for the considered downstream tasks, the two proposed strategies (\textit{Sparsemax} and \textit{Softmax}) perform better than the group selection baselines with a  $3.28$ EER gain for \textit{Sparsemax} against the \textit{RFE} approach on VoxCeleb, and $8.81$ of WER with \textit{Softmax} compared to the \textit{All} baseline. Interestingly, simply regrouping all the pseudo-labels together leads to a poor performance as witnessed on LibriSpeech with a very high $21.98\%$ WER. The fact that these \textit{intuitive} regrouping methods harm the performance motivates the need for a better pretext task group selection strategy such as the one introduced in this work with the WER dropping to $13.17\%$. As expected, introducing the joint decoding with a language model strongly decreases the WER but also biases the comparison as disparities are smoothed with an exogenous neural model. Nevertheless, this does not affect the ranking of our weighting strategy as it stille outperforms the concurrent baselines. In the context of speaker recognition, \textit{Sparsemax} beats \textit{Softmax} with an EER $0.61$ lower. One may question how strongly the group selection are related to the tasks they were computed on. To answer this, we selected the best learned model for LibriSpeech (i.e. softmax weighting) and fine-tuned it on VoxCeleb1 and IEMOCAP. It obtains an EER of $10.55\%$ and a  $59.9\%$ accuracy respectively. Although it performs slightly better than most of the baselines, the difference between these results and the task-related selections suggests that the weightings are indeed optimized towards the downstream task.

\section{Extending wav2vec 2.0 to multitask SSL}
\label{sec:wav2vec}

Contrastive predictive coding (CPC) \cite{oord2018representation} based techniques like wav2vec 2.0 \cite{baevski2020wav2vec} currently top most of the leader-boards for speech-related tasks. In contrast, multi-task speech representation learning has not yet proven to be a state-of-the-art alternative.  In a recent work, Sadhu and al. \citeyear{sadhu21_interspeech} showed that combining a reconstruction loss and CPC improves the performance of the wav2vec 2.0 representations especially in noisy settings. Based on this idea, we propose to further validate our selection method by extending wav2vec 2.0 to multi-task SSL and demonstrating its scaling capabilities. Thus, we extend the learning loss :
\begin{equation}
    L_{SSL} = L_{W2V} + \sum\nolimits_{i=1}^{k} \lambda_i \ell_1(Z_i).
\end{equation}
The SSL model uses the \textit{BASE} wav2vec 2.0 described in  the work of Baevski \textit{and al.}   (\citeyear{baevski2020wav2vec}) and is trained on the same Common Voice dataset. The pre-training pipeline is implemented within SpeechBrain. The trained \textit{BASE} model is compared to that obtained with the official Fairseq implementation of wav2vec 2.0, with strictly equivalent results. The entire recipe alongside with the set of hyperparameters needed to properly train a wav2vec 2.0 model are released under our anonymous repository and will be made available with SpeechBrain afterwards.

We strictly follow the SUPERB benchmark choices \cite{superb} both at the data selection and finetuning architecture levels. Hence, and conversely to the previous experiments, the ASR system solely optimises the CTC criterion over characters. For each of the three tasks (\textit{i.e.} ASR, SV, ER) we compare the standard \textit{BASE} wav2vec 2.0 model with one trained following the Sparsemax selection of multitask SSL. Sparsemax is chosen over softmax as it enforces the desirable sparsity and removes completely a few pseudo-labels from the SSL training, which is one of the goals of this work.  Another experiment is led with the ``naive'' pretext-task selection where all the pseudo-labels are kept and a constant $0.5$ weight is used across all pretext-tasks.  Appendix \ref{feature_weighting} reports the exact weights used in the experiments. Each wav2vec 2.0 model required $24$ NVIDIA Tesla V100 GPUs to train for $150$ epochs (approximatively $40$ hours). Finally, we also  compare frozen and unfrozen (\textit{i.e.} where the SSL encoder is fine-tuned with the downstream task)  models.

It is clear from the results reported in Table \ref{wav2vecresults} that our approach improves the performance over the standard wav2vec 2.0 framework for every considered downstream task. While adding pretext tasks naively improves the final performance, the difference in performance between the naive selection and the sparsemax weighting shows the benefit of our method in getting the best downstream performance. Unsurprisingly this difference is small (though statistically significant in all but one case), as the wav2vec 2.0 BASE is already powerful and the additional workers are anyway useful. Here, it is worth noting that the difference in performance compared to the literature mostly comes from the pre-training conditions. For instance, wav2vec 2.0 is commonly pre-trained with larger models on LibriSpeech to achieve lower WER on this dataset.  This comparison can be found in appendix \ref{impact:change} as it could bias the evaluation of our method with pre-training and target datasets being the same.

The results reported in Table \ref{wav2vecresults} show that our method performs better than the standard wav2vec 2.0 framework for all the downstream tasks considered. Although the naive addition of pretext tasks improves the final performance, the performance difference between naive selection and weighting by sparsemax shows that our method is valid for obtaining the best downstream performance. This difference is  unsuprisingly small aswav2vec 2.0 BASE is already powerful. However, the additional workers are useful anyway, as the improvement is statistically significant in all but one case. It is worth noting here that the difference in performance compared to the literature is mostly due to pre-training conditions, and precisely the dataset change. For example, wav2vec 2.0 is generally pre-trained on LibriSpeech with a larger model and achieves lower WER on this dataset. This comparison is provided in the appendix \ref{impact:change}  showing that keepig the same pre-training and target datasets may bias the evaluation.

\section{Conclusion}
 We present in this work an approach that aims to create an useful pretext task in multi-task self-supervised learning by combining pseudo-labels quickly and conveniently. The method enables an optimal choice of pseudo-labels through an inexpensive optimization process which dramatically reduces the time and computational effort required to design an optimal multi-task SSL model. The method is validated on three speech-related tasks and outperforms common pseudo-label selection strategies when combined with simple or with state-of-the-art SSL models. This opens up a variety of possibilities for discovering and selecting new pretext tasks in self-supervised learning on speech and other types of data.

\section{Introduction}

\end{document}