\section{Introduction}
\label{sec:intro}

\begin{figure}[!t]
\centering
\includegraphics[width=3.2in]{img/pic1-3.png}
\caption{Qualitative comparison of the proposed TAFormer with SeqFormer in the cases of appearance deformations and similar object interference. For the first two columns, SeqFormer struggles to accurately segment the instance of the black dog (marked by yellow circles) because of fast motion and scale variation. For the last two columns, it is difficult for SeqFormer to maintain the consistency of the tiger class due to the visual similarity and occlusions. Our TAFormer alleviates these two issues and remarkably improves the mask quality. Best viewed zoomed in.
}
\label{Fig. 1}
\vspace{-3mm}
\end{figure}

Video instance segmentation (VIS) is firstly proposed in the work~\cite{yang2019video} and gradually becomes an emerging task in computer vision.
The goal of VIS is to simultaneously perform detection, segmentation and tracking of the instances in the given videos.
Generally speaking, there are mainly two kinds of approaches.
First, the classic methods known as tracking-by-detection~\cite{bertasius2020classifying,yang2019video,cao2020sipmask,fu2021compfeat,liu2021sg,li2021spatial,yang2021crossover,xu2021segment} apply the standard image instance segmentation approaches~\cite{he2017mask, wang2020solo, wang2020solov2} to the video frame by frame.
A post-process step is applied to
track instances across frames by motion prediction~\cite{bertasius2020classifying}, classification~\cite{qi2021occluded}, and re-identification~\cite{cao2020sipmask, li2021spatial}.
However, such a tracking-by-detection paradigm is sensitive to appearance deformation such as occlusions, motion blur, and large variance in object scales.
The second branch of approaches~\cite{wang2021end,hwang2021video, wu2021seqformer,cheng2021mask2former,yang2022temporally} divide the whole video into multiple clips with overlapping and process the video clip by clip.

Thanks to the emerging advances in vision transformer architecture~\cite{carion2020end,zhu2020deformable,sun2021rethinking,liu2021swin}, recent transformer based VIS works~\cite{wang2021end,hwang2021video, wu2021seqformer,cheng2021mask2former,yang2022temporally} follow the second clip-level paradigm and represent each instance as a learned query embedding.
Specifically, VisTR~\cite{wang2021end} is the first approach that applies transformer to VIS.
It processes the video clip and directly regards the task of VIS as an end-to-end parallel sequence prediction problem~\cite{carion2020end}.
Subsequently, IFC~\cite{hwang2021video} proposes an inter-frame communication transformer and remarkably improves the computation and memory efficiency of frame-to-frame communications.
Moreover, TeViT~\cite{yang2022temporally} proposes a nearly convolution-free and temporally efficient vision transformer.
In SeqFormer~\cite{wu2021seqformer}, the instance of each frame is located and the corresponding salient representation of video-level instance is aggregated by temporal information.

Although, the state-of-the-art SeqFormer~\cite{wu2021seqformer} and transformer based methods of the similar paradigm achieve superior performance in VIS, two main challenges (see \cref{Fig. 1}) still exist.
First, since SeqFormer~\cite{wu2021seqformer} extracts per-frame features independently,
the ignorance of the spatial and temporal information in transformer encoder module probably leads to the problem of appearance deformations due to the fast motion, occlusion, and scale variation.
Second, the classification loss generally adopted in the image object detection task is not sufficient to discriminate the instances of the same category in the videos especially when the multiple instances are of the same category and visually similar.

In this paper, we propose the Temporal-Aware Transformer (TAFormer) on top of SeqFormer~\cite{wu2021seqformer} and try to alleviate the two problems mentioned above.
To alleviate the appearance deformation issue, in transformer encoder, we extend the deformable attention module~\cite{zhu2020deformable} to the spatio-temporal domain,
and introduce a spatio-temporal joint multi-scale deformable attention (STJ-MSDA) module to aggregate spatio-temporal features.
In the transformer decoder, a standalone instance query is decomposed to frame-level box queries where each of them represents the box location in the corresponding frame.
We observe that the box location in the current frame should be constrained by the corresponding box locations in adjacent frames.
Therefore, we introduce the consistency constraint to enhance the frame level box queries with the temporal relation.
To enhance the discriminability of visually similar instances, we also propose an instance-level contrastive loss on the box queries, where box queries from the same instance are positive samples and the others are negative samples.

We evaluate TAFormer on two benchmark VIS datasets (Youtube-VIS2019~\cite{yang2019video} and Youtube-VIS2021~\cite{yt2021}) with no additional training data.
With the ResNet50~\cite{he2016deep} backbone, our method achieves 48.1\% $\rm AP$ on Youtube-VIS2019 and 42.1\% $ \rm AP$ on Youtube-VIS2021, surpassing previous state-of-the-art method Mask2Former~\cite{cheng2021mask2former} by up to 2\%.

\begin{figure*}[!t]
\centering
\includegraphics [width=6.8in]{img/img1.png}
\caption{The overall illustration of our proposed TAFormer, mainly consisting of the backbone, spatio-temporal encoder, temporal augmented decoder, and output head. More specifically, given the multi-scale feature maps of a video clip, our encoder first extracts spatial and temporal features from the current frame and bidirectional neighboring frames by multi-scale deformable attention. Afterwards, the encoder dynamically integrates the spatial and temporal features to obtain enriched spatio-temporal features. For the temporal augmented decoder, the initial instance query is decomposed into frame-level box queries such that the temporal self-attention is conducted to keep consistency across frames.
Finally, the output head combines the encoder feature maps with instance queries to generate the sequential  segmentation results. In this figure, the same color represents the box queries belonging to the same video instance.  Best viewed in color.}
\label{Fig. 2}
\vspace{-3mm}
\end{figure*}

\section{Related Works}

\noindent \textbf{Image Instance Segmentation.} Image instance segmentation is a complex computer vision task that combines object detection and semantic segmentation, requiring pixel-level classification and localization of each instance. Image instance segmentation methods can be divided into three groups: two-stage, one-stage, and query-based methods.
Two-stage methods~\cite{he2017mask,cai2018cascade,chen2019hybrid,huang2019mask,kirillov2020pointrend}
first generates regions of interest and then performs regression in each ROI for classification and segmentation. Mask RCNN~\cite{he2017mask} is the representative two-stage detection framework, which is an extension of Fast RCNN~\cite{ren2015faster}.
One-stage methods~\cite{dai2016instance,li2017fully,chen2019tensormask,chen2020blendmask} get the results directly without proposal generation and the representatives include PolarMask~\cite{xie2020polarmask}, YOLACT~\cite{bolya2019yolact}, SOLO~\cite{wang2020solo} and CondInst~\cite{tian2020conditional}.
Query-based instance segmentation methods such as ISTR~\cite{hu2021istr}, QueryInst~\cite{fang2021instances}, and SOLQ~\cite{dong2021solq} treat the segmentation as a set prediction problem.
These methods use queries to represent the interest objects and perform regression on classification, detection, and mask simultaneously.

\noindent \textbf{Frame Level Video Instance Segmentation.}
The VIS task aims to classify and segment the instances in each frame and associate the same instance across the whole video.
VIS methods can be divided into two groups: frame-level methods~\cite{yang2019video,li2021spatial,fu2021compfeat,bertasius2020classifying,cao2020sipmask} and clip-level methods~\cite{athar2020stem,wang2021end,hwang2021video,wu2021seqformer,cheng2021mask2former}. The frame-level methods extend the image instance segmentation model to the VIS task by introducing an additional tracking branch. As a pioneering method, MaskTrack R-CNN~\cite{yang2019video} extends Mask R-CNN~\cite{he2017mask} and introduces an additional tracking head to address the issue of instance association. SipMask~\cite{cao2020sipmask} introduces the one-stage image detector FCOS~\cite{tian2019fcos} and YOLACT~\cite{bolya2020yolact++} to segment instances and associates them in a similar manner. STMask~\cite{li2021spatial} proposes a temporal fusion module to capture temporal correlation, the time-enhanced features are used for segmentation on each frame. CompFeat~\cite{fu2021compfeat} aggregates the temporal and spatial context information to refine the segmentation results in challenging video scenarios. Maskprop\cite{bertasius2020classifying} utilizes the predicted masks generated by the Hybrid Task Cascade Network\cite{chen2019hybrid} and propagates them to other frames for better association.

\noindent \textbf{Clip Level Video Instance Segmentation.}
The clip-level methods take a video clip as input and generate the sequential segmentation results.
STEm-Seg~\cite{athar2020stem} takes a video clip as a spatio-temporal volume and separates the instances by clustering.
More recently, there are various clip-level methods built upon the vision transformer~\cite{dosovitskiy2020image,child2019generating,ainslie2020etc,zhu2021long,liu2021swin,zhu2020deformable,zhou2022transvod,wang2022deformable} and making remarkable progress.
As a representative method,
VisTR~\cite{wang2021end} designs frame-specific instance queries and proposes a new transformer paradigm of joint tracking and segmentation.
However, the dense full attention across the spatio-temporal domain leads to expensive calculations.
In order to reduce the computation of VisTR, IFC~\cite{hwang2021video} proposes memory tokens in the transformer encoder and adopts video-level object queries for segmentation.
Later on, Mask2Former~\cite{cheng2021mask2former} and SeqFormer~\cite{wu2021seqformer} improve the performance on the benchmark datasets.
Specifically, Mask2Former~\cite{cheng2021mask2former} introduces a masked-attention within the transformer architecture for instance segmentation.
It trivially generalizes to video segmentation by directly predicting 3D segmentation results.
SeqFormer~\cite{wu2021seqformer} introduces a frame-independent transformer architecture, which decomposes the video-level instance queries to each frame for frame-wise detection and segmentation.
Both of them integrate Deformable DETR~\cite{zhu2020deformable} and introduce multi-scale representations in a low computational cost manner.
But there are only intra-frame multi-scale features utilized in attention computation.
Thereby, they ignore the spatio-temporal correlation across frames, which is crucial to handle challenging scenarios.
Our model is designed to sufficiently exploit the temporal context across frames, which can effectively handle these challenges and leads to more accurate results.

\section{Methods}
\subsection{Overall Architecture}

As illustrated in \cref{Fig. 2}, we propose Temporal-Aware Transformer (TAFormer) for video instance segmentation, which efficiently exploits the temporal correlation within a video clip. Our proposed TAFormer consists of four main modules: a vision backbone, a spatio-temporal transformer encoder, a temporal augmented transformer decoder, and an output head. Herein, let $\left \{  F_i \right \}_{i=t-d}^{t+d}$ be the input clip, which consists of frame $t$ and its bidirectional neighboring frames with a maximum distance of $d$. The backbone network extracts multi-scale features frame by frame. In each stage of the transformer encoder, we utilize a novel spatio-temporal joint deformable attention (STJ-MSDA) as the critical component of our pipeline. In particular, there are three sub-modules in the proposed STJ-MSDA architecture, mainly including spatial multi-scale deformable attention (S-MSDA),  temporal multi-scale deformable attention (T-MSDA) and dynamic attention fusion (DAF). Moreover, we further integrate the temporal self-attention block into the query-decomposed decoder~\cite{wu2021seqformer} to enhance the temporal correlation on frame-level box queries. Finally, we employ a contrastive loss~\cite{Wu2018UnsupervisedFL} over box queries from all frames for robustness during training.

\subsection{Spatio-temporal Transformer Encoder}
\label{3.2}

Lacking temporal information hinders promising segmentation results in challenging video scenarios. To solve this problem, we propose a Spatio-temporal Transformer Encoder (ST-Enc), which is composed of a stack of STJ-MSDA modules. Different from the original Multi-scale Deformable Attention Module~\cite{zhu2020deformable}, our proposed STJ-MSDA module employs temporal attention from adjacent frames and selectively aggregates it with spatial attention to obtain deeper context-aware representation.
Specifically, given an element query $q$ from frame $t$, STJ-MSDA first attends multi-scale spatial locations in the current frame and neighboring frames, facilitates acquiring highly-independent spatial and temporal deformable attentions.
It then dynamically aggregates information from these two attention via DAF. In this way,  more discriminative spatio-temporal representations can be provided. We elaborate on each component in detail in the following paragraphs.

\noindent \textbf{Spatial Multi-scale Deformable Attention.} Following Deformable DETR~\cite{zhu2020deformable},
S-MSDA computes attention on points sampling from the multi-scale feature maps of the current frame.
We define the $l\text{-}th$ feature map from frame $t$ as  $x^{l}_t \in  \mathbb{R}^{C\times H_{l} \times W_{l}}$ and the normalized coordinates of the reference point for $q$ as $\hat{p} _{q} \in {[0,1]}^2 $. Then given the query feature $z_{q}$, the S-MSDA can be described as follows:
\begin{normalsize}
\begin{equation}
\begin{aligned}
\mathcal{S\text{-}MSDA} & (z _{q}, \hat{p} _{q}, x_{t}^{l}) = \sum_{m=1}^{M}W_{m} \cdot [\sum_{l=1}^{L}\sum_{k=1}^{K_{intra}} A_{mlqk} \\ & \cdot W_{m}^{'} x^{l}_t(\phi_{l}(\hat{p} _{q})+\Delta p_{mlqk} ) ],
\end{aligned}
\end{equation}
\end{normalsize}
where $m$ denotes the index of multiple attention heads , $K_{intra}$ is the total sampled number from each feature level. $\Delta p_{mlqk}$ and $A_{mlqk}$ represent the pixel offset and attention weight of the $k\text{-}th$ sampling point in the $l\text{-}th$ feature level. Both of them are obtained via linear projection over $z_q$. The attention weight is normalized over all sampled points by $\sum_{l=1}^{L}\sum_{k=1}^{K_{intra}}A_{mlqk}=1$. Function $\phi_{l}$ re-scales the normalized coordinates $\hat{p} _{q}$ to the input feature map of the $l\text{-}th$ level.

\noindent \textbf{Temporal Multi-scale Deformable Attention.} We extend S-MSDA to the temporal domain and propose T-MSDA. The core idea of the T-MSDA is that, for each query in the current frame, it attends to a small set of key sampled points from multi-scale feature maps of neighboring frames. As such, our method can aggregate enriched temporal representation and maintain temporal consistency of the foreground instances. It can be formulated as follows:
\begin{normalsize}
\begin{equation}
\begin{aligned}
\mathcal{T\text{-}MSDA} & (z _{q}, \hat{p} _{q}, x_{t'}^{l}) = \sum_{m=1}^{M}W_{m} \cdot [\sum_{t'=t-d \atop t'!=t}^{t+d}\sum_{l=1}^{L}\sum_{k=1}^{K_{inter}} \\ & A_{mltqk}  \cdot W_{m}^{'} x_{t'}^{l} (\phi_{l}(\hat{p} _{q})+\Delta p_{mtlqk} ) ],
\end{aligned}
\end{equation}
\end{normalsize}
where $t'$ denotes the neighboring frame with frame $t$. $K_{inter}$ represents the number of sampled points in each feature level across neighboring frames. The scaled attention weight is normalized  by $\sum_{l=1}^{L}\sum_{t'=t-d}^{t+d}\sum_{k=1}^{K_{inter}}A_{mltqk}=1$. $p_{mtlqk}$ and $A_{mltqk}$ are obtained by two different linear projection over $z_q$, respectively.

\noindent \textbf{Dynamic Attention Fusion.} Let $E_{t}^{intra}$ and $E_{t}^{inter}$ represent the attention maps for frame $t$ obtained by $\mathcal{S\text{-}MSDA}$ and $\mathcal{T\text{-}MSDA}$, respectively. To our knowledge, $E_{t}^{intra}$ contains more salient instance information in the current frame, but struggles to handle the appearance deformations across frames. In contrast, $E_{t}^{inter}$ contains deeper contexts in order to adapt to the appearance changes. Therefore, the spatial and temporal attention play different roles in various video scenes. Based on the analysis above, we propose the Dynamic Attention Fusion module to emphasize the semantic information with channel-wise attention. We first aggregate them by an element-wise addition as:
\begin{normalsize}
\begin{equation}
\begin{aligned}
E'=E_{t}^{intra} + E_{t}^{inter}.
\end{aligned}
\end{equation}
\end{normalsize}
Then, we apply a fully connected (FC) layer to achieve the adaptive selection from $E'$. After that, we employ another two FC layers to generate the gate vectors $\left \{  g_1,g_2\right \} $, such that the information flows from these two attention maps $\left \{  E_{t}^{intra}, E_{t}^{inter}\right \} $ can be controlled. We further utilize softmax function along the channel dimension to generate adaptive weights $\left \{  w_1,w_2\right \}$ as follows:
\begin{normalsize}
 \begin{equation}
\begin{aligned}
w_1=\frac{e^{g_{1}}}{e^{g_{1}} + e^{g_{2}}} \quad  w_2=\frac{e^{g_{2}}}{e^{g_{1}} + e^{g_{2}}}.
\end{aligned}
\end{equation}
\end{normalsize}
Finally, the enriched spatio-temporal features of frame $t$ can be obtained by weighted addition:
\begin{normalsize}
\begin{equation}
\begin{aligned}
M_{t} = E_{t}^{intra} \odot w_1 + E_{t}^{inter} \odot w_2.
\end{aligned}
\end{equation}
\end{normalsize}

\subsection{Spatio-temporal Positional Encoding}
Different from SeqFormer that only employs spatial positional encoding, we further extend the positional encoding to the spatio-temporal domain, which can be described as follows:
\begin{normalsize}
\begin{equation}
\begin{aligned}
e_{pos} = e_{pos}^{t} +  e_{pos}^{s},
\end{aligned}
\end{equation}
\end{normalsize} where $e_{pos}^{s} \in  \mathbb{R}^{C\times 1\times H_{l} \times  W_{l}}$ denotes the original spatial encoding in SeqFormer and $e_{pos}^{t} \in  \mathbb{R}^{C\times T\times 1 \times 1}$ represents the corresponding temporal positional encoding. Both $e_{pos}^{s}$ and $e_{pos}^{t}$ are obtained by sine and cosine functions of different frequencies. With spatio-temporal positional encoding, we can deal with arbitrary length video clip.

\subsection{Temporal Augmented Transformer Decoder}
\label{3.4}

Following the SeqFormer architecture that decomposes the instance query into frame-level box queries and aggregates the box queries to obtain a temporal-aware instance query, we further employ a Temporal Self-Attention (TSA) block to enhance the temporal relationship. More specifically, the Temporal Self-Attention computes attention on box queries belonging to the same instance but different frames to obtain temporal-aware box queries.
These box queries are then passed through the self-attention module and cross-attention module to query the enriched features from ST-Enc, thus accurately exploring location information.

\subsection{Contrastive Loss on Box Queries}
\label{3.5}
Our encoder aims to produce enriched spatio-temporal features, whilst the decoder focuses on improving the temporal consistency for predictions. As such, the problem of appearance deformation in video instance segmentation can be efficiently solved. However, considering the challenging scenarios with multiple visual similar instances, it is easy to accumulate association errors over time due to the non-discriminative embeddings among queries. Motivated by the success of contrastive learning~\cite{Wu2018UnsupervisedFL,he2020momentum}, we propose a temporal contrastive loss to map different instances box queries to distinct representations. For each video clip, this loss regards the box queries belonging to the same instance as positive samples and all other box queries as negative samples.

As illustrated in \cref{Fig. 3}, we define $B_t={\left \{  b_t^q \right \} }_{q=1}^{Q}$ be the box queries of frame $t$, where $b_t^q\in R^C$ and $Q$ is the total number of queries. The core idea of contrastive learning is to maximize agreement between differently augmented views of the same example in the latent space. Automatically, ${\left \{  b_{t}^{i}, b_{t'}^{i} \right \} }$ from frame $t$ and its neighboring frame $t'$ can be treated as the different views of the $i\text{-}th$ instance in terms of temporal domain. We first adopt two FC layers to generate the latent representations of box queries. Then, InfoNCE~\cite{oord2018representation}~ is employed between two frames. It focuses on maximizing the representations between the pairs of box queries with the same order and minimizing the similarity with other pairs, which can be described as follows:
\begin{normalsize}
\begin{equation}
\begin{aligned}
\mathcal{L}_{N}(B_{t}, B_{t'})=-\frac{1}{Q}\sum_{i}^{Q}log\frac{exp(s(b_{t}^{i}, b_{t'}^{i})/\tau )}{\sum_{j=1}^{Q}exp(s(b_{t}^{i}, b_{t'}^{j})/\tau)},
\end{aligned}
\end{equation}
\end{normalsize} where $\tau$ is temperature hyper-parameter and the similarity function $s(b_{i}, b_{j})$ is defined as the cosine similarity. We further compute $\mathcal{L}_{N}$ between all pairs cross frames to obtain the contrastive loss, which can be formulated by:

\begin{normalsize}
\begin{equation}
\begin{aligned}
\mathcal{L}_{cl}=\sum_{t}^{T}\sum_{t'\ne t}^{T}\mathcal{L}(B_{t},B_{t'}).
\end{aligned}
\end{equation}
\end{normalsize}
The total losses for TAFormer can be defined as follows:
\begin{normalsize}
\begin{equation}
\begin{aligned}
\mathcal{L}_{total} = \lambda_{cls}\cdot\mathcal{L}_{cls} + \mathcal{L}_{box} + \mathcal{L}_{mask} + \mathcal{L}_{cl},
\end{aligned}
\end{equation}
\end{normalsize} where $\mathcal{L}_{cls}$, $\mathcal{L}_{box}$ and $\mathcal{L}_{mask}$ represent classification loss, bounding box loss, and segmentation mask loss. Herein, we use focal loss~\cite{lin2017focal} as the classification loss. The box loss consists of L1 loss and GIou loss~\cite{rezatofighi2019generalized}: $\mathcal{L}_{box} = \lambda_{L1}\cdot\mathcal{L}_{L1} + \lambda_{giou}\cdot\mathcal{L}_{giou} $. The mask loss is formulated as the combination of dice loss~\cite{MilletariNA16} and binary mask focal loss:~ $\mathcal{L}_{mask} = \lambda_{dice}\cdot\mathcal{L}_{dice} + \lambda_{focal}\cdot\mathcal{L}_{focal} $. Following~\cite{zhu2020deformable, wang2021end, hwang2021video, wu2021seqformer, cheng2021mask2former}, we apply auxiliary losses~\cite{al2019character} on box queries after each decoding layer.

\begin{figure}[!t]
\includegraphics [width=3in]{img/contra_pic.png}
\caption{Contrastive learning on frame-level box queries.}
\vspace{-3mm}
\label{Fig. 3}
\end{figure}

\begin{table*}[h]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{lcccccccc}
\hline
\multicolumn{1}{c}{backbone}                     & Methods          & Volume    & Params & $\rm AP$   & $\rm AP_{50}$ & $\rm AP_{75}$ & $\rm AR_{1}$  & $\rm AR_{10}$ \\ \hline
\multicolumn{1}{l|}{} & MaskTrack R-CNN~\cite{yang2019video}  & ICCV19    & 58.1M  & 30.3 & 51.1 & 32.6 & 31.0 & 35.5 \\
\multicolumn{1}{l|}{}                            & STEm-Seg~\cite{athar2020stem}         & ECCV20    & 50.5M  & 30.6 & 50.7 & 33.5 & 37.6 & 37.1 \\
\multicolumn{1}{l|}{}                            & SipMask~\cite{cao2020sipmask}          & ECCV20    & \textbf{33.2}M  & 33.7 & 54.1 & 35.8 & 35.4 & 40.1 \\
\multicolumn{1}{l|}{}                            & CompFeat~\cite{fu2021compfeat}         & AAAI21    & -      & 35.3 & 56.0 & 38.6 & 33.1 & 40.3 \\
\multicolumn{1}{l|}{}                            & VisTR~\cite{wang2021end}            & CVPR21    & 57.2M  & 36.2 & 59.8 & 36.9 & 37.2 & 42.4 \\
\multicolumn{1}{l|}{}                            & MaskProp~\cite{bertasius2020classifying}         & CVPR20    & -      & 40.0 & -    & 42.9 & -    & -    \\
\multicolumn{1}{l|}{{ResNet-50}}                & CrossVIS~\cite{yang2021crossover}         & ICCV21    & 37.5M  & 36.3 & 56.8 & 38.9 & 35.6 & 40.7 \\
\multicolumn{1}{l|}{}                            & Propose-Reduce~\cite{lin2021video}   & ICCV21    & 69.0M  & 40.4 & 63.0 & 43.8 & 41.1 & 49.7 \\
\multicolumn{1}{l|}{}                            & PCAN~\cite{KeLDTTY21}             & NeurIPS21 &  -     & 36.1 &54.9& 39.4 & 36.3 & 41.6 \\
\multicolumn{1}{l|}{}                            & IFC~\cite{hwang2021video}              & NeurIPS21 & 39.3M  & 42.8 & 65.8 & 46.8 & 43.8 & 51.2 \\
\multicolumn{1}{l|}{}                            & Mask2Former~\cite{cheng2021mask2former}      & CVPR22    & -      & 46.4 & 68.0 & 50.0 & -    & -    \\
\multicolumn{1}{l|}{}                            & SeqFormer~\cite{wu2021seqformer}        & ECCV22    & 49.3M  & 45.1 & 66.9 & 50.5 & 45.6 & 54.6 \\
\multicolumn{1}{l|}{}                            & \textbf{TAFormer} & -         & 52.2M  & \textbf{48.1} & \textbf{71.6} & \textbf{52.5} & \textbf{45.9} & \textbf{56.6} \\ \hline
\multicolumn{1}{l|}{}            & CrossVIS~\cite{yang2021crossover}        & ICCV21    & 56.6M  & 36.6 & 57.3 & 39.7 & 34.4 & 41.6 \\
\multicolumn{1}{l|}{}                             & PCAN~\cite{KeLDTTY21}              & NeurIPS21 & -      & 37.6 & 57.2 & 41.3 & 37.2 & 43.9 \\
\multicolumn{1}{l|}{}                             & VisTR~\cite{wang2021end}            & CVPR21    & 76.3M  & 38.6 & 61.3 & 42.3 & 37.6 & 44.2 \\
\multicolumn{1}{l|}{ResNet-101}                             & SeqFormer$\ddagger$~\cite{wu2021seqformer}        & ECCV22    & 68.4M  & 49.0 & 71.1 & 55.7 & \textbf{46.8} & 56.9 \\
\multicolumn{1}{l|}{}                             & Mask2Former~\cite{cheng2021mask2former}      & CVPR22    & -      & 49.2 & 72.8 & 54.2 & -    & -    \\
\multicolumn{1}{l|}{}                             & \textbf{TAFormer}           & -         & 71.3M  & \textbf{50.6} & \textbf{74.2} & \textbf{57.4} & 46.0 & \textbf{58.1} \\ \hline
\multicolumn{1}{l|}{}            & SeqFormer$\ddagger$~\cite{wu2021seqformer}        & ECCV22    & 220.0M  & 59.3 & 82.1 & 66.4 & 51.7 & 64.4 \\
\multicolumn{1}{l|}{Swin-L}                               & Mask2Former~\cite{cheng2021mask2former}              & CVPR22 & -      & 60.4 & 84.4 & 67.0 & - & - \\
\multicolumn{1}{l|}{}                             & \textbf{TAFormer}            & -    & 222.9M  & \textbf{61.2} & \textbf{85.0} & \textbf{67.4} & \textbf{54.6} & \textbf{66.7} \\ \hline
\end{tabular}
\caption{Performance of our method compared to state of the art on YTVIS-19 validation set. $\ddagger$ denotes adopting 80K train2017 images containing YTVIS-19 categories for joint training.}
\label{Tab1}
\end{table*}

\begin{table*}[h]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{ccccccccc}
\hline
Methods           & Backbone & Volume    & Params & $\rm AP$   & $\rm AP_{50}$          & $\rm AP_{75}$          & $\rm AR_{1}$  & $\rm AR_{10}$ \\ \hline
VisTR~\cite{wang2021end}          & ResNet50 & CVPR21    & 57.2M  & 31.8 & 51.7          & 34.5          & 29.7 & 36.9 \\
IFC~\cite{hwang2021video}               & ResNet50 & NeurIPS21 & \textbf{39.3M}  & 36.6 & 57.9          & 39.3          & -    & -    \\
Mask2Former~\cite{cheng2021mask2former}        & ResNet50 & CVPR22    & -      & 40.6 & 60.9          & 41.8          & -    & -    \\
SeqFormer$\ddagger$~\cite{wu2021seqformer}          & ResNet50 & ECCV22    & 49.3M  & 40.5 & 62.5          & 43.6          & 36.2 & 48.0 \\
TeViT~\cite{yang2022temporally}             & MsgShifT & CVPR22    & 172.3M & 37.9 & 61.2          & 42.1          & 35.1 & 44.6 \\
\textbf{TAFormer} & ResNet50 & -         & 52.2M  & \textbf{42.1} & \textbf{63.8} & \textbf{44.8} & \textbf{38.6} & \textbf{49.2} \\ \hline
\end{tabular}
\caption{Quantitative results of VIS methods on the YTVIS-21 validation set. $\ddagger$ denotes adopting 80K train2017 images containing YTVIS-21 categories for joint training.}
\label{Tab2}
\vspace{-3mm}
\end{table*}

\section{Experiments}
\subsection{Experimental Setup}
\noindent \textbf{Datasets.} We evaluate our TAFormer on both Youtube-VIS2019 (YTVIS-19) and Youtube-VIS2021 (YTVIS-21) datasets. The YTVIS-19 dataset consists of 2238 training, 302 validation, and 343 test videos. Each video has been manually annotated in pixel-level and the semantic category number is 40. The YTVIS-21 dataset adds more samples on the basis of YTVIS-19 with 3859 high-quality videos, whilst the number of annotations is also doubled so that it is more difficult in multi-instance scenarios.

\noindent \textbf{Evaluation Metrics.} We adopt Average Precision (AP) and Average Recall (AR) to evaluate the performance of our model. In addition, the Intersection over Union (IoU) computation is also performed in the whole video.

\noindent \textbf{Implements Details.}  We choose ResNet-50~\cite{he2016deep} as our default backbone network unless otherwise specified. Similar to~\cite{zhu2020deformable}, the feature maps $\left \{ C_3, C_4, C_5 \right \} $ with resolution $1/8$, $1/16$ and $1/32$ from the backbone are utilized to obtain multi-scale information. We also obtain the feature $C_6$ with resolution $1/64$ via a convolution on $C_5$. These four features are concatenated along channel dimensions and sent to our transformer. We set encoder layers $N_{enc} = 6$, decoder layers $N_{dec} = 6$ and hidden dimension $C=256$. Both sampling key numbers $K_{intra}$ and $K_{inter}$ are set to 4 for our STJ-MSDA module. The number of instance queries is set to 300. The coefficients for losses are set as $\mathcal{\lambda}_{cls} = 2$, $\mathcal{\lambda}_{L1} = 5$, $\mathcal{\lambda}_{giou} = 2$, $\mathcal{\lambda}_{dice} = 5$, $\mathcal{\lambda}_{focal} = 2$.

Following ~\cite{wu2021seqformer}, we use the AdamW~\cite{DBLP:conf/iclr/LoshchilovH19} optimizer with a base learning rate of $1 \times 10^{-4}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, and weight decay $=10^{-4}$. We set the length of video clip $T = 5$ and the temporal sampling windows $d$ to 2 during training phase. Similar to~\cite{athar2020stem , lin2021video, wu2021seqformer}, we first pretrain our model on COCO for 24 epochs where each image is randomly rotated five times with $\pm 20$ degree as a pseudo video clip. Next, we finetune it on YTVIS dataset for 6 epochs.
The learning rate is reduced by a factor of 0.1 at 3th and 5th epochs during finetune training. Our model is trained on 4 NVIDIA A100 GPUs with 4 video clips per GPU. During reference, we input the down-scaled 360p video as a whole to the model. Therefore, no additional data association is required.

\begin{figure*}[!t]
\centering
\includegraphics [width=6in]{img/vis.jpg}
\caption{Qualitative results of our TAFormer on the YTVIS-19 validation dataset.}
\label{Fig. 4}
\end{figure*}

\subsection{Main Results}

We compare TAFormer with state-of-the-art VIS methods on YTVIS-19 in \cref{Tab1}. SeqFormer and Mask2Former take a strong transformer architecture and achieve promising results. However, both of them only extract multi-scale spatial features and ignore the temporal consistency across frames. Among these competitive methods, our TAFormer achieves a new state-of-art of the 48.1$\% \rm AP$, using the same backbone and training data. Specifically, it provides an absolute gain of 4.7$\%$ at the threshold of $\rm AP_{50}$ over SeqFormer. The higher $ \rm AP $ and $ \rm AR$ validate that our method is able to utilize enrich spatio-temporal information to handle complex scenes and optimize the segmentation results. With the backbone of ResNet-101, our method achieves the overall $\rm AP$ of 50.6$\%$.

\cref{Tab2} illustrates the results of all compared methods on YTVIS-21. Our approach brings an improvement of 1.5$\%$ on $\rm AP$ over the second best method Mask2Former. It is also worth noting that TAFormer outperforms Mask2Former by 3$\%$ at the overlap threshold of $\rm AP_{75}$. It indicates that TAFormer can generate stable trajectories in longer video sequences.

\cref{Fig. 4} visualizes the results of TAFormer in six challenging scenarios. It shows TAFormer can tackle these scenarios well and maintain stable tracking trajectories.

\subsection{Ablation Study}

In this section, we use SeqFormer as our baseline and conduct several ablation studies on it to demonstrate the effectiveness of our proposed method.  All models are evaluated on the YTVIS-19 validation dataset.
\begin{table}[!t]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{l|ccc}
\hline
Model                           & $\rm AP$   & $\rm AP_{50}$  & $ \rm AP_{75}$ \\  \hline
Baseline                        & 44.9 & 66.8 & 50.3\\
Baseline + ST-Enc            &  46.4  &68.7 & 51.5\\
Baseline + ST-Enc + TA-Dec & 47.0   & 69.5  & \textbf{52.7}\\
Baseline + ST-Enc + TA-Dec + CL & \textbf{48.1} &\textbf{71.6} & 52.5\\ \hline
\end{tabular}
\caption{Ablation Study of our contributions on YTVIS-19 validation dataset. ST-Enc, TA-Dec, and CL denote the
proposed spatio-temporal encoder, temporal augmented decoder, and contrastive loss on box queries, respectively.}
\vspace{-3mm}
\label{Tab3}
\end{table}

\noindent \textbf{Spatio-temporal encoder.} We first verify the effectiveness of the spatio-temporal encoder.
The results are listed in \cref{Tab3}.
Compared to the baseline model, ST-Enc provides an absolute gain of 1.5 $\%$ $\rm AP$.
It proves that our encoder has the ability to aggregate meaningful and scalable context features from both intra and inter frames.
We further visualize the feature maps obtained by the baseline encoder and our encoder in \cref{Fig. 5}. It shows our encoder extracts better foreground regions.

Besides, we evaluate different schemes to aggregate the multi-scale deformable attention from intra-frame and inter-frame. As shown in \cref{Tab4}, the first variant simply combines $E_{t}^{intra}$ with $E_{t}^{inter}$ in an element-wise addition manner. The second variant concatenating $E_{t}^{intra}$ and $E_{t}^{inter}$ brings the $0.3\%$ $\rm AP$ improvement. Our dynamic attention fusion module further boosts the first variant by $0.5\%$ $\rm AP$. These experimental results indicate that our method can provide more discriminative embeddings from different frames and scales for the final segmentation. Thus, we adopt this strategy as the default setting for all experiments.

\begin{table}[!t]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{l|ccc}
\hline
Fusion strategies  & $\rm AP$   & $\rm AP_{50}$  & $ \rm AP_{75}$ \\  \hline
Element-wise addition                & 45.9 & 67.1 & 51.1\\
Concatenation               &           46.2 & 68.2 & 50.9\\
Dynamic attention fusion               &  \textbf{46.4}  &\textbf{68.7} & \textbf{51.5}\\ \hline
\end{tabular}
\caption{Variants of spatio-temporal attention aggregation. Our dynamic attention fusion achieves a relatively stable performance among all approaches.}
\vspace{-3mm}
\label{Tab4}
\end{table}

In \cref{Tab5}, we study the impact of feature levels on overall performance. We first use a single-scale feature map for deformable attention on the baseline, which is the same as previous transformer-based methods~\cite{wang2021end}. We convert the pretrained weight provided in~\cite{wu2021seqformer} adaptively and get $41.7\%$ AP. The use of temporal sampling on single-scale features can improve $\rm AP$ to $42.6\%$. The multi-scale deformable attention within intra-frame can bring an additional gain of $3.2\%$ on the baseline, which is adopted in MaskFormer and SeqFormer. When integrating temporal multi-scale sampling and dynamic attention module into it, there is a more significant gain ($1.5\%$) on overall $\rm AP$. It proves that the attention module proposed in our encoder leads to more accurate segmentation results since more context information can be utilized.

\begin{figure}[!t]
\centering
\includegraphics[width=3.2in]{img/feature_maps.png}
\caption{Visualization of the feature maps obtained by the encoder. We choose two difficult scenarios including occlusion and scale change. It can be clearly seen that our encoder enhances the regions of foregrounds and suppress the noise of backgrounds. The feature map is generated by average pooling along the channel dimension at the first-scale feature with the size of $H/4$ and $W/4$.}
\label{Fig. 5}
\end{figure}

\begin{table}[tb]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{c|ccc}
\hline
Model                                          & SFL & TFL & $\rm AP$   \\ \hline
\multicolumn{1}{c|}{\multirow{2}{*}{Baseline}} & 1   & 0   & 41.7 \\
\multicolumn{1}{c|}{}                          & 4   & 0   & 44.9 \\ \hline
\multicolumn{1}{c|}{\multirow{2}{*}{TAFormer}} & 1   & 1   & 42.6 \\
\multicolumn{1}{c|}{}                          & 4   & 4   & \textbf{46.4} \\ \hline
\end{tabular}
\caption{Impact of the feature levels used for sampling points prediction. SFL and TFL denote the spatial feature levels and temporal feature levels, respectively. }
\vspace{-3mm}
\label{Tab5}
\end{table}

\noindent \textbf{Temporal augmented decoder.} As shown in \cref{Tab3}, by performing temporal self-attention before spatial self-attention and deformable cross attention for box queries, it further brings the $ \rm AP$ to $47.0\%$. It is noticed that the performance gain becomes $1.2\%$ in higher overlap $\rm{AP}_{75}$. The reason is that temporal-enhanced box queries can attend more accurate locations with the movement of instances.

\begin{table}[h]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{ccccc}
\hline
CL       & Aux     & $\rm AP$ & $\rm AP_{50}$ & $\rm AP_{75}$ \\ \hline
             &              & 47.0 & 69.5         & \textbf{52.7}      \\
$\checkmark$ &              & 47.7 & 71.0         & 51.6      \\
$\checkmark$ & $\checkmark$ & \textbf{48.1} & \textbf{71.6}         & 52.5      \\ \hline
\end{tabular}
\caption{Impact of adding the contrastive loss. Aux denotes adding the auxiliary losses for contrastive learning.}
\vspace{-3mm}
\label{Tab6}
\end{table}

\noindent \textbf{Contrastive learning. }\cref{Tab6}~ shows the effect of contrastive loss on frame-level box queries. We first apply the contrastive loss on the box queries generated by the last layer of the decoder. It can be seen that our proposed loss can boost the gain of $0.7\%$ AP. When training our model with auxiliary loss, we achieve the final $ \rm AP$ of $48.1\%$. To further illustrate the role played by contrastive loss, we visualize the box queries of the video named 00f88c4f0 in YTVIS-19 with t-SNE~\cite{van2008visualizing}, as shown in \cref{Fig. 6}. When contrastive loss is not adopted, although instances of different categories (marked by dots and triangles) are far away from each other, instances of the same category intersect in the embedding space, which leads to poor segmentation of two snuggly instances. Whereas, after adding contrastive loss, even the instances of the same category are separated from each other in the embedding space.

\begin{figure}[!t]
    \begin{minipage}[t]{0.5\linewidth}
        \centering
        \includegraphics[width=\textwidth]{img/c1.png}
        \centerline{ \small (a)w/o contrastive loss}
    \end{minipage}%
    \begin{minipage}[t]{0.5\linewidth}
        \centering
        \includegraphics[width=\textwidth]{img/c2.png}
        \centerline{ \small (b)w contrastive loss}
    \end{minipage}
    \caption{Visualization of the box queries with t-SNE~\cite{van2008visualizing}. The input video has three instances and two of them have the same category~(marked by yellow\&blue dots). It can be seen that with contrastive loss, the  box queries in the same category but belonging to different instances are pushed away.}
    \label{Fig. 6}
    \vspace{-3mm}
\end{figure}

\noindent \textbf{Number of sampling points in ST-Enc.} It is also of interest to assess the effect of using how many temporal cues by adjusting the number of $K_{inter}$.  \cref{Tab7}~shows the results for this. We observe a large performance improvement when $K_{inter}$ changes from 1 to 3. When we further change $K_{inter}$ from 3 to 4, the gain of performance is not obvious. As a result, to achieve the balance between computation and precision, we set the value of $K_{inter}$ to 4 for all experiments.

\begin{table}[h]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{ccc}
\hline
$K_{inter}$ & Params &$\rm AP$   \\ \hline
1           &  50.4   &  46.6 \\
2           &  51.0   &  47.5 \\
3           &   51.6  &  47.9  \\
4           &    52.2 & 48.1 \\ \hline
\end{tabular}
\caption{Ablation on the number of temporal sampling points.}
\label{Tab7}
\vspace{-3mm}
\end{table}

\section{Conclusion}
This paper presents TAFormer which aims to alleviate two problems in video instance segmentation: appearance deformation and tracking error caused by visually similar instances. For the appearance deformation, TAFormer aggregates spatio-temporal features by introducing a spatio-temporal join multi-scale deformable attention module in transformer encoder and a temporal self-attention module in transformer decoder. For the instance tracking error, TAFormer uses an instance-level contrastive loss to increase the discriminability of instance query. TAFormer improves state-of-the-art approaches on YouTube-VIS 2019/2021 benchmarks, which demonstrates the effectiveness of using temporal information in both transformer encoder and decoder. We hope our insights in TAFormer can inspire future transformer-based VIS methods.
{\small
\bibliographystyle{ieee_fullname}

\section{App}

After receiving paper reviews, authors may optionally submit a rebuttal to address the reviewers' comments, which will be limited to a {\bf one page} PDF file.
Please follow the steps and style guidelines outlined below for submitting your author response.

The author rebuttal is optional and, following similar guidelines to previous CVPR conferences, is meant to provide you with an opportunity to rebut factual errors or to supply additional information requested by the reviewers.
It is NOT intended to add new contributions (theorems, algorithms, experiments) that were absent in the original submission and NOT specifically requested by the reviewers.
You may optionally add a figure, graph, or proof to your rebuttal to better illustrate your answer to the reviewers' comments.

Per a passed 2018 PAMI-TC motion, reviewers should refrain from requesting significant additional experiments for the rebuttal or penalize for lack of additional experiments.
Authors should refrain from including new experimental results in the rebuttal, especially when not specifically requested to do so by the reviewers.
Authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.

Just like the original submission, the rebuttal must maintain anonymity and cannot include external links that reveal the author identity or circumvent the length restriction.
The rebuttal must comply with this template (the use of sections is not required, though it is recommended to structure the rebuttal for ease of reading).

\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.
Overlength responses will simply not be reviewed.
This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.

\section{Formatting your Response}

{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}

All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The top margin should begin 1 inch (2.54 cm) from the top edge of the page.
The bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the page.

Please number any displayed equations.
It is important for readers to be able to refer to any particular equation.

Wherever Times is specified, Times Roman may also be used.
Main text should be in 10-point Times, single-spaced.
Section headings should be in 10 or 12 point Times.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Figure and table captions should be 9-point Roman type as in \cref{fig:onecol}.

List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response.
When referenced in the text, enclose the citation number in square brackets, for example~\cite{Alpher05}.
Where appropriate, include the name(s) of editors of referenced books.

\begin{figure}[t]
  \centering
  \fbox{\rule{0pt}{0.5in} \rule{0.9\linewidth}{0pt}}

   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
   \label{fig:onecol}
\end{figure}

To avoid ambiguities, it is best if the numbering for equations, figures, tables, and references in the author response does not overlap with that in the main paper (the reviewer may wonder if you talk about \cref{fig:onecol} in the author response or in the paper).
See \LaTeX\ template for a workaround.

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.
Please ensure that any point you wish to make is resolvable in a printed copy of the response.
Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your response in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it is almost always best to use \verb+\includegraphics+, and to specify the  figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.pdf}
\end{verbatim}
}

{\small
\bibliographystyle{ieee_fullname}

\section{APPENDIX}
\subsection{Extra Training Data}
Following SeqFormer, we also add coco80K for joint training on both YTVIS19 and YTVIS21 with the backbone of ResNet-50. The results are shown as \cref{Tab1} and \cref{Tab2}. For YTVIS19, it can boost the gain of $1.2\%$ $\rm AP$. Moreover, a large performance improvement can be seen in higher overlap $\rm{AP}_{75}$. For YTVIS21, our methods achieves the overall $\rm AP$ of $43.3$.

\begin{table}[h]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{ccccccccc}
\hline
Methods           & $\rm AP$   & $\rm AP_{50}$          & $\rm AP_{75}$          & $\rm AR_{1}$  & $\rm AR_{10}$ \\ \hline
 MaskTrack R-CNN   & 30.3 & 51.1 & 32.6 & 31.0 & 35.5 \\
 STEm-Seg          & 30.6 & 50.7 & 33.5 & 37.6 & 37.1 \\
SipMask         & 33.7 & 54.1 & 35.8 & 35.4 & 40.1 \\
 CompFeat           & 35.3 & 56.0 & 38.6 & 33.1 & 40.3 \\
 VisTR             & 36.2 & 59.8 & 36.9 & 37.2 & 42.4 \\
MaskProp         & 40.0 & -    & 42.9 & -    & -    \\
 CrossVIS          & 36.3 & 56.8 & 38.9 & 35.6 & 40.7 \\
 Propose-Reduce     & 40.4 & 63.0 & 43.8 & 41.1 & 49.7 \\
PCAN                & 36.1 &54.9& 39.4 & 36.3 & 41.6 \\
IFC              & 42.8 & 65.8 & 46.8 & 43.8 & 51.2 \\
Mask2Former          & 46.4 & 68.0 & 50.0 & -    & -    \\
TeViT $\dagger$       & 46.6 & 71.3 & 51.6 & 44.9    & 54.3 \\
 SeqFormer          & 45.1 & 66.9 & 50.5 & 45.6 & 54.6 \\
  SeqFormer $\ddagger$           & 47.4 & 69.8 & 51.8 & 45.5 & 54.8 \\
\textbf{TAFormer}  & 48.1 & 71.6 & 52.5 & \textbf{45.9} & 56.6 \\
\textbf{TAFormer $\ddagger$ }  & \textbf{49.3} & \textbf{72.1} & \textbf{55.4} & 45.7 & \textbf{57.8} \\ \hline
\end{tabular}
\caption{Quantitative results of VIS methods on the YTVIS-19 validation set. $\ddagger$ denotes adopting 80K train2017 images containing YTVIS-19 categories for joint training. $\dagger$ represents using MsgShift as backbone.}
\label{Tab1}
\vspace{-3mm}
\end{table}

\begin{table}[h]\small
\centering
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{ccccccccc}
\hline
Methods           & $\rm AP$   & $\rm AP_{50}$          & $\rm AP_{75}$          & $\rm AR_{1}$  & $\rm AR_{10}$ \\ \hline
VisTR        & 31.8 & 51.7          & 34.5          & 29.7 & 36.9 \\
IFC                & 36.6 & 57.9          & 39.3          & -    & -    \\
Mask2Former               & 40.6 & 60.9          & 41.8          & -    & -    \\
SeqFormer$\ddagger$           & 40.5 & 62.5          & 43.6          & 36.2 & 48.0 \\
TeViT $\dagger$             & 37.9 & 61.2          & 42.1          & 35.1 & 44.6 \\
\textbf{TAFormer}   & 42.1 & 63.8 & 44.8 & 38.6 & 49.2 \\
\textbf{TAFormer$\ddagger$}    & \textbf{43.3} & \textbf{65.7} & \textbf{46.5} & \textbf{38.6} & \textbf{51.0} \\ \hline
\end{tabular}
\caption{Quantitative results of VIS methods on the YTVIS-21 validation set. $\ddagger$ denotes adopting 80K train2017 images containing YTVIS-21 categories for joint training. $\dagger$ represents using MsgShift as backbone.}
\label{Tab2}
\vspace{-3mm}
\end{table}

\end{document}


\section{Introduction}
The study on correlations unachievable within the classical paradigm has been an integral constituent of theoretical \cite{P.Zoller} and experimental \cite{P.Zoller} probes in quantum information processing. Entanglement \cite{R1.Horodecki}, is the supreme contributor to such studies. However, entanglement although a form of quantum nonlocality, is not the strongest among such correlations. Bell nonlocality \cite{J.Bell} constitutes a stronger correlation significant both from a foundational \cite{V.Scarani} and pragmatic perspective \cite{V.Scarani}. The recent spurt in device-independent security protocols \cite{A.Acin} has provided for a driving force to studies on Bell nonlocality.

Quantum steering \cite{R.Uola} on the other hand lies between these two form of correlations and is usually asymmetric in nature. Schrödinger was the first to notice this phenomenon \cite{E.Schrodinger1,E.Schrodinger2} and considered it to be intriguing that the first party can by choosing a measurement steer the state on the other side into an eigenstate of position or momentum. However, apart from being a thought-provoking fundamental question this phenomenon didn't receive much attention till the work of Wiseman, Jones and Doherty \cite{H.Wiseman}. The authors formulated steering in the language of quantum information processing and gave an operational interpretation to it \cite{H.Wiseman}. In the modern perspective steering denotes the impossibility to describe the conditional states at one party by a local hidden state (LHS) model. Much like Bell nonlocality, quantum steering also has gained prominence recently due to its significance in semi-device-independent communication scenarios \cite{M.Pawlowski}. An interesting observation here is that although entanglement is necessary for steering and Bell nonlocality, it is not sufficient.

In any theory of information processing, entropies \cite{A.Wehrl} play a key role as they prove to be efficient quantifiers of information content. The Shannon entropy \cite{C.E.Shannon} in classical information theory and its quantum counterpart, the von Neumann entropy \cite{J.Neumann} are the paradigmatic examples. However, these are not the only versions of entropy to be used. Rényi $\alpha$- entropies \cite{AR}, Tsallis $\alpha$- entropies \cite{TS} and their quantum analogues \cite{M.Muller} are  used in different scenarios. In quantum information theory, we broadly distinguish tasks as falling into two regimes:(i) asymptotic regime, where many identically and independently distributed quantum systems appear and (ii) single shot regime, where scenarios involve only a single quantum system. While in (i) von Neumann entropies play a significant role, in (ii) it is the Rényi $\alpha$- entropy that takes center stage. Quantum entropies have not only played their part in information processing, but also have been used in the establishment of entropic Bell inequalities \cite{N.Cerf}, entropic steering inequalities \cite{J.Schneeloch,AC} and entropic uncertainty relations \cite{S.Wehner} which in turn is at the heart of quantum cryptography \cite{N.Gisin}.

Quantum entropy, being a measure of uncertainty of quantum systems, encompasses both classical and quantum uncertainty. In this perspective, consider a bipartite system shared between Alice and Bob and ask the question "\textit{What amount of information Bob has to learn in addition to the knowledge he already possesses to know the whole system ?}". The answer to this question is provided by conditional entropies. In the classical realm, conditional entropies are always nonnegative in consonance to our intuition. However, in the quantum situation conditional entropies can be negative \cite{N.J.Cerf} marking a radical departure from classical notions. In fact certain entangled states possess negative conditional entropy. An operational interpretation to quantum conditional von Neumann entropy was given by the authors in \cite{M.Horodecki} akin to the classical Slepian-Wolf theorem \cite{SLP}. Recently the negativity of conditional von Neumann entropy was identified as a resource \cite{M1.Vempati,M2.Vempati,MW}. In another recent work \cite{Gourinev}, it was noted that in any axiomatic characterization of conditional entropy, this negativity is inevitable.

The present submission investigates the relation between these two extremely significant manifestations of quantum states, namely steering and conditional entropies. Probes were made to see this in case of Bell-CHSH inequalities \cite{J.F.Clauser} and conditional von Neumann entropies in two-qubit systems \cite{N.Friis}. The relation of Bell's inequalities with quantum $ \alpha $ entropies were investigated in \cite{R.Horodecki}. As noted before, entropies were also used to form entropic Bell's inequalities \cite{N.Cerf} and entropic steering inequalities \cite{J.Schneeloch}. However, to the best of our knowledge, these studies have not been done in the framework of steering and LHS models. Our work tries to bridge this gap.

We start with two-qubit systems and probe the relation of conditional entropies with the violation of a three-settings steering inequality, quite prominently known as the Cavalcanti, Jones, Wiseman , Reid (CJWR) inequality \cite{E.Cavalcanti}. We show that for the Weyl states(states with maximally mixed marginals), the negativity of conditional Rényi 2-entropy is necessary and sufficient for the violation of the CJWR inequality. Upper bounds to the conditional entropy of generic two-qubit quantum states are also provided which guarantees the violation of the inequality. We then move from a particular steering inequality to LHS models. We use a sufficient criteria for unsteerability \cite{J.Bowles} and find that whenever Weyl states are unsteerable due to the criteria , they possess a non-negative conditional Rényi 2-entropy. However, some non-Weyl states present a different picture. We extend our study to two-qudit systems and observe the behavior of LHS models of the isotropic and Werner states \cite{R.Werner} vis a vis conditional Rényi 2-entropies. We observe, whenever the states admit a LHS model, they have non-negative conditional Rényi 2-entropy. We then introduce white noise in the Werner states and note that the LHS model of the transformed state exhibits the same characteristic.

The paper is arranged as follows: In the next section we revisit some key notions important for our work and fix the notations. In section(\ref{CJWRtwo}), we study the relation between violation of the CJWR inequality and conditional entropy of two-qubit systems. In section(\ref{LHSqubit}), we proceed to investigate the LHS models in two-qubit systems. We then explore the relation with LHS models in two-qudits in section(\ref{LHSqudit}). We then end with the concluding remarks.

\section{Notation and Preliminaries}
In this section we fix the notations and recapitulate some notions important for the study. \\
Our work is done in finite dimensional Hilbert spaces. Quantum states are described by density operators $ \rho $, i.e., positive semi-definite ($ \rho \ge 0 $), Hermitian (which also follows from positivity) with unit trace. $ \rho \in \mathfrak{B}(\mathbb{H}_{A} \otimes \mathbb{H}_{B}) $, where $ \mathfrak{B}(\mathbb{H}_{A} \otimes \mathbb{H}_{B}) $ (also known as the Hilbert-Schmidt space) represents the bounded linear operators acting on $ \mathbb{H}_{A} \otimes \mathbb{H}_{B} $. $ \mathbb{H}_{X},X=A,B $ denote the underlying Hilbert space. $ S(.) , S_\alpha(.), S^T_\alpha(.) $ denote respectively the von Neumann, Rényi $ \alpha- $ entropy and Tsallis $ \alpha- $ entropy of the quantum state being discussed.
\subsection{Bloch-Fano Decomposition of density matrices}

For bipartite quantum systems, the density matrices can be represented as \cite{N.Friis}
\begin{widetext}
\begin{equation} \label{blochqudit}
	\rho_{d_A \otimes d_B} = \frac{1}{d_A d_B} [\mathbb{I}_A \otimes \mathbb{I}_B + \sum_{m=1}^{d_A^2 - 1} a_m g_m^A \otimes \mathbb{I}_B + \sum_{n=1}^{d_B^2 - 1} b_n  \mathbb{I}_A \otimes g_n^B + \sum_{m=1}^{d_A^2 - 1} \sum_{n=1}^{d_B^2 - 1} t_{mn} g_m^A \otimes g_n^B],
\end{equation}
\end{widetext}
 where $ \text{dim}~ \mathbb{H}_{A} = d_A  $ and $ \text{dim}~ \mathbb{H}_{B} = d_B  $. The hermitian operators $ g_m^i $ for $ i=A,B $ are generalizations of the Pauli matrices, i.e., they are orthogonal $ Tr[g_m^i g_n^i] = 2 \delta_{mn} $ and traceless, $ Tr[g_m^i] =0  $ and for single qubit systems they coincide with the Pauli matrices. The coefficients $ a_m, b_n \in \mathbb{R} $ are the components of the generalized Bloch vectors $ \vec{a} , \vec{b} $ of the subsystems $ A,B $ respectively. The real coefficients $ t_{mn} $ are the components of the correlation tensor. For two-qubit systems the density matrices can be represented as,
 \begin{equation}\label{blochqubit}
 	\rho_{2 \otimes 2}= \frac{1}{4} [\mathbb{I}_{2}\times\mathbb{I}_2+\vec{\mathfrak{a}}.\vec{\sigma}\otimes \mathbb{I}_2+\mathbb{I}_2\otimes \vec{\mathfrak{b}}.\vec{\sigma} + \sum_{m=1}^{3} \sum_{n=1}^{3} t_{mn} \sigma_m^A \otimes \sigma_n^B ].
 \end{equation}

 An interesting class of states is the \textit{locally maximally mixed states or Weyl states} which in the two-qudit systems (up to local unitaries) is given by,

 \begin{equation}
 	\rho^{weyl}_{d} = \frac{1}{d^2} [\mathbb{I}_A \otimes \mathbb{I}_B + \sum_{n=1}^{d^2 - 1} w_{n} g_n^A \otimes g_n^B].
 \end{equation}
 The reduced marginals of such states are maximally mixed, i.e., $ Tr_A[\rho^{weyl}_{d}] = \mathbb{I}_A/d  $ and $ Tr_B[\rho^{weyl}_{d}] = \mathbb{I}_B/d.$

\subsection{Quantum Entropies}
The von Neumann entropy of a quantum state $ \rho $ is defined as,
\begin{equation}
	S(\rho) = -Tr [ \rho \log \rho].
\end{equation}
where the logarithms are taken to the base 2.
The von Neumann entropy has a special association with the eigenvalues of the density matrix, i.e, it is a function of the eigenvalues. The corresponding conditional von Neumann entropy(CVNE) is given by $ S(A|B)=S(\rho)-S(\rho_B) $.

The Rényi $\alpha$-entropy is given by
\begin{align}
	S_{\alpha}(\rho)= \frac{1}{1-\alpha} \log[Tr(\rho^{\alpha})], \alpha >0, \alpha \neq 1.
\end{align}
The von Neumann entropy is the limiting case of the Rényi entropy as
$\alpha\rightarrow 1$. The corresponding conditional Rényi $\alpha$- entropy (CRAE) is given by $ S_\alpha(A|B)=S_\alpha(\rho)-S_\alpha(\rho_B) $.

Tsallis $\alpha$-entropy is given by
\begin{align}
	S^T_{\alpha}(\rho)= \frac{1}{1-\alpha} [Tr(\rho^{\alpha})-1], \alpha >0, \alpha \neq 1.
\end{align}
The corresponding conditional Tsallis $\alpha$-entropy (CTAE) is given by \cite{Vollbrecht},
\begin{align*}
	S^T_{\alpha}(A|B)=\frac{Tr(\rho_B^\alpha)-Tr(\rho^\alpha)}{(\alpha-1)Tr(\rho_B^\alpha)}.
\end{align*}

\subsection{Quantum Nonlocality}
\textit{Bell nonlocality-} Consider two parties Alice and Bob to be spatially separated, sharing some quantum state $\rho_{AB}$. They perform measurements denoted by $ A_x , B_y $ respectively on their particles labelled by $ x $ and $ y $. The corresponding outcomes at Alice's and Bob's side are labelled as $ a $ and $ b$. Now, for some hidden variable $\lambda$ \& probability distribution $q(\lambda)$, if all joint probabilities $p(a,b|x,y)$ can be expressed as,
\begin{align} \label{LHV}
	p(a,b|x,y) = \int_{\lambda} d\lambda q(\lambda)p(a|x,\lambda)p(b|y,\lambda)
\end{align}
then we say that the state is local or has a local hidden variable (LHV) model.
Here, $p(a|x,\lambda)$ \& $p(b|y,\lambda)$ are said to be the local response functions of Alice and Bob respectively and $ q(\lambda) $ is the probability distribution of the hidden variable $ \lambda $. If equation (\ref{LHV}) does not hold, we say that it is nonlocal and hence will violate a suitably chosen Bell's inequality \cite{J.Bell}. \\
\textit{Bell-CHSH Criterion}- The Bell-CHSH criterion \cite{J.F.Clauser} gives a Bell's inequality when there are two parties performing  measurements in a two input-two output scenario. A necessary and sufficient condition for its violation was provided in \cite{HoroBell}. Let $\rho$ be a density operator and its correlation tensor be given by $T = [t_{mn}]$. Suppose that the largest two eigenvalues of matrix $T^{\dagger}T$ are $\lambda_{1}$ \& $\lambda_{2}$. Then the state $\rho$ violates the CHSH inequality iff their sum is greater than one,
i.e, $\lambda_{1}$+$\lambda_{2}$ $> 1$ \cite{HoroBell}. One should note here that a state may satisfy the Bell-CHSH criterion, however it may still be nonlocal (through the violation of any other Bell's inequality).\\
\textit{Quantum Steering-} Quantum steering is an asymmetric form of nonlocality, counted on the ability of Alice to steer the state on Bob's side with a choice of measurement on her side. Although the concept of steering is not new, dating back to the contributions of Schrödinger \cite{E.Schrodinger1,E.Schrodinger2}, the operational formulation was provided only in \cite{H.Wiseman}. The notion of steering can be conveniently expressed in the language of steering assemblages. Suppose that there are two parties Alice and Bob spatially separated, sharing quantum state $\rho_{AB}$. Now, Alice performs some type of measurement, say $x$ on her part of the state and the outcome obtained is $a$. Bob remains with unnormalized conditional state $\rho_{a|x}$ (steering assemblage) for each measurement of Alice. Now if Bob's conditional state $\rho_{a|x}$ can be expressed as,
\begin{align}
	\rho_{a|x}= \int_{\lambda} d\lambda p(\lambda)p(a|x,\lambda)\sigma_{\lambda},
\end{align}
where $\sigma_{\lambda}$ is the hidden state (on Bob's side) with probability $p(\lambda)$ and $p(a|x,\lambda)$ is the local response function of Alice,
then we say that the state $\rho_{AB}$ is unsteerable or has local hidden state (LHS) model. Otherwise, the state $\rho_{AB}$ is steerable from Alice to Bob. Steerability is an asymmetric trait. A state which is steerable from Alice to Bob might not be steerable from Bob to Alice.\\
\textit{The CJWR inequality for steering-} Cavalcanti, Jones, Wiseman and Reid(CJWR) derived a
series of correlators based inequalities \cite{E.Cavalcanti} for verifying steerability of $\rho:$
\begin{equation}\label{mon2}
	F_{n}(\rho,\nu) = \frac{1}{\sqrt{n}}|\sum_{l=1}^{n} \langle A_{l} \otimes B_{l} \rangle | \leq 1,
\end{equation}
where,\\
$A_{l} $$=$$ \hat{u}_{l}$$\cdot$$ \overrightarrow{\sigma},$ $B_{l}$$=$$\hat{v}_{l}$$\cdot$$ \overrightarrow{\sigma}$ with\\
$\hat{u}_{l} $$\in$$ \mathbb{R}^{3} \textmd{are unit vectors whereas}\,\hat{v}_{l} $$\in$$ \mathbb{R}^{3}\,$denote orthonormal vectors$.\,$$\nu $$=$$\{\hat{u}_{1},\hat{u}_{2},....\hat{u}_{n}, \hat{v}_{1},\hat{v}_{2},...,\hat{v}_{n} \}$ stands for the collection of measurement directions,  $\langle A_{l} \otimes B_{l} \rangle = \textmd{Tr}(\rho (A_{l} \otimes B_{l}))$. Violation of Eq.(\ref{mon2}) ensures both way steerability of $\rho$ in the sense that it is steerable from $A$ to $B$ and vice versa. In particular, for $n$$=$$3,$ CJWR inequality(Eq.(\ref{mon2})) for three settings takes the form:
\begin{equation}\label{mon2i}
	F_{3}(\rho,\nu) = \frac{1}{\sqrt{3}}|\sum_{l=1}^{3} \langle A_{l} \otimes B_{l} \rangle | \leq 1,
\end{equation}

In \cite{Costa}, analytical expression for the violation of the linear inequality (equation(\ref{mon2i})) was given. Any two-qubit state shared between $A$ and $B$ is both-way $F_{3}$ steerable if:
\begin{align}
	\sqrt{Tr(T^{\dagger}T)}> 1,
\end{align}
$ T $ being the correlation tensor.

 One must note here that entanglement is only a necessary criterion for Bell nonlocality and steering.
\section{Conditional Entropies and Steerability for two-qubit states} \label{CJWRtwo}
In this section we investigate the relation between conditional entropies and $ F_3 $ steerability for two-qubit Weyl states. We start our discussion with the Werner state, which is a special kind of Weyl state.
\subsection{Werner States}

\textit{Relation with conditional von Neumann entropy-} Consider the two-qubit Werner state $\rho^{wer}_{2}(p)=$ $\frac{1}{4}[\mathbb{I}_2 \otimes \mathbb{I}_2-\sum_{i=1}^{3}p(\sigma_{i} \otimes \sigma_{i})]$. Conditional von Neumann entropy is given by,
\begin{widetext}
\begin{align}
	S(A|B)_{\rho^{wer}_{2}(p)}=-3\left( \frac{1-p}{4}\right) \log\left(\frac{1-p}{4}\right)-\left(\frac{1+3p}{4}\right)\log\left(\frac{1+3p}{4}\right)-1.
\end{align}
\end{widetext}
CVNE is negative for $ p >  0.747614 $. The Werner state is $ F_3 $ steerable for $ p > 0.57735 $. Thus, if the Werner state has negative CVNE then it is 3-steerable.\\

\textit{Relation with other $ \alpha- $ conditional entropies-}
We see here the relation between $ F_3 $ steerability and CRAE for the two-qubit Werner state. The table below notes negativity of CRAE for $ \alpha = 2,3,4,5 $,

\begin{center}
	\begin{tabular}{|c|c|c|c|c|}
	\hline
	$ \alpha$ &  2 &  3 &  4 &  5 \\
	\hline
	Negative CRAE & $\frac{1}{\sqrt{3}}< p$  & $\frac{1}{2}<p$  &  $0.45786<p$ & $0.432041<p$  \\
	\hline
\end{tabular}\\
\end{center}

Note that the Werner state is $ F_3 $ steerable iff the conditional Rényi 2-entropy is negative, a fact that we prove to be true for all two-qubit Weyl states in the next subsection.
Since  $ S^T_{\alpha}(A|B) \ge 0 \Leftrightarrow S_{\alpha}(A|B) \ge 0  $ \cite{Vollbrecht}, we arrive at the same conclusion for the conditional Tsallis 2-entropy.\\

\subsection{Weyl States}

We consider the Weyl states in two-qubits given by,
\begin{equation}
\rho^{weyl}_{2}= \frac{1}{4}[\mathbb{I}_2 \otimes \mathbb{I}_2  +\sum_{i=1}^{3} t_{i}(\sigma_{i} \otimes \sigma_{i})].
\end{equation}
The conditional Rényi 2-entropy of $ \rho^{weyl}_{2}  $ is given by $S_2 (A|B)_{\rho^{weyl}_{2}}=1-\log(1+t_{1}^2 + t_{2}^2 + t_{3}^2) $ and $ F_3 (\rho^{weyl}_{2})= \sqrt{t_{1}^2+t_{2}^2+t_{3}^2} $. A simple algebraic calculation shows that $ S_2 (A|B)_{\rho^{weyl}_{2}} < 0 \Leftrightarrow F_3 (\rho^{weyl}_{2}) > 1 $.

We thus have the following theorem,

\begin{theorem}
	The two-qubit Weyl state is $ F_3 $ steerable iff its conditional Rényi 2-entropy is negative.
\end{theorem}
The same conclusion follows for the conditional Tsallis 2-entropy due to its equivalence with conditional Rényi 2-entropy.\\
\subsection{Two-Qubit general state}
We now consider the general two-qubit state $ \rho^{g}_{2} $, whose decomposition is given in Eq.(\ref{blochqubit}). Its conditional Rényi 2-entropy (CR2E) is given by $S_2 (A|B)_{\rho^{g}_{2}}= \log\left[\frac{2+2||\vec{b}||^{2}}{1+||\vec{a}||^{2}+||\vec{b}||^{2}+||\vec{T}||^{2}}\right]$where $|| . ||$ denote the Euclidean norm, $||T||^{2}=Tr(T^{\dagger} T)$ and it is $F_{3}$ steerable if $||T||=\sqrt{Tr(T^{\dagger}T)}>1$. An algebraic calculation shows that the state is $F_{3}$ steerable $\iff$ $S_2 (A|B)_{\rho^{g}_{2}}< \log\left[\frac{1+||\vec{b}||^{2}}{1+\frac{||\vec{a}||^{2}+||\vec{b}||^{2}}{2}}\right]$.\\
We thus have the following theorem,

\begin{theorem}
	The two-qubit general state is $ F_3 $ steerable iff its CR2E$< \log\left[\frac{1+||\vec{b}||^{2}}{1+\frac{||\vec{a}||^{2}+||\vec{b}||^{2}}{2}}\right]$.
\end{theorem}

\subsection{Absolute non-violation of the CJWR inequality by the Weyl states for a certain purity threshold }
An interesting class of states, which we denote by $ AF_3 $, is the collection of states which preserve $ F_3 $ unsteerability even under global unitary operations. In \cite{S.Bhattacharya}, the authors proved that a quantum state will remain $ F_3 $ unsteerable under arbitrary global unitary operations iff its purity $ \le \frac{1}{2} $. Therefore we have,
\begin{align}
	Tr[(\rho^{weyl}_{2})^2] \le \frac{1}{2} \nonumber \\
	\Leftrightarrow  1+t_{1}^2 + t_{2}^2 + t_{3}^2 \le 2 \nonumber \\
	\Leftrightarrow S_2 (A|B)_{\rho^{weyl}_{2}} \ge 0.
\end{align}

In fact the preceding shows that whenever the Weyl states are $ F_3 $ unsteerable, the unsteerability is robust even against global unitary operations.
 However, this observation doesn't hold true for a generic two qubit state, for which we have the following result,

The purity for a general two-qubit state is given by,
 $Tr[(\rho_{2}^{g})^{2}]=\frac{1}{4}\left[1+||\vec{a}||^{2}+||\vec{b}||^{2}+||\vec{T}||^{2}\right]$ and CR2E is given by $S_{2}(A|B)_{\rho_{2}^{g}}= \log\left[\frac{2+2||\vec{b}||^{2}}{1+||\vec{a}||^{2}+||\vec{b}||^{2}+||\vec{T}||^{2}}\right]$\\

 Now,
$Tr[(\rho_{2}^{g})^{2} \leq \frac{1}{2}$ $\implies$ $S_{2}(A|B)_{\rho_{2}^{g}}\geq 0$.
We thus have the following theorem
\begin{theorem}
    If $\rho_{2}^{g} \in AF_{3}$ then CR2E is non-negative.
\end{theorem}

\subsection{A non-Weyl state}
We now consider the state $ \rho_2^\theta = \beta | \psi_\theta \rangle \langle \psi_\theta | + (1-\beta) \mathbb{I}_2/2 \otimes \mathbb{I}_2/2$, where $ | \psi_\theta \rangle = \cos \theta |00 \rangle + \sin \theta | 11 \rangle  $ and $ \beta \in [0,1] $. The state is $ F_3 $ steerable and its conditional Rényi 2-entropy is negative when $ \cos4\theta < \frac{2\beta^{2}-1}{\beta^{2}}$ for $\theta \in \left(0,\frac{\pi}{2}\right)\cup\left(\frac{\pi}{2},\pi\right)\cup \left(\pi,\frac{3\pi}{2}\right) \cup \left(\frac{3\pi}{2},2\pi\right)$ and $\beta \in \left(\frac{1}{\sqrt{2-\cos4\theta}},1\right]$. Thus, the state violates the CJWR inequality if and only if it possesses a negative conditional Rényi 2-entropy, for the above restrictions on $ \theta $ and $ \beta $.

\section{Conditional entropies and LHS Models in two-qubits}\label{LHSqubit}

The CJWR inequality being a particular steering inequality, satisfaction of it doesn't imply that the state is unsteerable. There may be other steering inequalities that the state violates. Therefore, it is worthwhile to see the relation between unsteerability (in terms of admitting LHS models) and conditional entropies.

In \cite{J.Bowles}, a sufficient condition for the unsteerability of states was proposed, which is stated below. A two-qubit state of the form $ \rho_0 = \frac{1}{4} [\mathbb{I}_2 \otimes \mathbb{I}_2 + \vec{\mathfrak{a}}.\vec{\sigma}\otimes \mathbb{I}_2 + \sum\limits_{i=x,y,z} T_i \sigma_i \otimes \sigma_i ] $ is unsteerable if $ \max\limits_{\hat{x}} [(\vec{\mathfrak{a}}.\hat{x})^2 + 2 \| T \hat{x} \| ] \le 1 $, where $ \hat{x} $ is a unit normal vector and $ \| . \|  $ corresponds to Euclidean norm \cite{J.Bowles}. We now present below, our findings for some categories of states admitting LHS model.

\subsection{Weyl states admitting local hidden state models}
 For the Weyl states in two-qubits, the sufficient condition for unsteerability reads $ \max\limits_{\hat{x}} [ 2 \| T \hat{x} \| ] \le 1 $. Without loss of generality we take the singular values of $ T $ as $ t_1 \ge t_2 \ge t_3 $. The condition thus gives, $ t_1 \le \frac{1}{2} $. Since the conditional Rényi 2-entropy for the Weyl state is given by $  S_2 (A|B)_{\rho^{weyl}_{2}}=1-\log(1+t_{1}^2 + t_{2}^2 + t_{3}^2) $ , we have $ t_1 \le \frac{1}{2} $ implies $ S_2 (A|B)_{\rho^{weyl}_{2}} \ge 0 $. Thus, whenever Weyl states obey the sufficient criteria for unsteerability, they have a non-negative conditional Rényi 2-entropy.

\subsection{LHS models for non-Weyl states}
Consider a two-qubit non-Weyl state \cite{J.Bowles},
$\rho_2^{nl}=p|\psi_{x}\rangle\langle\psi_{x}| +(1-p)[\rho_{x}^{A}\bigotimes \frac{\mathbb{I}}{2}]$, where $|\psi_{x}\rangle = \cos x|00\rangle +\sin x|11\rangle $; $0<x\leq\frac{\pi}{4}$, $0\leq p\leq 1$. Sufficient condition for unsteerability of the state is given by,
\begin{equation} \label{unnwl}
	[\cos(2x)]^2 \geq \frac{2p-1}{(2-p)p^3}.
\end{equation}
Here, we observe that $ S_2 (A|B)_{\rho^{nl}_{2}} < 0 $.

However, on mixing white noise with the state we arrive at a different observation, which we note below.

A state $\rho_2^{'}$ admits LHS model if there exists a unit trace operator $O_{AB}$ such that $O_{AB}$ admits LHS model and $\rho_2^{'}= r O_{AB} + (1-r) \frac{I}{d}\otimes O_{B}$ \cite{D.Cavalcanti}. Consider now $ O_{AB} = \rho_{2}^{nl} $.

$S_{2}(A|B)_{\rho_2^{'}}$ is non-negative for
\begin{align}
	\cos4x\geq \frac{r^{2}+4r^{2}p^{2}-p^{2}-2}{2p^{2}-r^{2}-p^{2}(1-2r^{2})}.
\end{align}

and $ \rho_2^{'} $ is unsteerable for

\begin{align}
	\cos4x\geq \frac{2(2p-1)-(2-p)p^3}{(2-p)p^3}.
\end{align}

Let $ h(r,p)= \frac{r^{2}+4r^{2}p^{2}-p^{2}-2}{2p^{2}-r^{2}-p^{2}(1-2r^{2})}-\frac{2(2p-1)-(2-p)p^3}{(2-p)p^3}$ for $x\in(0,\frac{\pi}{4}]$,  $r\in[0,1]$, $p\in[0,1]$.\\
Then, $h(r,p)$ is non-negative for $r\in (0,1)$, $p \in \left(0,\sqrt{\frac{r^2}{1+2r^2}}\right);$
$r=0, p\in \left(0,2-\sqrt{3}\right);$
$r=1, p\in \left(0,\frac{1}{\sqrt{3}}\right)\cup \left(\frac{1}{\sqrt{3}},1\right)$  and $\rho$ is entangled for $p>\frac{1}{3}$.

Thus, implying that whenever the state $\rho^{'}$ admits an LHS model(due to the above criteria) its conditional Rényi 2-entropy is non-negative, for the above restrictions on $ r $ and $ p $.

\section{LHS models in $ d \otimes d $ systems and conditional entropies}\label{LHSqudit}

\subsection{Isotropic states}
For the isotropic state in $ d \otimes d $ dimensions, given by $ \rho_d^{iso}= \eta |\psi_d \rangle \langle \psi_d |+ (1-\eta) \frac{\mathbb{I}}{d} \otimes \frac{\mathbb{I}}{d} $, where $ |\psi_d \rangle = \frac{1}{\sqrt{d}} \sum\limits_{i=0}^{d-1} | ii \rangle $, the conditional Rényi 2-entropy is non-negative for $ \eta \in (\frac{1}{d+1},\frac{1}{\sqrt{d+1}}] $.

The isotropic state admits an LHS model for projective measurements when $\eta \le \frac{H_{d}-1}{d-1}$ \cite{H.Wiseman} where $H_{d}=1+1/2 + 1/3+...+1/d$. After solving above inequality we obtained $ \eta \in \left(\frac{1}{d+1},\frac{\ln(1+2d)-1}{d-1}\right] $ where $ln$ is logarithm taken in base $e$.

In the above intervals, the state is also entangled. Now if we consider the function $ f(x)= \frac{1}{\sqrt{x+1}}-\frac{\ln(1+2x)-1}{x-1} $, then $ f $ is an increasing function for $ x \ge 3 $. This indicates that for local dimensions which are at least 3, whenever the isotropic state admits an LHS model, its conditional Rényi 2-entropy is non-negative. However, when the local dimension is 2 then non-negativity of conditional Rényi 2-entropy implies that the state has an LHS model.

\subsection{Werner States}
If $d_{A}=d_{B}=d$ then the state in eq.(\ref{blochqudit}) denoted by $ \rho_{d}^{g} $ has its CR2E given by,
\begin{align}
	S_{2}(A|B)_{\rho_{d}^{g}}=\log\left[\frac{d+2||\vec{b}||^{2}}{1+\frac{2||\vec{a}||^{2}+2||\vec{b}||^{2}}{d}+\frac{4||T||^{2}}{d^2}}\right].
\end{align}
We have $S_{2}(A|B)_{\rho_{d}^{g}}$ is non-negative iff $||T||\leq \frac{1}{2}\left[\sqrt{d^{2}(d-1)-2d||\vec{a}||^{2}+2d(d-1)||\vec{b}||^{2}}\right]$.
Therefore, for the Werner state $\rho_{d}^{wer}=\frac{d-1+\eta}{d-1}\frac{I}{d^2}-\frac{\eta}{d-1}\frac{V}{d}$ where $V$ is flip operator and given by $V|ij\rangle = |ji\rangle $, CR2E is given by
\begin{align}
	S_{2}(A|B)_{\rho_{d}^{wer}}=\log\left(\frac{d^3}{d^{2}+4||T||^2}\right).
\end{align}
Here, $||T||^{2}=\frac{1}{4}\left[\frac{\eta^{2} d^{2}(d+1)}{d-1}\right]$.\\

CR2E is non-negative for the Werner state when,
\begin{align}
	||T||\leq \frac{d\sqrt{d-1}}{2}.
\end{align}
and it admits LHS model, when $\eta \le 1-\frac{1}{d}$ \cite{H.Wiseman}. After solving above inequality we obtained
\begin{align}
	||T||\leq \frac{\sqrt{d^{2}-1}}{2}.
\end{align}
Define a function $g(d)=\frac{d\sqrt{d-1}}{2}-\frac{\sqrt{d^{2}-1}}{2}$ for $d\geq2$.
Then $g(d)\geq 0$ for $d\geq 2$.
Thus, if the Werner state has LHS model then its CR2E is non-negative
$ \forall d \geq 2 $.

Consider now, $\rho_d^{'}= r O_{AB} + (1-r) \frac{\mathbb{I}}{d}\otimes O_{B}$, with $ O_{AB} = \rho_{d}^{wer} $.

So,  $\rho_d^{'}$ admits LHS model for
\begin{align}
	\eta \le 1-\frac{1}{d},
\end{align}
and
CR2E of $\rho_d^{'}$ is non-negative for
\begin{align}
	\eta \leq \frac{d-1}{r\sqrt{d+1}}.
\end{align}
Let $F(r,d)=\frac{d-1}{r\sqrt{d+1}}-(1-\frac{1}{d})$; $0\leq r\leq 1$; $d\geq 2$.\\
Then $F(r,d)$ is non-negative for $0 < r\leq 1$; $d\geq 2$. Thus, if the state $\rho_d^{'}$ has LHS model then its CR2E is non-negative.

\section{Conclusions}
The work here studies the relation between conditional entropies of quantum systems and their steerability. We observe that in the two-qubit scenario, states with maximally mixed marginals violate the CJWR steering inequality (a particular three-settings steering inequality) iff they have negative conditional Rényi 2-entropy. For the general two-qubit state an upper bound to it's conditional Rényi 2-entropy is given which guarantees violation of the CJWR inequality. Using a pertinent sufficiency criteria, we show that whenever Weyl states in two-qubits are unsteerable (irrespective of any particular inequality), they have a non-negative conditional Rényi 2-entropy. We extend our study to include LHS models in two-qudit systems, particularly for the isotropic and Werner states and find that whenever they admit LHS models, they possess a non-negative conditional Rényi 2-entropy. A noisy variant of the two-qudit Werner state exhibits the same trait.

The relation between quantum conditional entropies and Bell-type inequalities is intricate and thus calls for further studies. Especially, one significant future direction would be to explore the relation for multipartite systems.
\section*{Acknowledgement}

NG acknowledges support from the project grant received from SERB under the MATRICS scheme, vide file number MTR/2022/000101.