\subsubsection*{Author Contributions}

\section*{Acknowledgments}
We would like to thank Niki Parmar for initial discussions. Special thanks to Gabriel Bender and Thang Luong for reviewing the paper and providing constructive feedback. We appreciate the efforts of Kevin Murphy and David Fleet for advising the project and providing feedback throughout. We are grateful to Evan Rapoport, Douglas Eck and Zoubin Ghahramani for supporting this work in a variety of ways. Tim Salimans and Chitwan Saharia helped us with brainstorming and coming up with shared benchmarks. Jason Baldridge was instrumental for bouncing ideas. Alex Rizkowsky was very helpful in keeping things organized, while Erica Moreira and Victor Gomes ensured smooth resourcing for the project. Sarah Laszlo and Kathy Meier-Hellstern have greatly helped us incorporate important responsible AI practices into this project, which we are immensely grateful for. Finally, Blake Hechtman and Anselm Levskaya were generous in helping us debug a number of JAX issues.

\section{Related Works}

This paper is closely related to auto-regressive methods for text conditioned image and video generation. DALL-E~\cite{DALLE} \textit{translates} text tokens to discrete image embeddings learnt using a VQVAE~\cite{vqvae}. Parti~\cite{PARTI} has a similar architecture but can generate higher quality images by predicting tokens from a ViT-VQGAN~\cite{vit_vqgan} using a 21B parameters transformer. Similar architectures have been used for generating videos as well. GODIVA~\cite{godiva} uses a transformer to map text tokens to video tokens from a image based VQVAE. Given the large number of tokens from multiple frames, GODIVA relied on a local-attention mechanism. Similarly, NUWA~\cite{nuwa} and NUWA-Infinity~\cite{wu2022nuwa} both employ auto-regressive architectures to generate videos and images from text. NUWA generates fixed size outputs, while NUWA-Infinity introduces a second layer of auto-regressive computation to support variable size videos. Likewise, CogVideo~\cite{cogvideo} argues the main reason behind low quality video generation is the scarcity of good text-video data and tried to leverage pre-trained text to images models to generate high quality video.

While Phenaki\xspace sticks to the same architecture principles, it has major differences with previous work. Most notably, NUWA, NUWA-Infinity and CogVideo treat videos as a sequence of independent images. This can lead to poor modeling of dynamics and generate motion artifacts. To combat this, NUWA-infinity used the previous frame during decoding to combat this. In Phenaki\xspace, we go further and treat videos as a temporal sequence of images which substantially decreases the number of video tokens given the redundancy in video generation, and results in a much lower training cost. The auto-regressive nature of the \model also allows us to effectively condition on previous frames and generates longer videos as detailed in Section~\ref{sec:method}.

Diffusion models are another class of models which recently have been used for conditional and unconditional video generation, which we call VDM~\cite{ho2022video}. In VDM, authors proposed replacing the conventional U-Net architectures for 2D image modeling with a 3D space-time model to run the diffusion process directly on pixels.
While this approach provides an effective formulation for modeling videos, it is limited to fixed size videos. To address this issue, VDM provides an auto-regressive extension, which allows the model to generate longer videos but it is typically impractical due to high sampling time of diffusion models.

Text conditional video generation is a relatively new field of research, nonetheless, image conditional video generation, commonly known as video prediction, and unconditional video generation have been studied more comprehensively. These papers include deterministic methods using a combination of recurrent and convolutional networks~\cite{ranzato2014video,srivastava2015unsupervised,finn2016unsupervised, wang2017predrnn}, variational based stochastic methods~\cite{babaeizadeh2017stochastic,svg,villegas2019high,fitvid} and more recently by learning a discrete representation~\cite{videovqvae,rakhimov2020latent,transframer}, auto-regressive models~\cite{videotranf,videogpt,ccvs,harp}, diffusion models~\cite{mcvd,harvey2022flexible,yang2022diffusion,diff4pred} flow based models~\cite{kumar2019videoflow}, and finally adversarial based methods~\cite{vondrick2016generating,saito2017temporal,tulyakov2018mocogan,clark2019adversarial,saito2020train,trIVD}. These works mostly consider limited domain (e.g. robotic videos) prediction/generation, or short fixed size clips. Section~\ref{sec:exp} provides comparison with some of these models.

\section{Hyper-Parameters}
\begin{table}[hbtp]
    \centering
    \begin{tabular}{l|c|l}
         Symbol & Value & Description \\\hline
         $t_x, w_x, h_x, c_x$ & $11, 128, 128, 3$ & Video dimensions \\
         $t_p, w_p, h_p, c_p$ & $2, 8, 8, 3$ & Patches dimensions (all frames except the first one) \\
         $t_z, w_z, h_z$ & $6, 16, 16$ & Video tokens dimension (before linear projection) \\
         $h_z$ & 512 & Hidden size in the transformer layer \\
         $d_z$ & 32 & Embedding dimension (after linear projection) \\
         $-$ & 4 & Number of layers for spatial transformer \\
         $-$ & 4 & Number of layers for temporal transformer \\
         $-$ & 2048 & MLP size \\
         $|E|$ & 8192 & Codebook size \\
         \hline
         - & AdamW &  Optimizer \\
         $\beta_1$ & 0.9 & first moment of gradient \\
         $\beta_2$ & 0.99 & second moment of gradient \\
         - & 1e-4 & Learning rate \\
         - & 1e-4 & Weight decay \\
         - & Cosine decay & Learning rate scheduler \\
         - & 1M & Target number of training steps for learning rate scheduler \\
         - & 100K & Warmup steps \\
         - & 10 & Gradient clipping magnitude \\
         - & 1028 & Batch size \\

    \end{tabular}
    \caption{Hyperparamters used for C-\vivit\xspace architecture and optimizer.}
    \label{tab:hparams_cvivit}
\end{table}

\begin{table}[hbtp]
    \centering
    \begin{tabular}{l|c|l}
         Symbol & Value & Description \\\hline
         $|\mathbf{z}|$ & 1536 & Sequence Length \\
         - & 24 & Number of layer \\
         - & 2048 & Embedding dimension \\
         - & 8192 & MLP dimension \\
         - & 32 & Number of heads \\
         \hline
         - & AdamW &  Optimizer \\
         $\beta_1$ & 0.9 & first moment of gradient \\
         $\beta_2$ & 0.99 & second moment of gradient \\
         - & 1e-4 & Learning rate \\
         - & 1e-4 & Weight decay \\
         - & Cosine decay & Learning rate scheduler \\
         - & 4M & Target number of training steps for learning rate scheduler \\
         - & 10K & Warmup steps \\
         - & 10 & Gradient clipping magnitude \\
         - & 512 & Batch size \\
    \end{tabular}
    \caption{Hyperparamters used for MaskGIT architecture and optimizer.}
    \label{tab:hparams_maskgit}
\end{table}

\section{Details of Experiments}

\subsection{Video Quantization}
\label{app:sec:vid_quant}
\subsubsection{Network architecture}
All encoder-decoder baselines have approximately $50\text{M}$ parameters.
The Convolutional baseline encoder architecture consists of $5$ convolutional blocks with channel multipliers of $[1, 1, 2, 2, 4]$, $2$ residual layers and $128$ hidden units per block, and embedding dimension of 256.
The ViT baseline encoder architecture consists of an image patchification step over non-overlapping $8\times8$ spatial patches which are linearly transformed into image tokens.
Next, we follow with $8$ transformer layers with $512$ hidden units, $8$ attention heads, $2048$ mlp units, and embedding dimension of $32$.
C-\vivit\xspace encoder architecture patches the first frame to non-overlapping $8\times8$ patches, and then the rest of the frames to non-overlapping $2\times8\times8$ spatio-temporal patches which are linearly transformed into video embeddings.
Next, C-\vivit\xspace encoder architecture consists of $4$ spatial and $4$ temporal transformer layers with $512$ hidden units, $8$ attention heads, $2048$ mlp hidden units, and embedding dimension of $32$.
The decoder architecture for all models is the same as the encoder but in reverse to put the latent embeddings back to image space.
The VQ objective is trained with commitment loss of $\beta=0.25$ and codebook size of $8192$.
The discriminator architecture is the StyleGAN \cite{stylegan} discriminator with blur resample, and channel multiplier of $1$.

\subsubsection{Training}
We train all encoder-decoder baselines and with StyleGAN~\citep{stylegan} discriminators with a batch size of $128$ using Adam optimizer~\cite{adam} with $\beta_1=0.9$ and $\beta_2=0.99$. We use a linear learning rate warmup to a peak value of $1 \times 10^{-4}$ over $100,000$ steps and then decaying over the remaining $900,000$ steps with a cosine schedule, and use a decoupled weight decay \cite{adamw} of $1 \times 10^{-4}$ for the encoder-decoder and discriminator. To capture longer time horizons during training and better evaluate temporal coherence, we downsample the MiT dataset from 25 FPS to 6 FPS and evaluate on videos of 10 frames at spatial resolution of $128\times128$.

\subsection{Image conditional video generation}
\label{app:sec:vid_bair}

\subsubsection{BAIR Robot Push C-\vivit\xspace architecture} \label{app:sec:bair}
We use a similar setup as in Section \ref{app:sec:vid_quant}, but the video tokenization step is done over $4\times4$ spatial patches on the first image and $2\times4\times4$ spatio-temporal patches in the rest of the video. The spatial encoder consists of 8 layers and the temporal encoder consists of 6 layers.

\subsubsection{Kinetics-600 C-\vivit\xspace architecture}
We use a similar setup as in Section \ref{app:sec:bair}, but both the spatial encoder and temporal encoder consist of 8 layers.

\subsubsection{MaskGIT architecture}
To perform video prediction in latent space in the BAIR Robot Push and Kinetics-600 datasets, we use an unconditional transformer architecture consisting of $24$ layers, $768$ hidden
units, $16$ attention heads, dropout and attention dropout rate of $0.1$, $3072$ mlp hidden units.

\subsubsection{Training and Inference}
As described in Table~\ref{tab:hparams_maskgit}, we train C-\vivit\xspace with the same optimizer setup as in Sec~\ref{app:sec:vid_quant}, but we do not downsample the FPS of any of the datasets in this section for fair comparison with the video prediction baselines. We train MaskGIT on the video tokens extracted using C-\vivit\xspace in an unconditional setting, that is, we do not assume frames or text inputs to be given. During training, we use the Adam \cite{adam} optimizer with $\beta_1=0.9$ and $\beta_2=0.99$. We use a linear learning rate warmup up to a peak value of $1\times10^{-4}$ over $10,000$ steps, and constant learning rate schedule for ${~\sim}2M$ steps. At inference time, we initialize MaskGIT given a number of input frames, and predict the rest of the frames depending on the dataset on which we evaluate.

\subsection{Text conditional video generation}
\label{app:sec:text2vid}

\subsubsection{Architecture}
In our text conditional video generation, we use the same C-\vivit\xspace architecture and training described in Section \ref{app:sec:vid_quant}.
To train MaskGIT, we include a text conditioning in the form of T5X embeddings \cite{roberts2022scaling} which are used as input through the use of cross attention with the video tokens.
We reduce the number of parameters of our base model for fairness in the quantitative comparisons against NUWA. The MaskGIT architecture used against NUWA consists of $20$ transformer layers with $1536$ hidden units, $24$ attention heads, and $6144$ MLP hidden units, resulting in $0.9\text{B}$ parameters similar to NUWA. For the main experiments in this paper, we use a larger architecture that consists of consists of $24$ transformer layers with $2048$ hidden units, $32$ attention heads, and $8192$ mlp hidden units, resulting in $1.8\text{B}$ parameters.

\subsubsection{Training and inference}
For all our text-conditional video generation, we use the training parameters Table~\ref{tab:hparams_maskgit}.

\subsubsection{Inference parameters against NUWA}
We use $\lambda=0.1$, $12$ MaskGIT iterations, and temperature of $4.0$.

\subsubsection{Inference parameters for ablation of image and video data for training.}
We use $\lambda=6$, $24$ MaskGIT iterations, and temperature of $4.0$.

\subsubsection{Inference parameters for all videos in the paper.}
We use $\lambda=12$, $48$ MaskGIT iterations, and temperature of $8.0$.

\input{figures/story2}

\section{Experiments}
\label{sec:exp}
To evaluate Phenaki\xspace, we test it on the following tasks: 1)~text conditional video generation, 2)~text-image conditional video generation, 3)~time variable text conditional video generation (i.e.) story\xspace mode, 4)~video quantization and 5)~image conditional video generation a.k.a. video prediction. To the best of our knowledge, 3)~time variable text conditional video generation has not been explored in prior work. Given the dynamic nature of videos, we highly encourage readers to visit \href{https://phenaki.github.io/}{phenaki.github.io}\ to check the generated videos. The website also includes qualitative comparisons to a subset of the prompts from the CogVideo paper~\cite{cogvideo}. While the focus is on the text to video generation tasks, it is remarkable that \model is still competitive on the more traditional video tasks despite not being developed explicitly for these tasks. We implemented \model in JAX~\cite{jax2018github} using FLAX~\cite{flax2020github} library.
\subsection{Text conditional video generation}
\label{sec:t2v}
Currently there is no established benchmark for evaluating text to video methods.
This makes comparing \model to recent methods such as NUWA~\cite{nuwa}, CogVideo~\cite{cogvideo}, NUWA-Infinity~\cite{wu2022nuwa} and video diffusion models \cite{ho2022video} difficult.

Unless specified otherwise, we train a 1.8B parameter \model model on a corpus of ${\sim}15$M text-video pairs at 8 FPS mixed with ${\sim}50$M text-images plus ${\sim}400$M pairs of LAION-400M~\cite{schuhmann2021laion} (more details in Appendix~\ref{app:sec:text2vid}).  The model used in the visualisations in this paper was trained for 1 million steps at a batch size of 512, which took less than 5 days. In this setup 80\% of the training data came from the video dataset and each image dataset contributed 10\%.

\paragraph{Qualitative evaluation:} Samples from this model can be seen in Figure~\ref{fig:compos} and additional samples are provided at \href{https://phenaki.github.io/}{phenaki.github.io}. We observe that there is a high degree of control over both the actors and the background dynamics in the videos. The appearance of the actors and the video style can be adjusted by the text prompt as well (e.g. a regular video, a cartoon or a pencil drawing).

On \href{https://phenaki.github.io/}{phenaki.github.io}\ we provide examples from prompts that were provided in the CogVideo~\cite{cogvideo} demo. Since there are substantial differences between these methods it is hard to compare them on an equal footing. As an example, there are massive differences in scale: 9B parameters for CogVideo and 1.8B for our model. Additionally, the training data is different. Finally, we do not know how representative the prompts in the CogVideo demo are for the general performance of the CogVideo.

\paragraph{Quantative comparison:}
The NUWA~\cite{nuwa} paper provided a qualitative evaluation on Kinetics-400. Since the NUWA model is only $0.9$B parameters we also use a model of the same size. Our model was trained on 50\% video and 50\% image data in this experiment. The NUWA model fine-tuned on Kinetics but the \model model is not: it is evaluated in a \emph{zero shot setting}.  The results in Table~\ref{table:t2v_comp} show that \model achieves comparable generation quality, in a zero-shot setting, compared to previous text to video methods that were actually trained or finetuned on this dataset.

\paragraph{On the importance of joint text-to-image and text-to-video training}
\input{tables/t2v_joint}
While there are some text-video datasets, text-image datasets dominate the internet in terms of quality and quantity \cite{videocc}. Consequently, there is simply not enough video data available to cover all the concepts present in text-image datasets. For example using only our video data, concepts such as pencil drawings or different painting styles cannot be learned. To be able to learn a model that can combine video dynamics with these additional concepts we have to combine training on image and video data. In Table~\ref{tab:video_image}, we evaluate the performance of using different ratios of video and images. We start with data splits of only video, and vary the ratio of image and video datasets up to using $50\%$ image and $50\%$ video datasets. In our results, we find that there is a trade-off in performance between models trained with only video video (i.e., significantly better FVD), and models trained with more image data (i.e., better text-video and text-image alignment, and significantly better FID in image datasets). On \href{https://phenaki.github.io/}{phenaki.github.io}\ we show samples from different models side by side where this trade-off between control over the content and the quality of the dynamics can be seen. We believe that the trade-off between concepts and dynamics will be improved as the quality and size of text-video datasets increases in the future.

\subsection{Text-Image conditional video generation}
Given that \model can be conditioned on both still images and text, an interesting setup is to \textit{animate} existing images given a text prompt. For this experiment, we use the same model from Section~\ref{sec:t2v} but conditioned on unseen pictures (captured with our phones from local subjects) and a related prompt. As it can be seen in Figure~\ref{fig:cat} the model can generate coherent videos starting from the given images, while following the given prompts.
\input{figures/composition}%

\input{figures/cat}%

\subsection{Visual story telling by dynamic text inputs}
A notable and useful feature of \model is that it is auto-regressive in time. This allows for generating long videos, while the prompt changes over time. Time variable prompts can be thought of as a \textit{story\xspace}; a narration of the entire video where each prompt corresponds to a scene from the video. This allows for creating dynamically changing scenes. To the best our knowledge, this paper is the first work to generate such videos.
An example of this can be seen in Fig.~\ref{fig:story} and on \href{https://phenaki.github.io/}{phenaki.github.io}. The way it works is that we generate a video with the first prompt and then extend it in time by conditioning a possibly new prompt and on the last $N$, typically 5, previously generated frames.

\subsection{Video Encoding}
\label{sec:vid_quant}
To evaluate the video encoding and reconstruction performance of C-\vivit\xspace, we use the Moments-in-Time (MiT)~\citep{monfortmoments} dataset. MiT contains ${\sim}802$K training, ${\sim}33$K validation and ${\sim}67$K test videos at 25 FPS. The MiT dataset, in contrast to other publicly available video datasets, is a high quality balanced dataset with high coverage and density of verbs depicting moments of a few seconds~\cite{monfortmoments}. We compare C-\vivit\xspace against per-frame image based encoder-decoders that have been used as video quantizers for conditional video generation~\citep{harp,nuwa,cogvideo,nuwa,cogvideo,godiva}: a ViT \cite{vit_vqgan} and a convolutional VQ-GAN\cite{vqgan}. The experimental details can be found in the Appendix~\ref{app:sec:vid_quant}.

\input{tables/quant_results}

As demonstrated in Table~\ref{tab:quant}, we evaluate the video reconstruction quality using FID~\cite{heusel2017gans} and FVD~\cite{unterthiner2018towards}. Both FID and FVD compare the distribution of generated videos (or images) to the ground truth distribution. The FID ignores temporal coherency, while the FVD measures how well the spatio-temporal dynamics of the videos are reconstructed. Results in Table~\ref{tab:quant} show that per-frame image based methods slightly outperform our video method (indicated by marginally higher FID of C-\vivit\xspace), however, they do poorly at modeling the spatio-temporal dynamics in video (significantly lower FVD of C-\vivit\xspace). This is expected as C-\vivit\xspace has spatio-temporal connections between patches in each frame, allowing space and time to be modeled together. In addition, C-\vivit\xspace compresses the video into fewer tokens per video compared to the image based baselines. This is crucial as the number of tokens drastically impacts the computational cost of the transformer in downstream tasks. Furthermore, C-\vivit\xspace tokens are auto-regressive in time which enables variable length videos to be modeled with the same encoder which is important for video extrapolation conditioned on previously generated frames.

\subsection{Image conditional video generation a.k.a Video prediction}
\label{sec:vid_bair}

\input{tables/vidpred_results}
To evaluate the learnt video representation of C-\vivit\xspace beyond reconstruction, we test it on the task of frame-conditioned video generation, also commonly known as video prediction~\cite{fitvid}. In this experiment, we test \model on BAIR Robot Pushing benchmark~\cite{BAIRRobotPush} where the task is to generate 15 frames conditioned on a given single frame. For open domain videos, we test \model on Kinetics-600~\cite{Kinetics600} where the task is to predict 11 frames given 5 frames. More details about these experiments can be found in Appendix~\ref{app:sec:vid_bair}. \cref{tab:bair,tab:kinetics} show the results of these experiments. Note that \model is not specifically designed for video prediction, therefore, it lacks components such as skip connections in U-Nets which are known to improve the performance for video prediction methods~\cite{svg,villegas2019high,fitvid}. Nevertheless, our method is competitive on these benchmarks with SOTA video prediction methods. Overall, these experiments show that \model is strong at modeling dynamics of the videos which is required for generating coherent videos from text.

\section{Introduction}
\afterpage{%
\input{figures/story}%
}

It is now possible to generate realistic high resolution images given a description \cite{DALLE,ramesh2022hierarchical,nichol2021glide,saharia2022photorealistic,PARTI}, but generating high quality videos from text remains challenging. In essence, videos are just a sequence of images, but this does not mean that generating a long coherent video is easy.  In practice, it is a significantly harder task because there is much less high quality data available and the computational requirements are much more severe~\cite{clark2019adversarial}. For image generation, there are datasets with billions of image-text pairs (such as LAION-5B~\cite{schuhmann2021laion} and JFT4B~\cite{zhai2022scaling}) while the text-video datasets are substantially smaller~e.g. WebVid~\cite{bain2021frozen} with ${\sim}10$M videos, which is not enough given the higher complexity of open domain videos. As for computation, training current state-of-the-art image generation models is already pushing the state-of-the-art computational capabilities~\cite{PARTI}, leaving little to no room for generating videos, particularly videos of variable length.

To make the matters worse, one can argue that a single short text prompt is not sufficient to provide a complete description of a video (except for short clips), and instead, a generated video must be conditioned on a sequence of prompts, or \textit{a story\xspace}, which narrates what happens over time. Ideally, a video generation model must be able to generate videos of arbitrary length, all the while having the capability of conditioning the generated frames at time $t$ on prompts at time $t$ that can vary over time. Such capability can clearly distinguish the \textit{video} from a ``moving image" and open up the way to real-world creative applications in art, design and content creation. To the best our knowledge, story\xspace based conditional video generation has never been explored before and this is the first paper to take early steps towards that goal. A traditional deep learning approach of simply learning this task from data is not possible, since there is no story-based dataset to learn from. Instead, to achieve this we rely on a model that is designed specifically with this capability in mind.

In this paper, we introduce \model, a text to video model trained on both text to video and text to image data that can:
\begin{itemize}[leftmargin=0.2in]
    \item[--] Generate temporally coherent and diverse videos conditioned on open domain prompts even when the prompt is a new composition of concepts (Fig.~\ref{fig:compos}). The videos can be long (minutes) even though the model is trained on 1.4 seconds videos (at 8 fps).
    \item[--] Generate videos conditioned on a story\xspace (i.e. a sequence of prompts), e.g.~Fig.~\ref{fig:story} and Fig.~\ref{fig:story2}.

\end{itemize}

To enable these capabilities, we could not rely on current video encoders, because they either can only decode fixed size videos or they encode frames independently. Hence, we introduce C-\vivit\xspace, a novel encoder-decoder architecture that:
\begin{itemize}[leftmargin=0.2in]
\item[--] Exploits temporal redundancy in videos to improve reconstruction quality over a per frame model while compressing the number of video tokens by 40\% or more.
\item[--] Allows encoding and decoding of variable length videos given its causal structure.
\end{itemize}

\input{figures/models_diagram}

\section{The \model model}
\label{sec:method}
Inspired by the previous work in auto-regressive text to image~\cite{DALLE,PARTI,saharia2022photorealistic} and text to video~\cite{nuwa,wu2022nuwa,cogvideo}, Phenaki\xspace is designed with two main components (see Figure~\ref{fig:models}): an encoder-decoder model which compresses videos to discrete embeddings (i.e. tokens) and a transformer model to \textit{translate} text embeddings to video tokens. To get the text embeddings, Phenaki\xspace uses a pre-trained language model, T5X~\cite{roberts2022scaling}. We will discuss each one of these components in the following subsections.

\subsection{Encoder-decoder video model: C-\vivit\xspace}
One of the primary challenges for generating video from text, is to get a compressed representation of videos. Previous work on text to video either use per-frame image encoders~\cite{cogvideo,nuwa,harp} such as VQ-GAN~\cite{vqgan} or fixed length video encoders~\cite{godiva} such as VideoVQVAE~\cite{videovqvae}. The former allows for generating videos of arbitrary length, however in practice, the videos have to be short because the encoder does not compress the videos in time and the tokens are highly redundant in consecutive frames. The latter is more efficient in the number of tokens but it does not allow to generate variable length videos. In Phenaki\xspace, our goal is to generate videos of variable length while keeping the number of video tokens to a minimum so they can be modeled with a transformer within current computational limitations. To do so, we introduce C-\vivit\xspace, a causal variation of ViViT\xspace~\cite{vivit} with additional architectural changes for video generation, which can compress the videos in temporal and spatial dimensions, while staying auto-regressive in time, This capability allows for generating videos of arbitrary length auto-regressively.

\paragraph{Encoder architecture:}
As illustrated in Figure \ref{fig:models}, we start with a video sequence of $t_x + 1$ frames with a resolution of $w_x\times h_x$ and $c_x$ channels: $\mathbf{x} \in \mathds{R}^{(t_x + 1) \times h_x \times w_x \times c_x}$. This sequence will be compressed into a token representation of size $(t_z + 1)\times w_z \times h_z$ where the first $w_z\times h_z$ tokens represent the first frame independently from the rest of the video, and the remaining tokens represent spatio-temporal video tokens that auto-regressively depend on previous frames.
To do so, we extract non-overlapping image patches of size $w_p \times h_p \times c_p$ from the first frame and video patches of size $t_p\times w_p \times h_p \times c_p$ from the rest of the video.
We typically use all channels at once such that the number of patches equals the number of video tokens $t_z=\frac{t_x}{t_p}$, $w_z=\frac{w_x}{w_p}$ and  $h_z=\frac{h_x}{h_p}$.
Each of these patches is flattened and linearly projected into a $d_z$ dimensional space.
We combine the spatial dimensions to have a tensor of shape $(t_z + 1)\times w_z*h_z \times d_z$ where the spatial and temporal dimensions are separated.
Then multiple transformer layers are applied along the spatial dimensions with all-to-all attention.
This is followed by multiple transformer layers over the temporal dimension with causal attention such that each spatial token only observes spatial tokens from previous frames in an auto-regressive manner.
The effect of this is that the first frame can be completely independently encoded.
This opens up the possibility of text to image training to be embedded naturally into our video model.
The second advantage is that we can condition the video generation process on a number of starting frames.
The resulting patch embeddings $\mathbf{z}$ of shape $t_z\times w_z \times h_z \times d_z$ are then tokenized into learned codewords $c_z$ by vector quantization.
The codebook learning will be discussed later together with the losses.

\paragraph{Decoder architecture:}
The C-\vivit\xspace decoder is simply an upside down version of the encoder. First tokens are transformed into embeddings. This is followed by the temporal transformer, then the spatial transformer.
After the output of the spatial transformer, we apply a single linear projection without activation to map the tokens back to pixel space.

\paragraph{Quantization and Losses:}
To learn a discrete latent space, we quantize our encoder outputs into the entries of a learned codebook via the vector quantization (VQ) objective in VQVAEs \citep{vqvae},
\begin{equation}
    L_{\textit{VQ}} = \| \text{sg}(\mathbf{z}) - \mathbf{e} \|^2_2 + \beta \| \mathbf{z} - sg(\mathbf{e}) \|^2_2,
\end{equation}
where $\text{sg}(x) \equiv x$, and $\frac{\text{d}}{\text{d}x} \text{sg}(x) \equiv 0$ is the stop-gradient operator, $\beta$ is the commitment loss weight, and $\mathbf{e}$ is a codebook vector from codebook $\mathbf{E}$. The index to the codebook vector closest to $\mathbf{z}$ is found by $i = \text{argmin}_j \| \mathbf{z} - \mathbf{E}_j \|^2_2$.
In addition to the VQ objective, we adopt the factorized and $\ell_2$-normalized codes from ViT-VQGAN~\citep{vit_vqgan} to improve codebook usage and reconstruction quality.

To train our model, we use a combination of $L_2$ loss, image perceptual loss $L_{\textit{IP}}$~\citep{JohnsonPerceptual2016,ZhangPerceptual2018}, video perceptual loss $L_{\textit{VP}}$ by using the I3D network~\citep{i3d} as feature extractor, and adversarial loss $L_{\textit{Adv}}$ with StyleGAN architecture~\citep{stylegan}.
As training objective, we use the following
\begin{equation}
    L = L_{\textit{VQ}} + 0.1\times L_{\textit{Adv}} + 0.1\times L_{\textit{IP}} + 1.0\times L_{\textit{VP}} + 1.0\times L_2.
\end{equation}

\paragraph{Novelty over the ViViT\xspace architecture:}
While our proposed C-\vivit\xspace architecture is inspired by the factorized encoder in ViViT\xspace~\cite{vivit}, we modify their architecture to enable self-supervised learning from unlabeled videos. We first remove the \texttt{[CLS]} tokens in the spatial and the temporal transformers. Next, we apply temporal transformer for all spatial tokens computed by the spatial encoder, in contrast to  single run of the temporal transformer over the \texttt{[CLS]} tokens in ViViT\xspace.
Most importantly, the ViViT\xspace encoder requires a fixed length video input due to the all-to-all attention in time. Therefore, we apply causal attention instead such that our C-\vivit\xspace encoder becomes auto-regressive and allows for a variable number of input frames which are necessary to learn from image datasets, and auto-regressively extrapolate video or single frames into the future.

\subsection{Text-to-video generation with bidirectional transformers}
In this stage, the text-to-video task can be formulated as a sequence-to-sequence problem to predict video tokens given the paired text embeddings. Most of recent methods \citep{DALLE, PARTI, nuwa, cogvideo} adopt a  transformer model for these sequence-to-sequence tasks. In their models, they use an auto-regressive transformer which predicts the image or video tokens sequentially given the encoded text features.
As a result, the sampling time scales linearly with the sequence length, even when caching is used. This becomes impractical for long video sequence generation.

\paragraph{Masked bidirectional transformer:}

In this work, we aim to reduce the sampling time by having a small and fixed sampling step disregarding different video sequence lengths. Inspired by previous work for image generation~\citep{MASKGIT}, we use a bidirectional transformer since it can predict different video tokens simultaneously. For training step $i$, we first sample a mask ratio $\gamma_{i}$ from 0 to 1 and randomly replace $\lceil \gamma_{i} \cdot N \rceil$ tokens with the special token \texttt{[MASK]}, where $N$ is the video sequence length. Then
we learn the model parameters by minimizing the cross entropy loss on those masked tokens given the encoded text embeddings and unmasked video tokens. During inference, we first label all of the video tokens as the special token \texttt{[MASK]}. Then, at each inference step, we predict all the masked (unknown) video tokens in parallel conditioned on the text embeddings and unmasked (predicted) video tokens. We keep a ratio $\beta_{i}$ of the predicted tokens at sampling step $i$ and the remaining tokens are re-masked and re-predicted in the next step.

As discussed in MaskGIT~\citep{MASKGIT}, the masking schedule $\gamma_{i}$ and sampling schedule $\beta_{i}$ have a significant effect on the samples quality therefore we follow the same strategies. Compared to an auto-regressive transformer, the number of sampling steps is an order-of-magnitude smaller (typically we use values in the range of 12 to 48). Generally speaking, more sampling steps improves the quality.

\paragraph{Losses and training strategies:}
Given a pre-trained C-\vivit\xspace, videos are encoded into codebook ids $\textbf{a}$ of shape $(t_z + 1)\times w_z \times h_z$ which are flattened into a long vector using the raster ordering from \cite{vit_vqgan}. We then model the text-conditional video token distribution using \textit{Masked Visual Token Modeling} (MVTM)~\cite{MASKGIT}:
\begin{equation}
    L_{\text{mask}}= - \sum\nolimits_{\forall i\in[1,N],m_i=1} \log p(a_i | \textbf{a}_{\bar{M}}, \mathbf{p}),
\end{equation}
where $\textbf{a}_{\bar{M}}$ represents the masked version of $\textbf{a}$, $m_i$ is a binary variable indicating whether $a_i$ is masked or not, $N$ is the number of video tokens, and $\mathbf{p}$ is the text condition embedding.
In addition to the MVTM objective, we train using classifier-free guidance by dropping the text condition $10\%$ of the time during training~\cite{cfg,PARTI} .
Finally, we dynamically adjust the MVTM objective during training to allow the use of image and video datasets as a single large dataset.
We achieve this by only applying the masking ratio and objective on the first $w_z \times h_z$ tokens if only a single frame is given or over all video tokens if a full video is given.
This mixed image and video dataset training strategy allows our models to learn concepts only present in image datasets, and transfer them to concepts present video datasets (e.g., the pencil drawing styled video of the panda in Figure.\ref{fig:compos}).

\paragraph{Inference and auto-regressive generation of long videos:}
At inference time, we sample videos tokens by the same iterative process used in \cite{MASKGIT} with classifier-free guidance scale $\lambda$ to control alignment between the generation and the text condition. Once the first video is generated, we can extrapolate additional frames auto-regressively by encoding the last $K$ generated frames in the last video using C-\vivit\xspace, initializing MaskGIT with the tokens computed by our C-\vivit\xspace encoder, and proceed to generate the remaining video tokens conditioned on a text input. During video extrapolation, the text condition can be the same or a different one which enables our model to dynamically create visual transitions between the previous and current text condition visual content, effective generating a visual story an described by the input text.

\section{Conclusion}

We introduced \model, a model which is capable of generating variable length videos conditioned on a sequence of open domain text prompts. \model uses C-\vivit\xspace as video encoder. C-\vivit\xspace is a new model which provides temporal-spatial compression while being auto-regressive in time.
The C-\vivit\xspace model is a crucial part of \model that allows it to generate variable length videos. We demonstrate how joint training on images and videos can improve the generation quality, and diversity, given the existence of much larger image-text dataset with order of magnitude more samples. The \model model achieves good performance on video prediction, it can be used as to generate long videos conditioned on a text prompt. Additionally it is able to condition on both text and a starting frame. Finally, \model is not limited to generating a video depicting a single concept or caption. It is actually able to generate longer coherent video stories based on a sequence of text prompts. The more complex narratives it can visualize demonstrate how this can become a great creative tool for story telling.

\section*{Ethics Statement}
While we have not explored potential downstream applications of the generative models described in this work, we believe Phenaki can have a positive impact in a variety of creative settings. In general, many of the samples from the model will not perfectly correspond to the input caption or the user’s intent; however, the end-user is likely to gain considerable time savings even if only one of the generated samples aligns with their intent. We thus foresee Phenaki being useful in eventually empowering users to accelerate their creativity, especially since the model can so quickly generate videos. Phenaki and similar models will be part of an ever-broad toolset for artists and non-artists alike, providing new and exciting ways to express creativity.

The flip-side of this acceleration and ease-of-use is the potential for harmful impact, as with many of the prior or concurrent work in generative modeling. An easy-to-use system like Phenaki can be repurposed for generating maliciously fake content and enable spreading of such content much easier. While the quality of the videos generated by Phenaki is not yet indistinguishable from real videos, getting to that bar for a specific set of samples is within the realm of possibility, even today. This can be particularly harmful if Phenaki is to be used to generate videos of someone without their consent and knowledge.

Like DALLE-2~\cite{ramesh2022hierarchical}, Imagen~\cite{saharia2022photorealistic}, Parti~\cite{PARTI} and others, Phenaki is trained on a collection of datasets that is known to encode a number of undesirable biases. LAION-400M~\cite{schuhmann2021laion} specifically has a variety of issues regarding violence, pornography, gore. While our primary image and video datasets have minimal traits like this, we did incorporate LAION-400M into our training and observed better results. In a currently training version of Phenaki, we use a set of datasets that minimizes such problems.

Taken together, these issues contribute to our decision not to release the underlying models, code, data or interactive demo at this time. Before we can do that, we want to focus our efforts on better understanding of data, prompt and output filtering. We would also like to more explicitly measure the biases encoded in the outputs of Phenaki, so that we can further mitigate them actively, either in the data, models or pre/post-processing steps.


\subsection{Multi-stream Class Incremental Learning Model}

We define a multi-stream class incremental learning model \textbf{M} as a combination of three key components: 1) a single stream base method \textbf{X}, 2) an additional stream of information \textbf{Y}, and 3) a method of fusing stream \textbf{Z}.

\begin{center}
    \textbf{M} = Base method \textbf{X} + Feature stream \textbf{Y} + Fusion mechanism \textbf{Z}
\end{center}

At this point, the base method represents any method that follows the general setting described in Sec.\ref{subsec:conventionalCIL}. \textbf{Y} serves as a piece of additional domain information that gives cues to the learning model apart from RGB images.
Normally, \textbf{Y} is specific to the domain of the task. Lastly, \textbf{Z} presents a fusion mechanism that enables method \textbf{M} to incorporate additional information stream \textbf{Y} into the incremental learning process. From this decomposition, our CG-IMIF replaces: 1) the representative stream \textbf{Y} with color-specific information, and 2) the fusion technique \textbf{Z} with the proposed IMIF.
In the following, we describe our proposed Color histogram guidance stream and Incremental Multi-stream Intermediate Fusion technique in Sec.\ref{ssec:domain-specific information extractor} and Sec.\ref{subsec:fusion}, respectively.

\subsection{Color Histogram Guidance Stream} \label{ssec:domain-specific information extractor}

Pill images compose of various features which can be used as discriminative factors in classification problems. Among those is color distribution information which can be approximated by the color histogram of pill images. The color histogram represents the color distribution of the input image in terms of three different channels: red, green, and blue. In detail, it is often encoded into a single vector where the indexes of entries are mapped to a set of all possible colors. In grey-scale images, values of vector entries store the frequency that counts the total number of pixels having the intensity (\textit{i.e.}, color value), which is hashed by the corresponding index of the vector. However, in a three-dimensional image, color ranges for each channel are associated across color channels to formulate a unique combination. This combination accounts for those pixels of which color ranges lie in three discrete ranges of value: $[r_i, r_j];[g_k, g_l];[b_m, b_n]$. The color feature vectors are useful to make a distinction among pill instances that have similar shapes but different colors. The color ranges for each channel of the RGB images are divided into eight segments in our problem, where each segment represents 32 different consecutive color values. After that, the color histogram vector can be obtained by accumulating the quantities of pixels assigned to a specific color range for each channel to result in a vector with 8$\times$8$\times$8 = 512 elements. Fig.~\ref{fig:example_color_histogram} illustrates a few examples of extracting the color histogram stream for each corresponding cropped pill image.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\textwidth]{images/ColorHistogram.pdf}
    \caption{Samples of pill images from VAIPE-PCIL dataset are shown on the first row. Second row exhibits the corresponding color histogram information for images on the upper row respectively.}
    \label{fig:example_color_histogram}
\end{figure}

\subsection{Incremental Multi-stream Intermediate Fusion Technique}
\label{subsec:fusion}

\subsubsection{Traditional Early Fusion}
A naive fusion technique \textbf{Z} concatenates different streams of information right after the feature extraction phase. Specifically, feature vectors $f_{r} \in \mathbb{R}^{d_r}$, and $f_Y \in \mathbb{R}^{d_Y}$ are extracted from raw RGB images, and additional information stream \textbf{Y}, respectively. Both of these features are then fed into separate projection layers to project into the same latent space. In practice, the projection layers are implemented by a single hidden layer controlled by parameters $\Theta^p=[W^p, b^p]$ as follows
\begin{equation}
s_{r} = \sigma(W^p_r.f_r+b^p_r),\\  s_{Y} =\sigma(W^p_Y.f_Y+b^p_Y), \\  s_{g} = [s_r, s_Y].
\label{eq:projection_layer}
\end{equation}
The projected vectors are then concatenated to obtain the global feature vector $s_g \in \mathbb{R}^{d_g}$. This global feature $s_g$ can be considered as a single input into any traditional single stream CIL methods as shown in Fig.~\ref{fig:traditional_single_stream}.
\begin{figure}[h!]
    \centering
    \includegraphics[width=10cm,height=5cm]{images/traditional_single_stream.pdf}
    \caption{The traditional training paradigm with only single-stream information use by almost existing exemplar-based CIL methods.}
    \label{fig:traditional_single_stream}
\end{figure}
\noindent \subsubsection{Intermediate Fusion}
We observed and hypothesized that fusing information stream in an early manner for a class incremental learning problem is not optimal. The global feature $s_g$ is problematic since it is regularly updated at each incremental task. As a result, the projection layer in the early phase can not find good parameters that balance the performance of old and novel categories in different tasks. Therefore, we propose to relocate the projection layer to the intermediate phase instead (\textit{i.e}, incremental learning phase) by initiating an entirely novel projection layer in an incremental manner. When $C^t$ from new task $t$ arrives at the system, a new classification layer $CL_t$ is created . This layer accompanies with an attached projection layer specific to the information stream of a specific task. Therefore, the parameters controlling the projection layer for each information stream are different from those defined in the early fusion.

\begin{equation}
\Theta^{p^t}_{r} = [W^{p^t}_r, b^{p^t}_r],
\Theta^{p^t}_{Y} = [W^{p^t}_Y, b^{p^t}_Y].
\end{equation}

In this way, the projection layer could maintain its optimal performance for old tasks without significantly adjusting its weight toward the novel ones. Fig.~\ref{fig:our_X_CG_IMIF} illustrates how we direct streams of additional knowledge in an incremental manner.
\begin{figure}[h!]
    \centering
    \includegraphics[width=\textwidth]{images/X-CG-IMIF.pdf}
    \caption{Our proposed CG-IMIF architecture composes of: 1) color histogram feature extraction (orange block), and 2) intermediate fusion framework (purple block) to incorporate additional information stream.}
    \label{fig:our_X_CG_IMIF}
\end{figure}
\section{Introduction}
Pill image recognition task has attracted various studies recently with the aim to design high-quality algorithm for visual-based assistance system on pill images. This can help the healthcare community automatically identify unknown pill categories by taking several real-world pictures with mobile devices. It is noteworthy that real-world scenarios of pill images are often challenging due to the changing background as well as variances of pill instances in terms of shape, color, and texture. There have been several works that are developed to mitigate such challenges, most of them are based on hand-crafted features \cite{6467032, articleB, articleA,articleC}. These works are then utilized by Ling et al.\cite{9157392} and combined with a two-stage training strategy to create a novel framework for the pill recognition model in few-shot learning. Another approach is to explore external knowledge from medical text data (e.g. prescription) to improve the detection performance of visual-based models~\cite{nguyen2022image,nguyen2022novel}. However, existing models are often limited by novel instances of pill categories which frequently arrive at a pill recognition system. This often happens when a novel class of pill instance is introduced by images uploaded from the end-user using mobile devices or from the healthcare community. A report in \cite{fdaCDERsMolecular} shows that there are roughly 40-50 novel drugs being approved each year.
In such a scenario, the core learning model of the system, which is often deployed in a lightweight device (\textit{e.g}, mobile phones), might need to rewind the training process on the whole training data (in which novel categories participate). This is not an effective strategy for many reasons. Memory allocated for such extensively training data is often limited. Acquiring novel knowledge while maintaining what the model has learned so far requires the system to store a huge amount of samples for both old and new classes, which is infeasible. Another solution for this is to provide an initial training dataset for the model. The model is then fine-tuned on novel categories to update the model's knowledge about new pill instances. However, this fine-tuning scheme suffers from a serious behavior of the learning system which is widely known as catastrophic forgetting \cite{articleCatas, https://doi.org/10.48550/arxiv.1312.6211} (degrading performance on old tasks while accessing data of novel tasks). This system, therefore, is in need of a flexible and effective strategy to handle the novel real-world object categorization of pill image instances. In this way, it would be able to incrementally learn from new classes without exhaustively storing old category samples. This scenario is called class-incremental learning (CIL).

The progress of studies on class incremental learning (CIL) for visual tasks has been developed significantly for many years. The general setting of CIL is that the disjoint sets of different classes arrive at the learning algorithm gradually. Many works such as \cite{DBLP:journals/corr/abs-1903-07864,Hou_2019_CVPR,DBLP:journals/corr/RebuffiKL16,DBLP:journals/corr/abs-1807-09536, DBLP:journals/corr/abs-1905-13260} have proposed several methods which employed available techniques to tackle the mutual challenge: catastrophic forgetting. Knowledge distillation \cite{DBLP:journals/corr/HintonVD15} is the most common technique which is widely adopted to tackle catastrophic forgetting and was first applied to the CIL setting by Li et al.\cite{DBLP:journals/corr/LiH16e}. After that, a derived version \cite{DBLP:journals/corr/RebuffiKL16} with additional usage of representation learning was proposed, in which valuable herding exemplars are replayed frequently to keep track of the old knowledge. The strategy of herding is to pick those neighbors which are nearest to the mean sample of the class. Using this herding strategy, Castro et al.\cite{DBLP:journals/corr/abs-1807-09536} managed to build an end-to-end framework with an additionally balanced fine-tuning strategy. On the other hand, Wu et al.\cite{DBLP:journals/corr/abs-1905-13260} introduced a bias correction approach by adding a bias correction layer. This is conducted at the last layer of each incremental learning task to refine the overall scores for the final prediction. Meanwhile, Hou et al.\cite{Hou_2019_CVPR} identified the imbalance between previous and new data as the main issue leading to catastrophic forgetting. They tackled this imbalanced scenario by incorporating three main components: cosine normalization, less-forget constraint, and inter-class separation.

In this research, we aim to investigate the application of CIL methods in a pill classification system. Fig.\ref{fig:overview} illustrates the effect of such a system with and without class incremental learning capability. To the best of our knowledge, we are the first to explore incremental learning on the pill classification system. Existing single stream incremental learning methods \cite{DBLP:journals/corr/abs-1903-07864,Hou_2019_CVPR,DBLP:journals/corr/RebuffiKL16,DBLP:journals/corr/abs-1807-09536, DBLP:journals/corr/abs-1905-13260}, when being applied to a domain of application for practical usage, can be improved with the help of some domain-specific knowledge. This serves as additional information which might collaborate well with the original RGB image to alleviate catastrophic forgetting. The introduction of a supplementary information stream requires a prudent strategy to incorporate such information. Based on this motivation, we propose a novel integration framework that serves as a plug-in technique for any available class incremental learning algorithms. Our fusion framework enables the incremental learning methods to receive additional information streams as cues. This will then help to flexibly update corresponding feature representations in an optimal way for each learning task through the intermediate stage. To demonstrate the usage of such an integration framework, we consider color information as additional stream and devise an approach, named ``\textit{Color Guidance with Multi-stream intermediate fusion}"(CG-IMIF). Experimental results on a real-world incremental pill image classification dataset called VAIPE-PCIL show that the proposed learning framework consistently surpasses most metric scores of various state-of-the-art methods in different task settings.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.75\textwidth]{images/pipeline_overview.pdf}
    \caption{The pipeline for a learning algorithm to acquire knowledge of pill categories could be divided into two options: (a) feeding a fixed pill images database to an off-the-shelf deep learning algorithm; (b) maintaining a few samples of old categories as exemplars, combining with novel categories to form a growing pill image dataset, and finally feeding into a growing deep classification model.}
    \label{fig:overview}
\end{figure}

\noindent Our contributions can be summarized in the following three aspects:
\begin{enumerate}
	 \item {We introduce CG-IMIF, a novel incremental learning framework based on multiple streams for the task of pill classification from images. To the best of our knowledge, we are the first to introduce the incremental learning capability to this task and provide a new approach to tackle challenges in learning novel pill classes.}

	 \item {We conduct thorough experiments and in-depth ablation studies to demonstrate the effectiveness of the proposed approach on a real-world incremental pill image classification dataset. Experimental results show that the CG-IMIF consistently outperforms previous state-of-the-art methods by a large margin.}

\end{enumerate}
The rest of this paper is organized as follows. We briefly formulate the problem setting of pill CIL, which we aim to solve in Section 2. Details of our proposed CG-IMIF framework are described in Section 3. Experimental results and further analysis are presented in Section 4 and 5. Finally, we conclude the paper with our discussion on strengths and limitations in Section 6, and 7.
\section{Preliminaries}
\subsection{Problem Definition and Notation}
Generally, the \textbf{Class Incremental Learning (CIL)} problem represented by $\tau$ consists of a sequence of $n$ image classification learning tasks
\begin{equation}
    \tau = [(C^1, P_{train}^1, P_{test}^1), (C^2, P_{train}^2, P_{test}^2), ..., (C^n, P_{train}^n, P_{test}^n)] ,
    \label{equ:probform}
\end{equation}

\noindent where each tuple $(C^t, P_{train}^t, P_{test}^t)$ depicts a task $t$.
$C^t$ is a set of $m^t$ categories, \textit{i.e.}, $C^t=\{c_1^t, c_2^t, ..., c_{m^t}^t\}$, $P_{train}^t$ and $P_{test}^t$ denote the training and testing data, respectively. To represent the total number of classes up to the current task, we define $M^t=\sum_{i=1}^{t}\left | C^i \right |$. The training, and testing data is defined as $P^t = \{(X^t, Y^t)\}$ where $X^t$ and $Y^t$ denote the training images and their corresponding labels, respectively. During the training phase, the learning model at stage $t$ is presented with categories set $C^t$, training samples $P_{train}^t$, and an exemplar set $K_t$. In practice, $K_t$ is a fixed-size set acting as a support set which helps to retain a partial set of images and the corresponding labels from previous training data, \textit{i.e.}, $K_t \subseteq P_{train}^1\cup P_{train}^2 \cup ... \cup P_{train}^{t-1}$.
Therefore, a revised version of training samples at stage $t$ can be obtained by combining $K_t$ and $P_{train}^t$, $K_t \cup P_{train}^t=V_{train}^t$. It is also assumed that categories of different learning tasks do not overlap (\textit{i.e} $C^i \cap C^j = \varnothing$ where $i \neq j$). At testing time, the performance of learner $t$ is evaluated on all of the previous seen categories $\bigcup_{i=1}^{t}C^t$ with samples from $\bigcup_{i=1}^{t}P_{test}^t$.

\subsection{Conventional CIL Methods}
\label{subsec:conventionalCIL}
Several CIL methods have been proposed which consider various properties of CIL problem to tackle mutual challenge: catastrophic forgetting. Most CIL works are divided into two branches: exemplar-based, and non-exemplar-based approaches. While the latter is much more challenging, the former is more practical since it is reasonable to maintain a few samples of old classes to avoid performance degrading. Within the scope of this research, we aim to exploit the capability of exemplar-based CIL methods by attaching these to our proposed framework. Therefore, we first describe a few core components of exemplar-based CIL approaches as follows.\\
\textbf{Representative Memory} is a set of samples from categories of old tasks and is represented by $K_{t-1}$. It serves as an exemplar set to support model in revisiting knowledge acquired from old tasks. In an exemplar-based approach, the learning model can only access the previous category set $C^{t-1}$ through $K_{t-1}$. The size of the support set is often limited and mainly divided into two memory settings: 1) a constant number of exemplars per class, and 2) and a limited capacity of \textit{S} samples. In the first setting, the size of the support set $K_t$ grows with the number of classes. In addition, the size of $K_t$ in the second memory setting is constant over the time $t$. Samples across categories are manipulated frequently with two main operations: new sample selection and old sample removal.
For each class, a sorted list of its samples is maintained based on their distances to the class' mean feature vector. Hence, the most representative samples for each class are selected as members in the next support set $K_{t+1}$. Meanwhile, the remaining samples are ignored to reserve slots for novel samples from new classes.\\
\textbf{Growing Deep Neural Networks} in an exemplar-based method is constructed by two main factors: the common feature extractor backbone, and a growing classification layer module. At a specific learning stage $t$, new classification head $CL_t$ is initiated to allocate corresponding parameters $W_t$. Feature vectors, after being extracted by the feature extractor, are fed into $CL_t$ to produce prediction logits for the current category set $C^t$. The size of the logits after being input to $CL_t$ is equal to the size of the category set $C^t$. The prediction vectors are then utilized to compute the traditional cross-entropy loss which represents the training loss on the set of pill images $P^t_{train}$ for the current task. On the other hand, the old classification head $CL_{t-1}$ can be used to represent the old knowledge of the model. Samples from support set $K_t$ can be passed through a list of classification head module $CL_{i}$ from the first task to the latest old task $t-1$ (\textit{i.e} $i
\in [0, t-1]$) in order to obtain prediction logits.\\
\textbf{Cross-Distillation Loss Function} is common in most of exemplar-based methods. This is constructed by combining cross-entropy, and distillation loss function. The cross-entropy loss function helps minimize the overall empirical errors when learning on new category set $C_t$ at task $t$. Meanwhile, the distillation loss function plays a role in distilling the old model $M_{t-1}$ from previous tasks into the current model $M_t$ to avoid catastrophic forgetting. Let's consider the incremental learning model at a specific learning stage $t$ where it has obtained $t-1$ numbers of classification heads. New classification head $CL_t$, which is now added to learn on new task $t$, produce the prediction logits as $o(x) = [o_1(x), o_2(x), ..., o_t(x)]$ for any input $x$. Similarly, output logits which are produced by old classification head can be represented as $\hat{o}(x) = [\hat{o}_1(x), \hat{o}_2(x), ..., \hat{o}_{t-1}(x)]$. With these representation, the distillation loss can be computed for all samples from exemplar sets $K_{t-1}$ and from new classes $P^t_{train}$ (\textit{i.e.} $K_{t-1} \cup P^t_{train} = V_{train}^t$) as follows:

\begin{equation}
\begin{aligned}
L_{d} &=\sum_{x \in V_{train}^t} \sum_{k=1}^{t-1}-\hat{\pi}_{k}(x) \log \left[\pi_{k}(x)\right], \\
\hat{\pi}_{k}(x) &=\frac{e^{\hat{o}_{k}(x) / T}}{\sum_{j=1}^{t-1} e^{\hat{o}_{j}(x) / T}}, \quad \pi_{k}(x)=\frac{e^{o_{k}(x) / T}}{\sum_{j=1}^{t-1} e^{o_{j}(x) / T}},
\end{aligned}
\label{eq:distillation_loss}
\end{equation}

where T plays as the temperature scaling factor.
Meanwhile, the groundtruth label for each sample $x$ (\textit{i.e.} y(x)) for new category sets along with softmax of logits of the $k$-th category set (i.e. $p_k=softmax(o_k(x))$ ) can be used to compute the cross-entropy loss function as follows

\begin{equation}
    L_{c}=\sum_{x \in V_{train}^t} \sum_{k=1}^{t}-y(x) \log \left[p_{k}(x)\right],
\label{eq:cross_entropy}
\end{equation}

The final cross-distilled loss function; therefore, can be obtained by combining distillation loss function in Eqn.\ref{eq:distillation_loss} and cross-entropy loss function in Eqn.\ref{eq:cross_entropy}

\begin{equation}
    L = \alpha L_d + (1-\alpha) L_c,
\end{equation}

where the scalar value $\alpha$ controls the balance between the two functions.
\section{Methodology}
The majority of current pill identification methods rely on RGB images. Therefore, to the best of our knowledge, most existing systems fail to address hard examples (\textit{e.g}, pills with very similar shapes and colors)~\cite{9157392}. This problem becomes more challenging in the context of the class incremental learning. In this problem, we have to cope with two issues at the same time: 1) recognizing pill instances that belong to the novel classes, and 2) not forgetting the previously learned knowledge of the old ones.
We seek in this study robust domain-specific knowledge, which could be in good companion to traditional RGB image stream. However, the introduction of an additional stream issue a different challenge; the significant need for a stream integration method . To tackle such a challenge, we propose an Incremental Multi-stream Intermediate Fusion framework (IMIF). The IMIF allows additional information streams to be effectively propagated during the incremental learning phase. In the following subsections, we briefly define the multi-stream class incremental learning method and describe how it can be decomposed into different components.

\subsection{Multi-stream Class Incremental Learning Model}

We define a multi-stream class incremental learning model \textbf{M} as a combination of three key components: 1) a single stream base method \textbf{X}, 2) an additional stream of information \textbf{Y}, and 3) a method of fusing stream \textbf{Z}.
\begin{center}
    \textbf{M} = Base method \textbf{X} + Feature stream \textbf{Y} + Fusion mechanism \textbf{Z}
\end{center}
At this point, the base method represents any method that follows the general setting described in Sec.\ref{subsec:conventionalCIL}. \textbf{Y} serves as a piece of additional domain information that gives cues to the learning model apart from RGB images.
Normally, \textbf{Y} is specific to the domain of the task. Lastly, \textbf{Z} presents a fusion mechanism that enables method \textbf{M} to incorporate additional information stream \textbf{Y} into the incremental learning process. From this decomposition, our CG-IMIF replaces: 1) the representative stream \textbf{Y} with color-specific information, and 2) the fusion technique \textbf{Z} with the proposed IMIF.
In the following, we describe our proposed Color histogram guidance stream and Incremental Multi-stream Intermediate Fusion technique in Sec.\ref{ssec:domain-specific information extractor} and Sec.\ref{subsec:fusion}, respectively.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.5\textwidth]{images/ColorHistogram.pdf}
    \caption{Samples of pill images from VAIPE-PCIL dataset are shown on the first row. The second row exhibits the corresponding color histogram information for images on the upper row respectively.}
    \label{fig:example_color_histogram}
\end{figure}

\subsection{Color Histogram Guidance Stream} \label{ssec:domain-specific information extractor}

Pill images compose of various features which can be used as discriminative factors in classification problems. Among those is color distribution information which can be approximated by the color histogram of pill images. The color histogram represents the color distribution of the input image in terms of three different channels: red, green, and blue. In detail, it is often encoded into a single vector where the indexes of entries are mapped to a set of all possible colors. In grey-scale images, values of vector entries store the frequency that counts the total number of pixels having the intensity (\textit{i.e.}, color value), which is hashed by the corresponding index of the vector. However, in a three-dimensional image, color ranges for each channel are associated across color channels to formulate a unique combination. This combination accounts for those pixels of which color ranges lie in three discrete ranges of value: $[r_i, r_j];[g_k, g_l];[b_m, b_n]$. The color feature vectors are useful to make a distinction among pill instances that have similar shapes but different colors. The color ranges for each channel of the RGB images are divided into eight segments in our problem, where each segment represents 32 different consecutive color values. After that, the color histogram vector can be obtained by accumulating the quantities of pixels assigned to a specific color range for each channel to result in a vector with 8$\times$8$\times$8 = 512 elements. Fig.~\ref{fig:example_color_histogram} illustrates a few examples of extracting the color histogram stream for each corresponding cropped pill image.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.75\textwidth]{images/traditional_single_stream.pdf}
    \caption{The traditional training paradigm with only single-stream information used by almost existing exemplar-based CIL methods.}
    \label{fig:traditional_single_stream}
\end{figure}

\subsection{Incremental Multi-stream Intermediate Fusion Technique}
\label{subsec:fusion}

\textbf{Traditional Early Fusion}
A naive fusion technique \textbf{Z} concatenates different streams of information right after the feature extraction phase. Specifically, feature vectors $f_{r} \in \mathbb{R}^{d_r}$, and $f_Y \in \mathbb{R}^{d_Y}$ are extracted from raw RGB images, and additional information stream \textbf{Y}, respectively. Both of these features are then fed into separate projection layers to project into the same latent space. In practice, the projection layers are implemented by a single hidden layer controlled by parameters $\Theta^p=[W^p, b^p]$ as follows
\begin{equation}
s_{r} = \sigma(W^p_r.f_r+b^p_r),\\  s_{Y} =\sigma(W^p_Y.f_Y+b^p_Y), \\  s_{g} = [s_r, s_Y].
\label{eq:projection_layer}
\end{equation}
The projected vectors are then concatenated to obtain the global feature vector $s_g \in \mathbb{R}^{d_g}$. This global feature $s_g$ can be considered as a single input into any traditional single stream CIL methods as shown in Fig.~\ref{fig:traditional_single_stream}.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.75\textwidth]{images/X-CG-IMIF.pdf}
    \caption{Our proposed CG-IMIF architecture composes of: 1) color histogram feature extraction (orange block), and 2) intermediate fusion framework (purple block) to incorporate additional information stream.}
    \label{fig:our_X_CG_IMIF}
\end{figure}
\noindent\textbf{Intermediate Fusion}
We observed that fusing information stream in an early manner for a class incremental learning problem is not optimal. The global feature $s_g$ is problematic since it is regularly updated at each incremental task. As a result, the projection layer in the early phase can not find good parameters that balance the performance of old and novel categories in different tasks. Therefore, we propose to relocate the projection layer to the intermediate phase instead (\textit{i.e}, incremental learning phase) by initiating an entirely novel projection layer in an incremental manner. When $C^t$ from new task $t$ arrives at the system, a new classification layer $CL_t$ is created . This layer accompanies by an attached projection layer specific to the information stream of a specific task. Therefore, the parameters controlling the projection layer for each information stream are different from those defined in the early fusion.
\begin{equation}
\Theta^{p^t}_{r} = [W^{p^t}_r, b^{p^t}_r],
\Theta^{p^t}_{Y} = [W^{p^t}_Y, b^{p^t}_Y].
\end{equation}

\section{Experiments}
\subsection{Dataset}

We employ a real-world image dataset, namely VAIPE-Pill (VAIPE Pill Identification) \cite{VAIPE} to exploit CIL capability on the pill image classification problem. This dataset is created to promote the research on recognizing distinct types of medicines from mobile devices. The dataset contains 7,294 pill images of 262 categories taken in real-world scenarios. The characteristics of VAIPE-Pill dataset are illustrated in Tab.~\ref{tab:data_stat}.

To facilitate research of CIL in pill image classification tasks, we derive a dataset version, namely VAIPE-PCIL (VAIPE Pill Class Incremental Learning) dataset from the original VAIPE-Pill data. VAIPE-PCIL is obtained by cropping pill instances from the original data. We only select those categories which satisfy either of the following conditions: 1) the number of samples should not be too small (\textit{i.e.}, and 2) larger than 10 samples), image size of samples should be at least $64 \times 64$. Samples of pill image from VAIPE-PCIL can be found in Fig.~\ref{fig:overview}. All of our experiments are conducted on the VAIPE-PCIL dataset to study the performance of CG-IMIF.

\begin{table}
    \centering
    \caption{Statistics of VAIPE-Pill dataset on different characteristics.}
    {
\begin{tabular}{c|c|c|c}
\hline \textbf{Characteristic} & \textbf{Training set} & \textbf{Testing set} & \textbf{Total} \\
\hline Number of images & 6,461 & 833 & 7,294 \\
Number of pill categories & 262 & 262 & 262 \\
Instances per category & $179.75$ & $23.56$ & $203.2$ \\
Image size (pixel$\times$pixel, mean) & $3,311 \times 3,276$ & $3,276 \times 3,469$ & $3,300 \times 3,400$ \\
Instances per image & $7.28$ & $7.4$ & $7.3$ \\
Number of bounding box annotations & 47,097 & 6,174 & 53,271 \\
Number of categories per image & $5.18$ & $5.76$ & $5.32$\\
\hline
\end{tabular} }
\label{tab:data_stat}
\end{table}

\subsection{Experimental Protocol}

\textbf{Settings} We follow the standard benchmark protocol proposed in \cite{Hou_2019_CVPR}. We fix class arrangements in random order.
After each training stage $t$, the resulting learner is evaluated on the testing data $\bigcup_{i=1}^{t}P_{test}^t$ which represents for all of the testing data up to the current task $t$. Since no test data from the previous learning stage are hidden from the learner, it is guaranteed that no overfitting can occur.

There are two commonly different task evaluation settings in class incremental learning: task-awareness and task-agnostic. The first setting is much easier for the algorithm since it has access to the task ID (\textit{i.e.}, ID or set of categories) about the incoming test data. Therefore, it is reasonable to only use the corresponding classification head in the incremental learning phase, which is trained on that task-ID to evaluate the performance. This task setting, however, is not practical in many real-world circumstances since task-ID is not always available. We evaluate our performance in terms of task-agnostic instead. In task-agnostic, the model is not given the task identities of the test data. Hence, the evaluation results are achieved by taking the results of all prediction logits, which are predicted by all of the classification heads. In this way, the model has to learn to resolve the confusion among classes from a different set of classes.

 \noindent\textbf{Evaluation Metrics} We adopt two commonly used benchmark metrics from \cite{Hou_2019_CVPR} for CIL problems: average accuracy and average forgetting rate. The average accuracy and forgetting rate records of performance for each incremental learning phase are often if a single number is preferable. Meanwhile, the average phase accuracy and forgetting rate would be used to observe learning behaviors during incremental tasks for each method.
\subsection{Implementation Details}

All of our experiments and methods are implemented with Pytorch \cite{NEURIPS2019_9015} and trained on a single NVIDIA GeForce RTX 3090. We inherit the codebase from FACIL \cite{masana2020class}. They have already implemented various state-of-the-art methods for CIL problems in a well-structured manner. Details of base models as well as the implementation of our IMIF framework are discussed below. In all experiments, we attach our IMIF framework to several state-of-the-art methods in CIL: BiC\cite{DBLP:journals/corr/abs-1905-13260}, EEIL\cite{DBLP:journals/corr/abs-1807-09536}, and LUCIR\cite{Hou_2019_CVPR}. Since these methods followed the common prototype of exemplar-based methods, we discussed some of the general settings of exemplar-based methods in our experiments before diving into details about the setting of each base one. There are two common strategies to reserve samples for old classes: 1) exemplar-management stores a constant number of samples for each old class, or 2) it maintains a fixed capacity (\textit{e.g}, $R_{total} = 2,000$ for CIFAR-100 \cite{cifar} and $R_{total} = 20,000$ for ImageNet \cite{deng2009imagenet}). In our experiments, we follow the first setting since it is usually more challenged than the second one. In addition, the exemplars are randomly selected among different categories. For the training network, we made use of 50-layer ResNet \cite{DBLP:journals/corr/HeZRS15} with no pre-trained weights as the backbone for feature extraction module, which is applicable to the VAIPE-PCIL dataset. To fairly compare the performance, we fixed the number of training epochs (200 epochs) across different methods. The learning rate is initialized with 0.1 and is divided by 1.5 if the loss function suffers from non-decreasing circumstances for a specific number of attempts (\textit{e.g}, $lr_{patience} = 5$). The networks are trained using stochastic gradient descent with mini-batches of 32 samples. The training images are resized to the same shape of $256 \times 256 \times 3$ with only one transformation (\textit{e.g}, flipping). The class orders across different methods are randomly fixed for a fair comparison.

In terms of configuration for base methods, we follow the same settings for the original version BiC \cite{DBLP:journals/corr/abs-1905-13260}, and our improved version BiC-CG-IMIF. BiC\cite{DBLP:journals/corr/abs-1905-13260} proposed to integrate a bias correction layer attached to the end of each classification head to adjust the classification score. The number of training epochs for the bias correction layer is 200 epochs in our setting. Moreover, we set 0.1 as the ratio of the number of exemplars that are used for the validation. EEIL \cite{DBLP:journals/corr/abs-1807-09536} performs an additional fine-tuning phase after each official training phase to balance the performance between old and novel categories. In our experiments, we fix 40 as the number of epochs for fine-tuning and the learning rate fine-tuning factor as 0.01 across different methods. We also adhere to the base setting of LUCIR\cite{Hou_2019_CVPR} method where they removed ReLU in the penultimate layer to take both positive and negative values.
For the IMIF framework, the projection layer implemented is represented by a single hidden layer. Therefore, the output size for different projection layers should be the same so that the transformed feature vectors, then can be fused in the shared space. In terms of the color-guided information, color ranges for each channel of the RGB images are divided into 8 segments where each segment represents 32 different consecutive pixel values.

\subsection{Experimental Results}
\label{subsec:exp_result}
\begin{table*}[htbp]
  \small
  \centering
  \caption{
  Average accuracy $\bar{\mathcal{A}}$ (\%) and forgetting rate $\mathcal{F}$ (\%) of CG-IMIF compared to other state-of-the-art results in different task settings. Best scores are marked in bold for both evaluation metrics.
  }
  {
  \begin{tabular}{llccccccccccccccc}
  \toprule
  \multirow{2.5}{*}{Metric} & \multirow{2.5}{*}{Method} & \multicolumn{3}{c}{\emph{Task Settings}} \\
  \cmidrule{3-5}
  & & $N$=5 & $N$=10  & $N$=15 \\
   \midrule
  & EEIL\cite{DBLP:journals/corr/abs-1807-09536} & 63.83 & 62.40 & 57.41 &\\
  & \textbf{EEIL-CG-IMIF} & \textbf{\textbf{70.80}} & \textbf{\textbf{64.85}} & \textbf{\textbf{60.93}} \\
  \cmidrule{2-13}
  \textbf{\emph{Average acc.} (\%) $\uparrow$} & BiC \cite{DBLP:journals/corr/abs-1905-13260} & 53.83 & 55.75 & 53.77  \\
  {$\bar{\mathcal{A}}=\frac{1}{n}\sum_{i=1}^{n}\mathcal{A}_i$}& \textbf{BiC-CG-IMIF} & \textbf{\textbf{65.53}} & \textbf{\textbf{63.59}} & \textbf{\textbf{54.83}}\\
  \cmidrule{2-13}
    \multirow{1}{*} & LUCIR \cite{Hou_2019_CVPR}  & 69.63 & 62.90 & 55.49\\
  & \textbf{LUCIR-CG-IMIF} & {\textbf{76.85}} & {\textbf{69.94}} & {\textbf{64.97}} \\
  \midrule
   & EEIL \cite{DBLP:journals/corr/abs-1807-09536} & 49.82 & 45.46 & 48.27 \\
  & \textbf{EEIL-CG-IMIF } & \textbf{\textbf{46.68}} & \textbf{\textbf{44.64}} & \textbf{\textbf{46.23}} \\
  \cmidrule{2-13}
  \textbf{\emph{Forgetting rate}. (\%) $\downarrow$} & BiC \cite{DBLP:journals/corr/abs-1905-13260} & 20.05 & 30.50 & 26.93  \\
  \multirow{1}{*}{$\bar{\mathcal{F}}=\frac{1}{n}\sum_{i=1}^{n}\mathcal{F}_i$} & \textbf{BiC-CG-IMIF } & \textbf{\textbf{7.75}} & \textbf{\textbf{22.01}} & \textbf{27.35} \\
  \cmidrule{2-13}
    \multirow{1}{*} & LUCIR \cite{Hou_2019_CVPR}  & 44.13 & 44.32 & 47.11\\
  & \textbf{LUCIR-CG-IMIF } & {\textbf{33.15}} & {\textbf{37.88}} &  {\textbf{39.79}} \\
  \bottomrule
    \multicolumn{13}{l}{${}^{\diamond}$ Using the similar exemplar settings and selection for fair comparison.}
\end{tabular}
}

  \label{tab:experimental_result}

\end{table*}
We evaluate our proposed CG-IMIF approach and report the overall performance in comparison with several state-of-the-art approaches in Tab.\ref{tab:experimental_result} Experimental results show that most of the state-of-the-art approaches attached with our proposed IMIF tool and color-specific information as additional stream help to achieve consistent improvements over task settings.
The setting consists of three tasks in total where the number of categories is uniformly distributed for 5, 10, and 15 tasks. It is noticeable that the lower score of forgetting rate indicates that the model is more unlikely to forget about old knowledge. In addition, average phase accuracy and forgetting rate is also illustrated in Fig.~\ref{fig:average_phase_acc},\ref{fig:average_forgetting_rate} to inspect the learning behaviour of each method through incremental phases. Dashed and solid lines with different colors are utilized to differentiate the base ones (X) and our CG-IMIF, respectively. In terms of the average phase accuracy, LUCIR-CG-IMIF obtains the highest performance where it can consistently and significantly surpass other methods (also the base one-LUCIR). On the other hand, BiC-CG-IMIF is better at mitigating the forgetting constraint. However, it is not consistent over tasks and the curve fluctuates. One possible explanation is that the bias layer inside the traditional BiC method and BiC-CG-IMIF might cause the model to sacrifice the performance of the current task to maintain the memory of the old ones.
\begin{figure*}[htbp]
	\centering
	{
    \includegraphics[width=\textwidth]{images/legend.pdf}
		\begin{tabular}[width=\textwidth]{ccc}
			\subfigure[5 tasks] {\includegraphics[height=25mm]{images/paper_average_acc_5_tasks.pdf}}  &
			\subfigure[10 tasks] {\includegraphics[height=25mm]{images/paper_average_acc_10_tasks.pdf}}   &
			\subfigure[15 tasks] {\includegraphics[height=25mm]{images/paper_average_acc_15_tasks.pdf}}
		\end{tabular}
		\caption{\textbf{Incremental accuracy} for different task settings among the original version and our method CG-IMIF.}
		\label{fig:average_phase_acc}
	}
	\centering
	{
		\begin{tabular}[width=\textwidth]{cccc}
			\subfigure[5 tasks] {\includegraphics[height=25mm]{images/paper_forg_acc_5_tasks.pdf}}  &
			\subfigure[10 tasks] {\includegraphics[height=25mm]{images/paper_forg_acc_10_tasks.pdf}}   &
			\subfigure[15 tasks] {\includegraphics[height=25mm]{images/paper_forg_acc_15_tasks.pdf}}
		\end{tabular}
		\caption{\textbf{Incremental forgetting rate} for different task settings among the original version and our method CG-IMIF.}
		\label{fig:average_forgetting_rate}
	}
\end{figure*}
\section{Ablation Studies}
To examine the effect of additional information stream usage and fusion framework, we perform extensive ablation studies. This is aim to observe the effect of different components in our proposed framework where LUCIR is chosen as the base method. LUCIR is preferable because of the consistency and high performance of LUCIR across task settings in the experimental results which have been discussed in Sec. \ref{subsec:exp_result}.
In addition to color information, edge signals might be a good candidate to discriminate different pill categories based on their shape. To understand the importance of different stream usage in our method, we compare 4 different settings: 1) RGB image only, 2) RGB and edge images, 3) RGB and color histogram, and 4) a combination of all three streams. Each separated row in Tab.\ref{tab:ablation_study}  refers to each scenario of information stream usage with two different fusion techniques. Concretely, the setting that combines RGB and color histogram streams achieves the highest score. One possible explanation for this result is that the edge signal might not be sufficiently strong to push the performance.

In addition, we implement the basic fusion technique where additional information streams are fused in an early manner. Each separated row in Tab.\ref{tab:ablation_study} illustrates the results of two different fusion mechanisms. Our fusion technique (on the second line of each row) outperforms the traditional one in various metrics and task settings. The best result is LUCIR-CG-IMIF which integrates color histogram information into the traditional LUCIR method with IMIF.

\section{Discussions}
\begin{table*}
  \small
  \centering
  \caption{
  Ablation performance to compare variants of combination which utilize different information streams as well as different fusion techniques. The Combination which achieves highest performance over different tasks is our CG-IMIF and is marked in green.
  }
  {
  \begin{tabular}{llccccccccccccc}
  \toprule
  \multirow{2.5}{*}{Variant of Combination} & \multicolumn{3}{c}{\emph{Average acc.} (\%) $\uparrow$} && \multicolumn{3}{c}{\emph{Forgetting rate.} (\%) $\downarrow$} \\
  \cmidrule{2-4} \cmidrule{6-8}
  & $N$=5 & $N$=10  & $N$=15 && $N$=5 & $N$=10  & $N$=15 \\
  \cmidrule{1-8}
  RGB only & 69.93 & 62.90 & 55.49 && 44.13 & 44.32 & 47.1\\
  \cmidrule{1-8}
  RGB-\textcolor{blue}{Edge} + \textcolor{red}{Early} & 70.94 & 63.90 & 55.28 && 42.4 & 42.13 & 45.80  \\
  RGB-\textcolor{blue}{Edge} + \textcolor{orange}{Intermediate} & \textbf{72.58} & \textbf{68.38} & \textbf{62.90} && \textbf{38.78} & \textbf{38.19} & \textbf{41.02}  \\
  \cmidrule{1-8}
     RGB-\textcolor{cyan}{Color} + \textcolor{red}{Early} & 73.58 & 64.57 & 53.56 && 37.825 & 42.86 & 46.15   \\
  RGB-\textcolor{cyan}{Color}+ \textcolor{orange}{Intermediate} & \textcolor{ForestGreen}{76.85} & \textcolor{ForestGreen}{69.94} & \textcolor{ForestGreen}{64.97} && \textcolor{ForestGreen}{33.15} & \textcolor{ForestGreen}{37.88} & \textcolor{ForestGreen}{39.79}  \\
  \cmidrule{1-8}
  RGB-\textcolor{blue}{Edge}-\textcolor{cyan}{Color} + \textcolor{red}{Early} & 69.99 & 63.33 & 56.34 && 42.35 & 44.17 & 46.24   \\
  RGB-\textcolor{blue}{Edge}-\textcolor{cyan}{Color}+ \textcolor{orange}{Intermediate} & \textbf{73.65} & \textbf{68.32} & \textbf{62.15} && \textbf{36.30} & \textbf{38.48} & \textbf{40.58} \\
  \bottomrule
\end{tabular}
}
  \label{tab:ablation_study}
\end{table*}
\textbf{Key Findings}. To the best of our knowledge, this work is the first to tackle the class incremental learning problem for the pill image domain, which is crucial and applicable for real-world pill recognition systems. Also, we empirically showed that the technique of intermediate fusion with the additional stream is superior to the early fusion technique. One plausible explanation for this effect is the flexibility of the fusion layer after it has been relocated to the intermediate stage. This allows the additional information to maintain its optimal performance for old tasks while learning to adapt to new tasks.\\
\\
\noindent \textbf{Limitations}. Though the proposed framework has superior performance over several state-of-the-art methods in CIL, it contains some limitations in different aspects. The new fusion layer at the intermediate phase might enlarge the model's size in terms of the number of parameters. Considering the scenario when a massive amount of tasks are encountered in the learning progress, the learning model could create sequences of abundant layers. This might create a side effect when too much memory is reserved for storing the model's parameters. Such reservation is unreasonable in a real-world deployment. Another restriction with the proposed framework is related to additional stream utilization. Apart from the traditional RGB stream, another information channel that is specific to the domain of usage might impose a disagreement with the original RGB channel. This requires a careful study of a different combination of streams accompanying the traditional stream to observe its effect.

\section{Conclusion}
This paper introduces the incremental learning capability to the traditional pill image classification systems. To this end, we propose a novel framework, namely Incremental Multi-stream Intermediate Fusion (IMIF) which integrates an additional stream of information to improve the performance of the single stream CIL method. We then devise CG-IMIF which utilizes IMIF along with a color histogram as guidance information. Our CG-IMIF is flexible and can be attached to any exemplar-based approach to improve the performance of the base ones. We experimentally show that CG-IMIF outperforms many existing state-of-the-art methods on the VAIPE-PCIL dataset. We hope our work would lay the foundation and could benefit several types of future research into the continual learning ability of intelligent machines in smart health applications.\\
\\
\textbf{Acknowledgements}. This work was funded by Vingroup Joint Stock Company (Vingroup JSC),Vingroup, and supported by Vingroup Innovation Foundation (VINIF)
under
project code VINIF.2021.DA00128. Trong-Tung Nguyen was funded and and supported by the Master, PhD Scholarship Programme/Post-Doctoral Scholarship Programme of Vingroup Innovation Foundation (VINIF), Vingroup Big Data Institute (VinBigdata), code VINIF.2021.ThS.JVN.07.
\bibliographystyle{splncs04}