

point out  the typora .

After Transformer released, pretrained models spurt. GPT\cite{GPT} and BERT\cite{BERT} reach SOTA in NLG and NLU tasks.  A typical NLP pipeline is pretraining a big model from scratch or initialed on public corpus as first, then posttrain on custom domain corpus, and finetune downstream task (NLU, NLU and text representation) on labeled dataset in final. Some work try to simplify the pipeline. Unifying NLP tasks as text to text achieves good performance.  The main factor of these progress is increasing model size. Even big models are fewshot or zeroshot learners \cite{GPT3}. 

Some works \cite{2020arXiv200410964G} show that it is still helpful to tailor a pretrained model to the domain of a target task. \cite{TLM} uses task data as queries to retrieve a tiny subset of the general corpus and jointly optimizes the task objective and the language modeling objective from scratch.

Prompt \cite{P3} avoid fine-tuning the big model and use the prompts to utilize pretrained knowledge. Train a large model \cite{Compress}, then distil the big model to a small model for inference. If a sample is easy enough, Elastic \t{Elastic} jump out early to save predict time.

\subsection{Retrieval}
Retrieval seems a complex system in usual. BM25 \cite{BM25} is a strong and simple sparse retrieval model. In recent, pretraining representation model and pretraining interaction model are introduced into dense retrieval models \cite{2021arXiv211113853F}.  Some works dig similar samples, such as Inverse Cloze Task (ICT) \cite{ICT} and learn representation by contrastive learning  \cite{Spider}. 

\subsection{Representation}
The goal of representation learning \cite{Representation} is to learn an embedding space in which similar examples stay close to each other while dissimilar ones are far apart \cite{Dimensionality}. In contrastive learning, the learning procedure is formulated as a classification problem given similar and dissimilar candidates. 

Contrastive learning can be applied in both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning. \cite{CLIP} use in-batch negative samples.  \cite{ConSERT}, \cite{SimCSE} provide text argument method.  Some work \cite{seonwoo2023rankingenhanced}, \cite{liu2023rankcse} use complex methods capturing a more accurate semantic similarity between similar sentences.


 Please use JSON format in your answer.
user