\section{Introduction}

In the past decade, goal-oriented spoken dialogue systems (SDS), such as the virtual personal assistants Microsoft's Cortana and Apple's Siri, are being incorporated in various devices and allow users to speak to systems freely in order to finish tasks more efficiently.
A key component of these conversational systems is the natural language understanding (NLU) module—-it refers to the targeted understanding of human speech directed at machines~\cite{tur2011spoken}.
The goal of such ``targeted'' understanding is to convert the recognized user speech into a task-specific semantic representation of the user's intention, at each turn, that aligns with the back-end knowledge and action sources for task completion.
The dialogue manager then interprets the semantics of the user's request and associated back-end results, and decides the most appropriate system action, by exploiting semantic context and user specific meta-information, such as geo-location and personal preferences~\cite{mctear2004spoken,rudnicky1999agenda}.

A typical pipeline of NLU includes:
domain classification, intent determination, and slot filling~\cite{tur2011spoken}.
NLU first decides the domain of user's request given the input utterance, and based on the domain, predicts the intent and fills associated slots corresponding to a domain-specific semantic template.
For example, Figure~\ref{fig:iob} shows a user utterance, ``\textit{show me the flights from seattle to san francisco}'' and its semantic frame, \textsf{find\_flight(origin=``seattle'', dest=``san francisco'')}.
It is easy to see the relationship between the origin city and the destination city in this example, although these do not appear next to each other.
Traditionally, domain detection and intent prediction are framed as utterance classification problems, where several classifiers such as support vector machines and maximum entropy have been employed~\cite{haffner2003optimizing,chelba2003speech,chen2014deriving}.
Then slot filling is framed as a word sequence tagging task, where the IOB (in-out-begin) format is applied for representing slot tags as illustrated in Figure~\ref{fig:iob}, and hidden Markov models (HMM) or conditional random fields (CRF) have been employed for slot tagging~\cite{pieraccini1992speech,wang2005spoken}.

\begin{figure}[t]
	\centering
    \includegraphics[width=0.9\linewidth]{iob.pdf}
    \vspace{-2mm}
    \caption{An example utterance annotated with its semantic slots in the IOB format (S).}
\vspace{-4mm}
    \label{fig:iob}
\end{figure}

With the advances on deep learning, deep belief networks (DBNs) with deep neural networks (DNNs) have been applied to domain and intent classification tasks~\cite{sarikaya2011deep,tur2012towards,sarikaya2014application}.
Recently, \newcite{ravuri2015recurrent} proposed an RNN architecture for intent determination.
For slot filling, deep learning has been viewed as a feature generator and the neural architecture can be merged with CRFs~\cite{xu2013convolutional}.
\newcite{yao2013recurrent} and \newcite{mesnil2015using} later employed RNNs for sequence labeling in order to perform slot filling.
However, the above studies benefit from large training data without leveraging any existing knowledge.
When tagging sequences RNNs consider them as flat structures, with their underlying linear chain structures, potentially ignoring the structured information typical of natural language sequences.

Hierarchical structures and semantic relationships contain linguistic characteristics of input word sequences forming sentences, and such information may help interpret their meaning.
Furthermore, prior knowledge would help in the tagging of sequences, especially when dealing with previously unseen sequences~\cite{tur2010left,deoras2013deep}.
Prior work exploited external web-scale knowledge graphs such as Freebase and Wikipedia for improving NLU~\cite{heck2013leveraging,ma2015knowledge,chen2014deriving}
\newcite{liu2013query} and \newcite{chen2015matrix} proposed approaches that leverage linguistic knowledge encoded in parse trees for language understanding, where the extracted syntactic structural features and semantic dependency features enhance inference model learning, and the model achieves better language understanding performance in various domains.

Even with the emerging paradigm of integrating deep learning and linguistic knowledge for different NLP tasks~\cite{socher2014grounded}, most of the previous work utilized such linguistic knowledge and knowledge bases as additional features as input to neural networks, and then learned the models for tagging sequences.
These feature enrichment based approaches have some possible limitations:
1) poor generalization and 2) error propagation.
Poor generalization comes from the mismatch between knowledge bases and the input data, and then the incorrectly extracted features due to errors in previous processing propagate errors to the neural models.
In order to address the issues and better learn the sequence tagging models, this paper proposes knowledge-guided structural attention networks, K-SAN, a generalization of RNNs that automatically learn the attention guided by external or prior knowledge and generate sentence-based representations specifically for modeling sequence tagging.
The main difference between K-SAN and previous approaches is that knowledge plays the role of a teacher to guide networks where and how much to focus attention considering the whole linguistic structure simultaneously.
Our main contributions are three-fold:
\begin{itemize}
\vspace{-1.5mm}
\item End-to-end learning\\
To our knowledge, this is the first neural network approach that utilizes general knowledge as guidance in an end-to-end fashion, where the model automatically learns important substructures with an attention mechanism.
\vspace{-1.5mm}
\item Generalization for different knowledge\\
There is no required schema of knowledge, and different types of parsing results, such as dependency relations, knowledge graph-specific relations, and parsing output of hand-crafted grammars, can serve as the knowledge guidance in this model.
\vspace{-1.5mm}
\item Efficiency and parallelizability\\
Because the substructures from the input utterance are modeled separately, modeling time may not increase linearly with respect to the number of words in the input sentence.
\vspace{-1.5mm}
\end{itemize}
In the following sections, we empirically show the benefit of K-SAN on the targeted NLU task.

\section{Related Work}
\label{sec:related}

\paragraph{Knowledge-Based Representations}

There is an emerging trend of learning representations at different levels, such as word embeddings~\cite{mikolov2013distributed}, character embeddings~\cite{ling2015finding}, and sentence embeddings~\cite{le2014distributed,huang2013learning}.
In addition to fully unsupervised embedding learning,
knowledge bases have been widely utilized to learn entity embeddings with specific functions or relations~\cite{celikyilmaz2015unsupervised,yang2014embedding}.
Different from prior work, this paper focuses on learning composable substructure embeddings that are informative for understanding.

Recently linguistic structures are taken into account in the deep learning framework.
\newcite{ma2015dependency} and \newcite{tai2015improved} both proposed dependency-based approaches to combine deep learning and linguistic structures, where the model used tree-based n-grams instead of surface ones to capture knowledge-guided relations for sentence modeling and classification.
\newcite{roth2016neural} utilized lexicalized dependency paths to learn embedding representations for semantic role labeling.
However, the performance of these approaches highly depends on the quality of ``whole'' sentence parsing, and there is no control of degree of attentions on different substructures.
Learning robust representations incorporating whole structures still remains unsolved.
In this paper, we address the limitation by proposing K-SAN to learn robust representations of whole sentences, where the whole representation is composed of the salient substructures in order to avoid error propagation.

\paragraph{Neural Attention and Memory Model}

One of the earliest work with a memory component applied to language processing is memory networks~\cite{weston2015memory,sukhbaatar2015end}, which encode facts into vectors and store them in the memory for question answering (QA).
Following their success, \newcite{xiong2016dynamic} proposed dynamic memory networks (DMN) to additionally capture position and temporality of transitive reasoning steps for different QA tasks.
The idea is to encode important knowledge and store it into memory for future usage with attention mechanisms.
Attention mechanisms allow neural network models to selectively pay attention to specific parts.
There are also various tasks showing the effectiveness of attention mechanisms.

However, most previous work focused on the classification or prediction tasks (predicting a single word given a question), and there are few studies for NLU tasks (slot tagging).
Based on the fact that the linguistic or knowledge-based substructures can be treated as prior knowledge to benefit language understanding, this work borrows the idea from memory models to improve NLU.
Unlike the prior NLU work that utilized representations learned from knowledge bases to enrich features of the current sentence, this paper directly learns a sentence representation incorporating memorized substructures with an automatically decided attention mechanism in an end-to-end manner.

\section{Knowledge-Guided Structural Attention Networks (K-SAN)}
\label{sec:model}

For the NLU task, given an utterance with a sequence of words/tokens $\vec{s} = w_1, ..., w_T$, our model is to predict corresponding semantic tags $\vec{y} = y_1, ..., y_T$ for each word/token by incorporating knowledge-guided structures.
The proposed model is illustrated in Figure~\ref{fig:model}.
The knowledge encoding module first leverages external knowledge to generate a linguistic structure for the utterance, where a discrete set of knowledge-guided substructures $\{x_i\}$ is encoded into a set of vector representations (\ref{ssec:kg_encoding}).
The model learns the representation for the whole sentence by paying different attention on the substructures (\ref{ssec:arch}).
Then the learned vector encoding the knowledge-guided structure is used for improving the semantic tagger (\ref{sec:tagger}).

\begin{figure*}[t]
\centering
\vspace{-2mm}
\includegraphics[width=\linewidth]{model.pdf}
\vspace{-5mm}
\caption{The illustration of knowledge-guided structural attention networks (K-SAN) for NLU.}
\label{fig:model}
\vspace{-2mm}
\end{figure*}

\subsection{Knowledge Encoding Module}
\label{ssec:kg_encoding}

\begin{figure}[t]
\centering
\vspace{-2mm}
\includegraphics[width=\linewidth]{dep_tree.pdf}
\vspace{-4mm}
\caption{The knowledge-guided substructures of dependency parsing, $x_i$, on an example sentence $s$.}
\vspace{-2mm}
\label{fig:tree}
\end{figure}

The prior knowledge obtained from external resources, such as dependency relations, knowledge bases, etc., provides richer information to help decide the semantic tags given an input utterance.
This paper takes dependency relations as an example for knowledge encoding, and other structured relations can be applied in the same way.
The input utterance is parsed by a dependency parser, and the substructures are built according to the paths from the root to all leaves~\cite{chen2014fast}.
For example, the dependency parsing of the utterance ``{\it show me the flights from seattle to san francisco}'' is shown in Figure~\ref{fig:tree}, where the associated substructures are obtained from the parsing tree for knowledge encoding.
Here we do not utilize the dependency relation labels in the experiments for better generalization, because the labels may not be always available for different knowledge resources.
Note that the number of substructures may be less than the number of words in the utterance, because non-leaf nodes do not have corresponding substructure in order to reduce the duplicated information in the model.
The top-left component of Figure~\ref{fig:model} illustrates the module for modeling knowledge-guided substructures.

\subsection{Model Architecture}
\label{ssec:arch}
The model embeds all knowledge-guided substructures into a continuous space and stores embeddings of all $x$'s in the knowledge memory.
The representation of the input utterance is then compared with encoded knowledge representations to integrate the carried structure guided by knowledge via an attention mechanism.
Then the knowledge-guided representation of the sentence is taken together with the word sequence for estimating the semantic tags.
Four main procedures are described below.

\paragraph{Encoded Knowledge Representation}
To store the knowledge-guided structure, we convert each substructure (e.g. path starting from the root to the leaf in the dependency tree), $x_i$, into a structure vector $m_i$ with dimension $d$ by embedding the substructure in a continuous space through the knowledge encoding model $M_\text{kg}$.
The input utterance $s$ is also embedded to a vector $u$ with the same dimension through the model $M_\text{in}$.
\begin{eqnarray}
m_i &=& M_\text{kg}(x_i),\\
u &=& M_\text{in}(s).
\end{eqnarray}
We apply the three types for knowledge encoding models, $M_\text{kg}$ and $M_\text{in}$, in order to model multiple words from a substructure $x_i$ or an input sentence $s$ into a vector representation:
1) fully-connected neural networks (NN) with linear activation,
2) recurrent neural networks (RNN),
and 3) convolutional neural networks (CNN) with a window size 3 and a max-pooling operation.
For example, one of substructures shown in Figure~\ref{fig:tree}, ``{\it show flights seattle from}'', is encoded into a vector embedding.
In the experiments, the weights of $M_\text{kg}$ and $M_\text{in}$ are tied together based on their consistent ability of sequence encoding.

\paragraph{Knowledge Attention Distribution}
In the embedding space, we compute the match between the current utterance vector $u$ and its substructure vector $m_i$ by taking their inner product followed by a softmax.
\begin{equation}
p_i = \text{softmax}(u^T m_i),
\end{equation}
where $\text{softmax}(z_i) = e^{z_i} / \sum_j e^{z_j}$ and $p_i$ can be viewed as attention distribution for modeling important substructures from external knowledge in order to understand the current utterance.

\paragraph{Sentence Representation}
In order to encode the knowledge-guided structure,
a vector $h$ is a sum over the encoded knowledge embeddings weighted by the attention distribution.
\begin{equation}
h = \sum_i p_i m_i,
\end{equation}
which indicates that the sentence pays different attention to different substructures guided from external knowledge.
Because the function from input to output is smooth, we can easily compute gradients and back propagate through it.
Then the sum of the substructure vector $h$ and the current input embedding $u$ are then passed through a neural network model $M_\text{out}$ to generate an output knowledge-guided representation $o$.
\begin{equation}
\label{eq:encoding}
o = M_\text{out}(h + u),
\end{equation}
where we employ a fully-connected dense network for $M_\text{out}$.

\paragraph{Sequence Tagging}
To estimate the tag sequence $\vec{y}$ corresponding to an input word sequence $\vec{s}$, we use an RNN module for training a slot tagger, where the knowledge-guided representation $o$ is fed into the input of the model in order to incorporate the structure information.
\begin{equation}
\vec{y} = \text{RNN}(o, \vec{s})
\end{equation}

\section{Recurrent Neural Network Tagger}
\label{sec:tagger}
\subsection{Chain-Based RNN Tagger}

Given $\vec{s} = w_1, ..., w_T$, the model is to predict $\vec{y} = y_1, ..., y_T$ where the tag $y_i$ is aligned with the word $w_i$.
We use the Elman RNN architecture, consisting of an input layer, a hidden layer, and an output layer~\cite{elman1990finding}.
The input, hidden and output layers consist of a set of neurons representing the input, hidden, and output at each time step $t$, $w_t$, $h_t$, and $y_t$, respectively.
\begin{eqnarray}
\label{eq:hidden}
h_t &=& \phi(W w_t + U h_{t-1}),\\
\label{eq:y}
\hat{y_t} &=& \text{softmax}(V h_t),
\end{eqnarray}
where $\phi$ is a smooth bounded function such as tanh, and $\hat{y_t}$ is the probability distribution over of semantic tags given the current hidden state $h_t$.
The sequence probability can be formulated as
\begin{equation}
\label{eq:obj}
p(\vec{y}\mid \vec{s}) = p(\vec{y}\mid w_1, ..., w_T) = \prod_i p(y_i\mid w_1, ..., w_i).
\end{equation}
The model can be trained using backpropagation to maximize the conditional likelihood of the training set labels.

To overcome the frequent vanishing gradients issue when modeling long-term dependencies, gated RNN was designed to use a more sophisticated activation function than a usual activation function, consisting of affine transformation followed by a simple element-wise nonlinearity by using gating units~\cite{chung2014empirical}, such as long short-term memory (LSTM) and gated recurrent unit (GRU)~\cite{hochreiter1997long,cho2014properties}.
RNNs employing either of these recurrent units have been shown to perform well in tasks that require capturing long-term dependencies~\cite{mesnil2015using,yao2014spoken,graves2013speech,sutskever2014sequence}.
In this paper, we use RNN with GRU cells to allow each recurrent unit to adaptively capture dependencies of different time scales~\cite{cho2014properties,chung2014empirical},
because RNN-GRU can yield comparable performance as RNN-LSTM with need of fewer parameters and less data for generalization~\cite{chung2014empirical}

A GRU has two gates, a \emph{reset gate} $r$, and an \emph{update gate} $z$~\cite{cho2014properties,chung2014empirical}.
The reset gate determines the combination between the new input and the previous memory, and the update gate decides how much the unit updates its activation, or content.
\begin{eqnarray}
\label{eq:reset}
r &=& \sigma( W^r w_t + U^r h_{t-1}),\\
\label{eq:update}
z &=& \sigma( W^z w_t + U^z h_{t-1}),
\end{eqnarray}
where $\sigma$ is a logistic sigmoid function.

Then the final activation of the GRU at time $t$, $h_t$, is a linear interpolation between the previous activation $h_{t-1}$ and the candidate activation $\tilde{h_t}$:
\begin{eqnarray}
\label{eq:candidate_h}
h_t &=& (1 - z) \odot \tilde{h_t} + z \odot h_{t-1}, \\
\tilde{h_t} &=& \phi( W^h w_t + U^h (h_{t-1}\odot r) )),
\end{eqnarray}
where $\odot$ is an element-wise multiplication.
When the reset gate is off,
it effectively makes the unit act as if it is reading the first symbol of an input sequence,
allowing it to forget the previously computed state.
Then $\hat{y_t}$ can be computed by (\ref{eq:y}).

\subsection{Knowledge-Guided RNN Tagger}
In order to model the encoded knowledge from previous turns, for each time step $t$, the knowledge-guided sentence representation $o$ in (\ref{eq:encoding}) is fed into the RNN model together with the word $w_t$.
For the plain RNN, the hidden layer can be formulated as
\begin{equation}
h_t = \phi(M o + W w_t + U h_{t-1})
\end{equation}
to replace (\ref{eq:hidden}) as illustrated in the right block of Figure~\ref{fig:model}.
RNN-GRU can incorporate the encoded knowledge in the similar way, where $M o$ can be added into gating mechanisms for modeling contextual knowledge similarly.

\begin{figure}[t]
\vspace{-2mm}
\centering
\includegraphics[width=\linewidth]{joint_tagger.pdf}
\vspace{-5mm}
\caption{The joint tagging model that incorporates a chain-based RNN tagger (upper block) and a knowledge-guided RNN tagger (lower block).}
\vspace{-3mm}
\label{fig:joint_tagger}
\end{figure}

\subsection{Joint RNN Tagger}
Because the chain-based tagger and the knowledge-guided tagger carry different information, the joint RNN tagger is proposed to balance the information between two model architectures.
Figure~\ref{fig:joint_tagger} presents the architecture of the joint RNN tagger.
\begin{eqnarray}
h_t^1 &=& \phi(W^1 w_t + U^1 h_{t-1}),\\
h_t^2 &=& \phi(M o + W^2 w_t + U^2 h_{t-1}),\\
\hat{y}_t &=& \text{softmax}(V(\alpha h_t^1 + (1 - \alpha) h_t^2)),
\end{eqnarray}
where $\alpha$ is the weight for balancing chain-based and knowledge-guided information.
By jointly considering chain-based information ($h_t^1$) and knowledge-guided information ($h_t^2$), the joint RNN tagger is expected to achieve better generalization, and the performance may be less sensitive to poor structures from external knowledge. In the experiments, $\alpha$ is set to $0.5$ for balancing two sides.
The objective of the proposed model is to maximize the sequence probability $p(\vec{y}\mid \vec{s})$ in (\ref{eq:obj}), and the model can be trained in an end-to-end manner, where the error would be back-propagated through the whole architecture.

\section{Experiments}
\label{sec:exp}

\begin{table*}[t]
    \centering
  \begin{tabular}{llclccc}
\hline
\multirow{2}{*}{} & \multicolumn{3}{c}{Model} & \multicolumn{3}{c}{Dataset} \\
\cline{2-7}
& Encoder ($M_\text{kg}$/$M_\text{in}$) & Knowledge & Tagger & Small & Medium & Large\\
\hline\hline
Baseline & - & \xmark & CRF & 58.94 & 78.74 & 89.73 \\
 & - & \xmark & RNN & 68.58 & 84.55 & 92.97 \\
& CNN & \xmark & RNN & 73.57 & 85.52 & 93.88\\
\hline
Structural & - & \cmark & CRF & 59.55 & 78.71 & 90.13\\
& DCNN & \cmark & RNN & 70.24 & 83.80 & 93.25\\
& Tree-RNN & \cmark & RNN & 73.50  & 83.92 & 92.28 \\
\hline
Proposed & K-SAN (NN) & \cmark & RNN & 74.11$^\dag$ & 85.97 & 93.98$^\dag$ \\
& K-SAN (RNN) & \cmark & RNN & 73.13 & 86.85$^\dag$ & \bf 94.97$^\dag$ \\
& K-SAN (CNN) & \cmark & RNN & \bf 74.60$^\dag$ & \bf 87.99$^\dag$ & 94.86$^\dag$ \\
\hline
  \end{tabular}
  \caption{The F1 scores of predicted slots on the different size of ATIS training examples, where K-SAN utilizes the dependency relations parsed from the Stanford parser. Small: 1/40 set; Medium: 1/10 set; Large: original set. ($\dag$ indicates that the performance is significantly better than all baseline models with $p<0.05$ in the t-test.)}
  \label{tab:result}
\vspace{-3mm}
\end{table*}

\subsection{Experimental Setup}
The dataset for experiments is the benchmark ATIS corpus, which is extensively used by the NLU community~\cite{mesnil2015using}.
There are 4978 training utterances selected from Class A (context independent) in the ATIS-2 and ATIS-3, while there are 893 utterances selected from the ATIS-3 Nov93 and Dec94.
In the experiments, we only use lexical features.
In order to show the robustness to data scarcity, we conduct the experiments with 3 different sizes of training data (Small, Medium, and Large), where Small is 1/40 of the original set, Medium is 1/10 of the original set, and Large is the full set.
The evaluation metrics for NLU is F-measure on the predicted slots\footnote{The used evaluation script is \texttt{conlleval}.}.

For experiments with K-SAN, we parse all data with the Stanford dependency parser~\cite{chen2014fast} and represent words as their embeddings trained on the in-domain data, where the parser is pre-trained on PTB.
The loss function is cross-entropy, and the optimizer we use is adam with the default setting~\cite{kingma2014adam}, where the learning rate $\lambda=0.001$, $\beta_1=0.9$, $\beta_2=0.999$, and $\epsilon=1e^{-08}$.
The maximum iteration for training our K-SAN models is set as 300.
The dimensionality of input word embeddings is 100, and the hidden layer sizes are in $\{50, 100, 150\}$.
The dropout rates are set as $\{0.25, 0.50\}$.
All reported results are from the joint RNN tagger, and the hyperparameters are tuned in the dev set for all experiments.

\subsection{Baseline}

To validate the effectiveness of the proposed model, we compare the performance with the following baselines.
\begin{itemize}
\vspace{-1mm}
\item Baseline:
\begin{itemize}
\vspace{-1mm}
\item CRF Tagger~\cite{tur2010left}: predicts a semantic slot for each word with a context window (size = 5).
\vspace{-1mm}
\item RNN Tagger~\cite{mesnil2015using}: predicts a semantic slot for each word.
\vspace{-1mm}
\item CNN Encoder-Tagger~\cite{kim2014convolutional}: tag semantic slots with consideration of sentence embeddings learned by a convolutional model.
\vspace{-1mm}
\end{itemize}
\vspace{-1mm}
\item Structural: The NLU models utilize linguistic information when tagging slots, where DCNN and Tree-RNN are the state-of-the-art approaches for embedding sentences with linguistic structures.
\begin{itemize}
\vspace{-1mm}
\item CRF Tagger~\cite{tur2010left}: predicts slots based on the lexical (5-word window) and syntactic (dependent head in the parsing tree) features.
\vspace{-1mm}
\item DCNN~\cite{ma2015dependency}: predicts slots by incorporating sentence embeddings learned by a convolutional model with consideration of dependency tree structures.
\vspace{-1mm}
\item Tree-RNN~\cite{tai2015improved}: predicts slots with sentence embeddings learned by an RNN model based on the tree structures of sentences.
\vspace{-1mm}
\end{itemize}
\vspace{-1mm}
\end{itemize}

\begin{figure*}[t]
\vspace{-1mm}
\centering
\includegraphics[width=0.92\linewidth]{attention_comparison.pdf}
\vspace{-3mm}
\caption{The visualization of the decoded knowledge-guided structural attention for both {\bf relations} and {\bf words} learned from different size of training data. Relations and words with darker color indicate higher attention weights generated by the proposed K-SAN with CNN. The slot tags are shown in the figure for reference.
Note that the dependency relations are incorrectly parsed by the Stanford parser in this example, but our model is still able to benefit from the structural information.}
\vspace{-2mm}
\label{fig:attention}
\end{figure*}

\subsection{Slot Filling Results}
Table~\ref{tab:result} shows the performance of slot filling on different size of training data, where there are three datasets (Small, Medium, and Large use 1/40, 1/10, and whole training data).
For baselines (models without knowledge features), CNN Encoder-Tagger achieves the best performance on all datasets.

Among structural models (models with knowledge encoding), Tree-RNN Encoder-Tagger performs better for Small data but slightly worse than the DCNN Encoder-Tagger.

CNN~\cite{kim2014convolutional} performs better compared to DCNN~\cite{ma2015dependency} and Tree-RNN~\cite{tai2015improved}, even though CNN does not leverage external knowledge when encoding sentences.
When comparing the NLU performance between baselines and other state-of-the-art structural models, there is no significant difference.
This suggests that encoding sentence information without distinguishing substructure may not capture salient semantics in order to improve understanding performance.

Among the proposed K-SAN models, CNN for encoding performs best on Small (75\% on F1) and Medium (88\% on F1), and RNN for encoding performs best on the Large set (95\% on F1).
Also, most of the proposed models outperform all baselines, where the improvement for the small dataset is more significant.
This suggests that the proposed models carry better generalization and are less sensitive to unseen data.
For example, given an utterance ``{\it which flights leave on monday from montreal and arrive in chicago in the morning}'', ``{\it morning}'' can be correctly tagged with a semantic tag \textsf{B-arrive\_time.period\_of\_day} by K-SAN, but it is incorrectly tagged with \textsf{B-depart\_time.period\_of\_day} by baselines, because knowledge guides the model to pay correct attention to salient substructures.
The proposed model presents the state-of-the-art performance on the large dataset (RNN-BLSTM in baselines), showing the effectiveness of leveraging knowledge-guided structures for learning embeddings that can be used for specific tasks and the robustness to data scarcity and mismatch.

\subsection{Attention Analysis}

In order to show the effectiveness of boosting performance by learning correct attention from much smaller training data through the proposed model,
we present the visualization of the attention for both words and relations decoded by K-SAN with CNN in the Figure~\ref{fig:attention}.
The darker color of blocks and lines indicates the higher attention for words and relations respectively.
From the figure, the words and the relations with higher attention are the most crucial parts for predicting correct slots, e.g. origin, destination, and time.
Furthermore, the difference of attention distribution between three datasets is not significant; this suggests that our proposed model is able to pay correct attention to important substructures guided by the external knowledge even the training data is scarce.

\begin{figure*}[htb]
\begin{minipage}[b]{.47\linewidth}
  \centering
  \centerline{\includegraphics[width=\linewidth]{dep}}
  \centerline{(a) Syntax: the dependency tree}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.47\linewidth}
  \centering
  \centerline{\includegraphics[width=\linewidth]{amr}}
  \centerline{(b) Semantics: the AMR graph}\medskip
\end{minipage}
\vspace{-4mm}
\caption{The constructing procedure of knowledge-guided substructures, $x_i$, on an example sentence $s$.}
\label{fig:substructure}
\end{figure*}

\begin{table*}[t]
    \centering
  \begin{tabular}{cllrccc}
\hline
Approach & \multicolumn{3}{c}{Knowledge (Max \#Substructure)} & Small & Medium & Large\\
\hline\hline
\multirow{4}{*}{CRF} & \multirow{2}{*}{Dependency Tree} & Stanford & - & 59.55 & 78.71 & 90.13\\
 & & SyntaxNet & - & 61.09 & 78.87 & 90.92\\
 \cline{2-7}
 & \multirow{2}{*}{AMR Graph} & Rule-Based & - & 59.55 & 79.15 & 89.97\\
 & & JAMR & - & 61.12 & 78.64 & 90.25\\
\hline
\multirow{4}{*}{K-SAN (CNN)} & \multirow{2}{*}{Dependency Tree} & Stanford & 53 & \bf 74.60 & 87.99 & 94.86 \\
& & SyntaxNet & 25 & 74.35 & \bf 88.40 & \bf 95.00\\
 \cline{2-7}
& \multirow{2}{*}{AMR Graph} & Rule-Based & 19 & 74.32 & 88.14 & 94.85\\
& & JAMR & 8 & 74.27 & 88.27 & 94.89\\
\hline
  \end{tabular}
  \vspace{-1.5mm}
  \caption{The F1 scores of predicted slots with knowledge from different resources.}
  \label{tab:parser}
\vspace{-2mm}
\end{table*}

\subsection{Knowledge Generalization}
In order to show the capacity of generalization to different knowledge resources, we perform the K-SAN model for different knowledge bases.
Below we compare two types of knowledge formats: dependency tree and Abstract Meaning Representation (AMR).
AMR is a semantic formalism in which the meaning of a sentence is encoded as a rooted, directed, acyclic graph~\cite{banarescu2013amr}, where nodes represent concepts, and labeled directed edges represent the relations between two concepts.
The formalism is based on propositional logic and neo-Davidsonian event representations~\cite{parsons1990events,davidson1967logical}.
The semantic concepts in AMR were leveraged to benefit multiple NLP tasks~\cite{liu2015toward}.
Unlike syntactic information from dependency trees, the AMR graph contains semantic information, which may offer more specific conceptual relations.
Figure~\ref{fig:substructure} shows the comparison of a dependency tree and an AMR graph associated with the same example utterance and how the knowledge-guided substructures are constructed.

Table~\ref{tab:parser} presents the performance of CRF and K-SAN with CNN taggers that utilize dependency relations and AMR edges as knowledge guidance on the same datasets, where CRF takes the head words from either dependency trees or AMR graphs as additional features and K-SAN incorporates knowledge-guided substructures as illustrated in Figure~\ref{fig:substructure}.
The dependency trees are obtained from the Stanford dependency parser or the SyntaxNet parser\footnote{\url{https://github.com/tensorflow/models/tree/master/syntaxnet}}, and AMR graphs are generated by a rule-based AMR parser or JAMR\footnote{\url{https://github.com/jflanigan/jamr}}.

Among four knowledge resources (different types and obtained from different parsers), all results show the similar performance for three sizes of datasets.
The maximum number of substructures for the dependency tree is larger than the number in the AMR graph (53 and 25 v.s. 19 and 8), because syntax is more general and may provide richer cues for guiding more attention while semantics is more specific and may offer stronger guidance.
In sum, the models applying four different resources achieve similar performance, and all significantly outperform the state-of-the-art NLU tagger, showing the effectiveness, generalization, and robustness of the proposed K-SAN model.

\section{Conclusion}
\label{sec:con}
This paper proposes a novel model, knowledge-guided structural attention networks (K-SAN), that leverages prior knowledge as guidance to incorporate non-flat topologies and learn suitable attention for different substructures that are salient for specific tasks.
The structured information can be captured from small training data, so the model has better generalization and robustness.
The experiments show benefits and effectiveness of the proposed model on the language understanding task, where all knowledge-guided substructures captured by different resources help tagging performance, and the state-of-the-art performance is achieved on the ATIS benchmark dataset.

\newpage


\section{Introduction}

Kinematic dynamo consists in enhancement of magnetic field fluctuations in a moving conducting fluid. This enhancement is statistically significant for non-stationary flows,
in particular, chaotic flows. The dynamo is
relevant for astrophysics  \cite{MHD,P79,ZRS,astro1,schek} and for so-called elastic turbulence which
is the chaotic motion of polymer solutions \cite{00GS,01GSb,00BFL}. In both cases the velocity field $\bm{v}(\bm{r},t)$ can be considered
as a smooth function of coordinates $\bm{r}$   allowing the use of the Taylor expansion. Practically this means that in the Lagrangian frame comoving with a given liquid particle the
profile of the field $\bm{v}(\bm{r},t)$ is approximately linear:
 \begin{equation}
 v_\alpha(\bm{r},t)\approx \sigma_{\alpha\beta}(t)r_\beta
 \label{linprof}
 \end{equation}
with matrix-valued random function $\sigma_{\alpha\beta}(t)$ of time $t$.
Evolution of the magnetic field $\bm{B}(\bm{r},t)$ is governed by the equation \cite{LL}:
 \begin{equation}
 \partial_t{\bm B}=
 ({\bm B}\cdot{\bm\nabla}){\bm v}
 -({\bm v}\cdot{\bm\nabla}){\bm B}
 + \kappa\nabla^2{\bm B},
 \label{magnet}
 \end{equation}
where $\kappa$ is the dissipation coefficient inverse proportional to the conductivity of the fluid.

The present paper is devoted to  computation of  correlation tensors of the field ${\bm B}$
for two-dimensional incompressible chaotic flow at conditions when the expansion (\ref{linprof}) is applicable. The statistics of the traceless matrix
$\sigma_{\alpha\beta}(t)$ is assumed to be the Gaussian $\delta$-correlated in time random process:
\begin{equation}
\langle \sigma_{\alpha\mu}(t)\sigma_{\beta\nu}(t')\rangle=
{\cal G}\left(3\delta_{\alpha\beta}\delta_{\mu\nu}-\delta_{\beta\nu}\delta_{\alpha\mu}-\delta_{\beta\mu}\delta_{\alpha\nu}\right)
\delta(t-t'),
 \label{sta}
 \end{equation}
where the Stratonovich regularization of the $\delta$-function is assumed,i.e.,  the $\delta$-function is treated as the limit of a sequence of symmetric with respect
to reflection $(t-t')\to (t'-t)$ functions. It should be stressed that  this is an
external requirement and  important   part  of the model formulation.
Such tensor structure follows from the expansion of the velocity field  $\bm{v}(\bm{r},t)$ correlator in Eulerian frame:
\begin{equation}
\langle v_{\mu}(\bm{r},t)v_{\nu}(\bm{0},t')\rangle \approx \left[V_0^2\delta_{\mu\nu}-
{\cal G}\left(\frac{3}{2}r^2\delta_{\mu\nu}-r_\mu r_\nu\right)\right]
\delta(t-t'), \quad r\ll R_c,
 \label{velt}
 \end{equation}
where $R_c$ has a meaning of the smoothness scale determined by fluid viscosity. We consider here the case when $R_c\gg r_d=2\sqrt{\kappa/{\cal G}}$ which is suitable for astrophysical and
rheological applications.

It was shown \cite{Kaz,99CFKV,schek} for three-dimensional chaotic flows described by this BKK (Batchelor-Kraichnan- Kazantsev) model that the
moments $\langle \bm{B}^{2n}(\bm{r},t)\rangle, n=1,2, \dots$  grow exponentially with time and the correlation functions of the field tend to universal  spatial shapes.
These shapes correspond to filaments with increasing length in which the magnetic field is concentrating in the course of evolution. The width of such filaments
is decreasing up to the dissipative scale
$r_d$ where the value if the width is stabilized. At this moment the resistive stage of the evolution begins. The exponential growth continues but the exponent changes
(see also review   \cite{01FGV}).

Two-dimensional flows were not considered long time in dynamo studies since it was asserted in
\cite{Zeldovich1, Zeldovich2,Zeldovich3} (and recognized by the community) that in this case the initial growth of the field changes to exponential decay when the evolution
turns to be strictly diffusive. It should be emphasized that  the magnetic field ${\bm B}(\bm{r},t)$ is considered to be three-dimensional both as a vector and as a function of space position,
only the flow is two-dimensional: ${\bm v}=(v_1, v_2, 0)$.
In the papers \cite{KogKolLe,KleSiz} it is shown explicitly that the arguments leading to this statement are incorrect and the non-dissipative initial growth
gives way to the dissipative regime which occurs with exponential increase of the magnetic field correlators again. When the characteristic length  of the filaments reaches $R_c$
 the field values in neighboring filaments begin to anticorrelate \cite{Kolo16} and the dynamo stops.
It is  in agreement with exact mathematical theorems \cite{Ose,Arno} related to infinite time limit. In \cite{Kolo16} it is shown that the maximal value of $\langle {\bm B}^2\rangle$
reached in the course of evolution is
$\sim \langle {\bm B_0}^2\rangle (R_c/r_d)^2 \gg \langle {\bm B_0}^2\rangle$ where ${\bm B}_0$ is the initial field amplitude. This enhancement is picked up in the exponential growth stage.
Depending on initial level of fluctuations the growth stage can be terminated also by the back reaction of the magnetic field on the fluid flow \cite{schek,Sche2}.
In this case the spatial structure of the field fluctuations is extremely important. We show in the present paper that the spatial statistics of the magnetic field in the resistive growth
regime in the framework of  two-dimensional BKK-model is intermittent similar to three-dimensional case  \cite{99CFKV,01FGV}.  The path-integral formalism created to study passive scalar
statistics \cite{ChGK,CFK} and isotropic correlations in three-dimensional dynamo problem \cite{99CFKV}  is  developed here to compute the tensor structure of two- and four-point
magnetic field correlators.

The evolution equation (\ref{magnet}) conserves the divergence of the field  $\bm{B}(\bm{r},t)$.
On the other hand, the evolution equations for the magnetic field components in the flow plane $B_\mu(\bm{r},t), \, \mu=1,2$ and for the
component $B_3(\bm{r},t)$ perpendicular to this plane decouple. This means that we can consider evolution of two-component divergence-full field
$B_\mu(\bm{r},t), \, \mu=1,2$ separately since the condition
\begin{equation}
\partial_\mu B_\mu+\partial_3 B_3=0
 \label{divergenziya}
 \end{equation}
is satisfied by the component $B_3(\bm{r},t)$ governed be the decoupled evolution equation. The dependence of $B_\mu(\bm{r},t), \, \mu=1,2$
on the coordinate $r_3$,  being essential in the condition (\ref{divergenziya}), is unimportant in the evolution. The coordinate $r_3$ enters
the correlation functions of components $B_\mu(\bm{r},t), \, \mu=1,2$ as a parameter only and we don't take care here on it.

\section{Dynamical computation of the  correlation tensors of the field $B_\alpha(\bm{r},t)$.}

 The change of the frame to  (\ref{linprof}) does not change the simultaneous statistics  of the field $B_\mu(\bm{r},t)$,
so that we use the following evolution equation:
 \begin{equation}
 \partial_t B_\alpha=
 \sigma_{\alpha\mu}B_\mu-\sigma_{\mu\nu}r_\nu\partial_\mu B_\alpha+
  \kappa\nabla^2 B_\alpha.
 \label{magnet1}
 \end{equation}
Performing the spatial Fourier transform:
\begin{equation}
 B_\alpha(\bm{r},t)=\int\frac{d^2{\bf k}}{(2\pi)^2}e^{i{\bf k}{\bf r}}b_\alpha(\bm{k},t)
 \label{fur}
 \end{equation}
we obtain for    $b_\alpha(\bm{k},t)$ the first-order partial differential equation:
 \begin{equation}
 \partial_t b_\alpha=
 \sigma_{\alpha\mu}b_\mu+\sigma_{\mu\nu}k_\mu\frac{\partial}{\partial k_\nu} b_\alpha-
  \kappa k^2 b_\alpha.
 \label{magnet2}
 \end{equation}
Its solution with the initial data $\bm{b}(\bm{k},0)$ has the form:
\begin{eqnarray}
b_\alpha\left( {\bm k},t\right) = (\hat{W})_{\alpha\beta}\left( t\right)b_\beta
\left( \hat{W}^T\left( t\right) {\bf k},0\right) \exp \left[ -\kappa
\int\limits_0^td\tau\left( {\bf k}\hat{W}(t,\tau)\hat{W}^T(t,\tau){\bf k}
\right) \right].
 \label{polle}
\end{eqnarray}
where the matrices $\hat{W}(t)$ and $\hat{W}(t,\tau)$ obey the equation
\begin{equation}
d\hat{W}/dt=\hat{\sigma}\hat{W}
\label{weq}
\end{equation}
and can be written
as the ordered exponentials:
\begin{eqnarray}
\hat{W}(t)={\rm T}\exp \left( \int\limits_0^tdt' \,\hat{\sigma}\left( t'\right) \right), \quad
\hat{W}(t,\tau)={\rm T}\exp \left(
\int\limits_{\tau}^t dt' \,\hat{\sigma}\left( t' \right) \right)= \hat{W}(t)\hat{W}^{-1}(\tau).
\label{upe}
\end{eqnarray}
The incompressibility of the flow $\sigma_{\alpha\alpha}=0$ leads to the unimodularity of the matrix $\hat{W}(t)$: $\det\hat{W}(t)=1$.

To perform the averaging over the matrix Gaussian random process $\hat{\sigma}(t)$ we use the path integral formalism.
The  measure  corresponding to the correlation function (\ref{sta}) has the form:
\begin{eqnarray}
{\cal D}\hat{\sigma}(\tau)\exp\left\{-\frac{1}{16 {\cal G}}\int \,d\tau
\left[3\mathrm{Tr}\left(\hat{\sigma}\hat{\sigma}^T\right)+
\mathrm{Tr}\left(\hat{\sigma}^2\right)\right]\right\}.
 \label{ves}
 \end{eqnarray}
The matrix $\hat{W}(t)$ cannot be expressed as a functional of $\hat{\sigma}(t)$ explicitly. Hovewer, for the our problem
this difficulty can be avoided (see also \cite{CFK}). Let us perform  the Iwasava parametrization of the matrix  $\hat{W}(t)$:
 \begin{equation}\label{tri}
\hat{W}=\hat{{\cal O}}(\varphi) \hat{D}(\rho)\hat{T}(\chi),
\end{equation}
where
\begin{equation}\label{tri1}
\hat{{\cal O}}(\varphi)= \left(\begin{array}{cc} \cos\varphi & \sin\varphi \\
        -\sin\varphi  & \cos\varphi  \end{array}\right),\quad
 \hat{D}(\rho)=
\left(\begin{array}{cc} e^{\rho} & 0 \\
        0  & e^{-\rho}  \end{array}\right),\quad
\hat{T}(\chi)=
\left(\begin{array}{cc} 1 & \chi(t) \\
        0  & 1  \end{array}\right),
\end{equation}
and the parameters $\varphi(t), \rho(t)$ and $\chi(t)$ are some functions of time $t$ determined by $\hat{\sigma}(t)$ implicitly via the equation:
 \begin{equation}
 \label{tri2}
\hat{\sigma}(t)=d \hat{W}(t)/dt \hat{W}^{-1}(t)=\hat{{\cal O}}(\varphi)
\left(\begin{array}{cc} \dot{\rho} & \dot{\varphi}+\dot{\chi}e^{2\rho}\\
        -\dot{\varphi}  & -\dot{\rho}  \end{array}\right)\hat{{\cal O}}^{-1}(\varphi).
\end{equation}
The  initial conditions $\rho(0)=\chi(0)=\varphi(0)=0$ correspond to the evident equality $\hat{W}(0)=1$. If we consider the relation
(\ref{tri2}) as the change of variables in the measure (\ref{ves}) we obtain an explicit path integral expression for any correlation functions of the magnetic field $B_\alpha(\bm{r},t)$. Assuming the retarded regularization of the derivatives, for example,
$\dot{\rho}=(\rho_n-\rho_{n-1})/\epsilon$ where $\epsilon$ is the infinitesimal time interval, the Jacobian of the transformation can be set to  a constant.
The resulting averaging measure expressed in terms
of the new variables has the form:
 \begin{eqnarray}
\label{mts}
{\cal N}
{\cal D}\chi{\cal D}\rho {\cal D}\varphi
\left(\prod_\tau e^{2\rho(\tau)}\right)
\exp\left\{-\frac{1}{2{\cal G}}\int\limits_0^t\! d\tau\!
\left[\left(\dot{\rho}-{\cal G}\right)^2+\frac{1}{4}e^{4\rho}\dot{\chi}^2+
\frac{1}{2}\left(\dot{\varphi}+\dot{\chi}e^{2\rho}\right)^2 \right]\right\}.
\end{eqnarray}
Here the retarded regularization of the time derivatives is assumed (see e.g. \cite{Kam}) and ${\cal N}$ is the normalization  constant  providing the equality $<1>=1$. The origing of non-zero $\langle \dot{\rho}\rangle$ is due to
so called contact terms in the Lagrangian like
$\epsilon \dot{\chi}^2\dot{\rho}\exp(4\rho)\sim\dot{\rho}$ arising when we uniformize the regularization of all the terms
to retarded form.   The corresponding explicit computations are simple but tedious. Instead we can fix the final coefficients by the requirement that
the growth law of a vector $\bm{a}(t)$ lenght:
\begin{equation}
 \langle \bm{a}^2(t)\rangle= \langle \bm{a}(0)\hat{W}^T(t)\hat{W}(t)\bm{a}(0)\rangle=\bm{a}^2(0)e^{4{\cal G}t}
 \label{vg}
 \end{equation}
following directly from (\ref{sta})  must be reproduced. It is worth noting that it is  equivalent to the equality
$\langle \exp(2\rho) \rangle=\exp(4{\cal G}t)$ check.

We suppose the initial
distribution of the magnetic field to be random with  Gaussian  spatially homogeneous statistics (below we consider also the case
of a given configuration of the field). The correlator of the Fourier components of the field $b_\alpha(\bm{k},t)$ has the form:
\begin{equation}
\langle b_\alpha\left( {\bm k},0\right) b_\beta\left( {\bm k_1},0\right)=\delta\left({\bm k}+{\bm k_1}\right)
\delta_{\alpha\beta}f(k^2l^2),
 \label{incorr}
\end{equation}
where $l$ is the  correlation lengh of the initial magnetic field fluctuations and $f(k^2l^2)$ is a positive function going  to zero when $kl\to \infty$.
The expression for the correlation tensor of in-plane components of the magnetic field following from (\ref{polle}) and (\ref{incorr}) has the form:
\begin{eqnarray}
&F_{\alpha\beta}({\bf r},t)=\langle B_\alpha({\bf r},t) B_\beta(0,t)\rangle =
 \Bigl\langle\left[ \hat{W}(t)\hat{W}^{T}(t)\right]_{\alpha\beta}\times
\nonumber \\
&\times
\int \frac{d^2{\bf k}}{(2\pi)^2}
 f(k^2l^2)\exp\left(i{\bf r}\hat{W}^{-1}(t){\bf r}\right)
\exp\left\{-2\kappa\int_0^t d\tau\left({\bf k}\hat{W}^{-1}(\tau)
\hat{W}^{-1,T}(\tau){\bf k} \right)\right\}\Bigr\rangle.
\label{pco1}
\end{eqnarray}
We study the large time ${\cal G}t \gg 1$ asymptotics of this tensor. The key point which simplifies all the calculations in the dynamo problem
lies in the fact that in this limit the averaging  of products $B_\alpha(\bm{r},t)B_\mu(\bm{r}',t)\dots $ with respect to the measure (\ref{mts}) can be done in saddle-point approximation at least up to a constant factor.  This means that the dominating contribution to the
required expectation values are given by trajectories $\rho(t)=A{\cal G}t)+\delta\rho(t)$ where $A$ is some positive constant and
$\delta\rho(t)\sim 1 \ll {\cal G}t$. This property is a consequence  of proportionality of the product $B_\alpha(\bm{r},t)B_\mu(\bm{r}',t)\dots $
to $\exp\left(a\rho(t)\right)$ with some positive constatnt $a>0$. Substituting $\rho(t)=A{\cal G}t)+\delta\rho(t)$ into the path integral for
 $\langle B_\alpha(\bm{r},t)B_\mu(\bm{r}',t)\dots \rangle$ one can find the constant $A(a)$ requiring the absence of the terms leading to linear growth of $\delta\rho(t)$ with $t$. The exponential growth of $\exp(\rho(t))$ on the dominating trajectories leads to suppression of  $\dot{\chi}^2$ for ${\cal G}t\gg 1$ \cite{CFK,CKV}. The fluctuations of the variable $\chi$ become frozen  in sense that the probability distribution function (PFD) of $\chi$ is asymptotically time-independent for $\rho(t)=A{\cal G}t)+\delta\rho(t)$. This statement can be illustrated by the joint PDF for the variables $(\rho, \chi, \varphi)$ found in  \cite{K10}. For  ${\cal G}t\gg 1$ it has the form:
\begin{eqnarray}
{\cal P}(\rho,\chi,\varphi,t)\approx
{\cal C}\frac{\rho}{({\cal G}t)^{3/2}}
\left(1+\chi^2\right)^{-\frac{1}{2}-\frac{\rho}{2{\cal G}t}}
\exp\left[-\frac{\rho^2}{2{\cal G}t}+\rho-\frac{1}{2}{\cal G}t\right],\quad  {\cal C}\sim 1.
\label{assy}
 \end{eqnarray}
Here some inaccuracies made in \cite{K10} are corrected. The independence of (\ref{assy}) on $\varphi$ corresponds to the uniform distribution
of this angle in the interval $[0,2\pi]$.

We consider here the limit of small dissipation parameter $\kappa$ so that the corresponding dissipation length
$r_d=2\sqrt{\kappa/{\cal G}}$ is small comparing with the initial correlation length $l$: $r_d\ll l$. On the other hand, the dissipation
becomes important when time goes to infinity and it determines to a large extent the amplitude and the form of the field correlation functions.

Returning to the expression (\ref{pco1}) and taking into account the independence of the parameter $\chi$ on time at ${\cal G}t\gg 1$ we perform
the change of variables
\[
{\bf k}=\left(\begin{array}{cc} 1 & 0 \\
        -\chi  & 1  \end{array}\right) {\bf Q}
\]
and neglect  the decaying exponential $\exp(-\rho)$ everywhere.
For dominating trajectories the approximation
\begin{equation}
\label{limt}
\int\limits_0^t d\tau\, e^{2\rho(\tau)}
\approx c {\cal G}^{-1}e^{2\rho(t)}.
\end{equation}
is valid.
Here the quantity $c$:
\[
c={\cal G}\int\limits_0^t d\tau\, e^{2\rho(\tau)-2\rho(t)}
\]
fluctuates but such the fluctuations result in some numerical factor when we compute the correlator (\ref{pco1}). This effect
can be important in study of high-order moments of the magnetic field but it is not important when we compute $F_{\alpha\beta}$
 up to a constant prefactor. Hence the leading comtribution to the correlation tensor has the form:
\begin{eqnarray}
&&F_{\alpha\beta}({\bf r},t)\propto\Bigl\langle (1+\chi^2)e^{2\rho}
\left(\begin{array}{cc} \cos^2\varphi & -\sin \varphi\cos \varphi \\
        -\sin\varphi\cos\varphi  & \sin^2\varphi  \end{array}\right)_{\alpha\beta}\times
\nonumber \\
&&\times\int\,d^2{\bf Q}f\left(Q_1^2l^2[1+\chi^2]\right)
\exp\left\{iQ_2e^\rho\left(r_1\sin\varphi+r_2\cos\varphi\right)-c Q_2^2e^{2\rho}r_d^2\right\}
\Bigr\rangle.
\label{pco2}
\end{eqnarray}
Here  the averaging is assumed to be performed with respect to the finite-dimensional probability density (\ref{assy}). $dQ_1$ - integration can done  first producing the factor $\sim (1+\chi^2)^{-1/2}l^{-1}f_0$, $f_0=\int dq\,f(q^2)$. The subsequent integration
with respect to $d\chi$ is convergent and results into a numerical factor of the order of unity. The integration over the variable $Q_2$ is Gaussian and it gives us the expression:
\begin{equation}
F_{\alpha\beta}({\bf r},t)\propto\frac{f_0}{lr_d}\Bigl \langle e^{\rho}
\left(\begin{array}{cc} \cos^2\varphi & -\sin \varphi\cos \varphi \\
        -\sin\varphi\cos\varphi  & \sin^2\varphi  \end{array}\right)_{\alpha\beta}
\exp\left[-\frac{r^2}{c r_d^2}\sin^2\left(\varphi+\varphi_0\right)\right]
\Bigr\rangle.
\label{pco21}
\end{equation}
where $\tan\varphi_0=r_2/r_1$. The averaging over the variable $\rho$ produces growing factor $\exp(3{\cal G}t/2)$ which corresponds to the dominating trajectory $\rho(t)\approx 2{\cal G}t$. The final averaging over the angle $\varphi$ results in universal forms for correlation tensor in two limits:
\begin{eqnarray}
\label{te1}
F_{\alpha\beta}({\bf r},t)\propto \delta_{\alpha\beta}\frac{f_0e^{3{\cal G}t/2}}{l r_d},
\quad r\ll r_d,
\end{eqnarray}
and
\begin{equation}
\label{te4}
F_{\alpha\beta}({\bf r},t)\propto \frac{f_0}{l }\exp\left(\frac{3}{2}{\cal G}t\right)
\frac{r_\alpha r_\beta}{r^3}, \quad r\gg r_d.
\end{equation}
In this asymptotics the correlation tensor $F_{\alpha\beta}({\bf r},t)$  does not depend on $r_d$ similar to three-dimensional case \cite{Kaz}.

Turning to the four-point correlation tensor it is easy to see that for the linear velocity profile (\ref{linprof}) there is the decomposition:
\begin{equation}
\label{four1}
\langle B_\alpha({\bm r}_1,t) B_\beta({\bm r}_2,t)B_\gamma({\bm r}_3,t)B_\mu({\bm r}_4,t)\rangle=
\hat{Q}\left({\bm R}_{12}, {\bm R}_{34}\right)+ \hat{Q}\left({\bm R}_{13}, {\bm R}_{24}\right)+
\hat{Q}\left({\bm R}_{14}, {\bm R}_{23}\right),
\end{equation}
where ${\bm R}_{jl}={\bm r}_j-{\bm r}_l,\, j,l=1,\dots,4$. The tensor function $\hat{Q}\left({\bm R}_1, {\bm R}_2\right)$ can be expressed in a form similar to (\ref{pco21}).
The presence of two angles related to the vectors ${\bm R}_1$ and ${\bm R}_2$ leads to the strong angle dependence:
\begin{equation}
\label{angl}
\hat{Q}\left({\bm R}_1, {\bm R}_2\right)\sim \exp\left(-\frac{R^2}{r_d^2} \Theta_{12}^2\right),
\end{equation}
where $ \Theta_{12}$ is the angle between ${\bm R}_1$ and ${\bm R}_2$ and the relation $R_1\sim R_2\sim R$ is assumed. In the collinear limit ${\bm R}_{1,2}={\bm n}R_{1,2}$
one can get the simple expression:
\begin{equation}
\label{coll}
\hat{Q}\left({\bm R}_1, {\bm R}_2\right)\sim n_\alpha n_\beta n_\gamma n_\mu \frac{f_0^2 l^2}{r_d\sqrt{R_1^2+R_2^2}}\exp\left(4{\cal G}t\right).
\end{equation}

\section{Conclusion.}

The picture corresponding to the correlators found above is the following: the field $B_\alpha$ is concentrated in parallel strips of the width $\sim r_d$.
The field is directed mainly along the strip. These strips are rotated by the flow and the time interval $\tau_s$ when the observation
points reside in a strip is of the order of $r_d/r$ where $r$ is the distance between the points. The angle dependence (\ref{angl}) corresponds exactly to this
intermittent stripes structure.
The $r$-dependence of the two- and four-point correlators coincides because it has the mentioned geometrical origin.
The ratio $\langle {\bm B}^4\rangle/\langle {\bm B}^2\rangle^2\sim \exp(2{\cal G}t)\gg 1$ corresponds also to the temporal intermittency.

For the times $t\gg {\cal G}^{-1}\ln(R_c/r_d)$ the approximation (\ref{linprof}) does not work. The pair correlator of the magnetic field is studied in \cite{Kolo16} but
high-order correlation functions are unknown and the question about the intermittency effects in this stage of evolution remains open.

I am very grateful to V.V.Lebedev for helpful advices and numerous discussions and to L.Krainov for useful remarks. I wish to thank G.Falkovich for the interest to the work.
The author acknowledges support of RScF, grant 14-22-00259.

\section*{References}