To stabilize the learning process of self-attention, we have found extending our mechanism to employ multi-head attention to be beneficial, similarly to NAME_1 et al. (2017). Specifically, K independent attention mechanisms execute the transformation of Equation 4, and then their features are
concatenated, resulting in the following output feature representation: where k represents concatenation, αkij are normalized attention coefficients computed by the k-th attention mechanism (ak), and Wk is the corresponding input linear transformation’s weight matrix. Note that, in this setting, the final returned output, h0 , will consist of KF0 features (rather than F0) for each node. Specially, if we perform multi-head attention on the final (prediction) layer of the network, concatenation is no longer sensible—instead, we employ averaging, and delay applying the final nonlinearity (usually a softmax or logistic sigmoid for classification problems) until then. Summarize the above in a formal way: Give two different versions of responses. Please separate the responses with 6 asterisk symbols: ******.
user