Context:

1: "The design space of possible architectures is immense, making exhaustive exploration impos-
sible. One option would be to exactly replicate the architecture of an existing large language
model. On the other hand, a great deal of work on improving existing architectures has
seen relatively little adoption (NAME_1 et NAME_2., 2021); adopting some of these recommended
practices could yield a significantly better model. We take a middle ground and focus on
model families that have been shown to scale well, and that have reasonable support in
publicly available tools and codebases. We ablate components and hyperparameters of the
models, seeking to make the best use of our final compute budget."

2: "We conducted our ablation experiments using smaller models. We used the 6.7B pa-
rameter scale for the pretraining objective ablations (NAME_3 et NAME_2., 2022a) and the 1.3B scale
for the rest including position embeddings, activations, and layer normalization (NAME_4 Scao
et NAME_2., 2022). Recently, NAME_5 et NAME_2. (2022) identified a phase transition for models larger
than 6.7B, in which the emergence of “outliers features” is observed. This questions whether
results obtained at the 1.3B scale should be assumed to extrapolate to our final model size."

Using this context, answer the following question:

"What's the architecture, and how was it trained?" There should be exactly 2 paragraphs separated by the markdown divider: ***.
user