\section{}

\bibliographystyle{elsarticle-num-names}

\section{Introduction}
\label{Intro}
Measurements in nuclear $\beta$-decay have historically contributed to
the establishment of the standard model of electroweak physics as a
theory containing massive bosons coupling only to left-handed
chirality leptons. Today, precision measurements search for and
constrain possible new physics.  For example, in isobaric analog,
mixed Fermi-Gamow Teller $\beta^{\pm}$ decays, the angular
distribution of the leptons with respect to the spin direction of the
parent nucleus is sensitive to a variety of new physics including
right-handed currents and scalar or tensor
interactions~\cite{Holstein1977, Naviliat-Cuncic1991, Herczeg2001,
  Severijns2006}.  Additionally, if we ignore this class of standard
model extensions, this measurement can be combined with other
measurements of isospin $T=1/2$ mirror-transitions to extract the
$V_{ud}$ element of the Cabibbo-Kobayashi-Maskawa quark mixing
matrix~\cite{Severijns2008, Naviliat-Cuncic2009}. This technique is
complementary to and independent of the most precise value obtained
using $T=0$ super-allowed decays~\cite{Hardy2015}. In general, to
complement high-energy searches for exotic currents in the weak
interaction, these experiments should aim for a precision of
$\sim0.1\%$~\cite{Cirigliano2013a}.

To reach this ambitious goal, we have developed the techniques at the
{\scshape Triumf}{} Neutral Atom Trap ({\scshape Trinat}{}) to confine the $\beta^+$-emitter,
\isotope[37]{K} ($I^\pi=3/2^+\rightarrow3/2^+$, $t_{1/2}=\SI{1.2}{\second}$),
in an alternating current magneto-optical trap
(AC-MOT)~\cite{Harvey2008, Anholm2014} and observe its decay
products~\cite{Melconian2007, Melconian05}.  Furthermore, the atoms
are spin-polarized by optical pumping (OP) while the MOT is off,
creating an ideal source of polarized atoms decaying nearly from rest
in an exceptionally open geometry.

Using this setup, we have previously measured the $\nu$-asymmetry
($B_\nu$ in~\cite{Jackson1957}) to $3.6\%$
uncertainty~\cite{Melconian2007} and the $\beta$-asymmetry ($A_\beta$)
to $1.5\%$ uncertainty~\cite{Behling2015}. Although the polarization
measurement in reference~\cite{Behling2015} was consistent with tests
using naturally occurring \isotope[41]{K}, the \emph{in situ}
measurement of \isotope[37]{K} polarization was limited by
statistics. We have recently taken data for a second measurement of
$A_\beta$ with the goal of a final uncertainty less than $0.5\%$; more
precise than any previous measurement in a nucleus. Once reaching this
level of precision, we will evaluate the prospects for an
even-more-precise measurement.

In our geometry shown in figure~\ref{fig:chamber}, the
$\beta$-asymmetry can be simply determined using:

\begin{equation}
\label{eq:aobs}
A_\beta=\frac{A_\mathrm{obs}}{P}=\frac{1}{P}\frac{r^\uparrow-r^\downarrow}{r^\uparrow+r^\downarrow}.
\end{equation}

Here, $A_\mathrm{obs}$ is the observed $\beta$-asymmetry as measured in the
nuclear detectors, $P$ is the nuclear polarization, and $r^\uparrow$
($r^\downarrow$) is the rate of positrons detected along (against) the
nuclear polarization direction. In forming the asymmetry in
equation~\ref{eq:aobs}, it is possible to use a symmetric pair of
detectors along a fixed polarization axis or to use a single
$\beta$-detector and periodically reverse the sign of the
polarization. In our case, we eliminate many systematic effects by
doing both, utilizing the ``super-ratio'' technique~\cite{Gay1992,
  Plaster2012}. In addition to the nuclear measurement of $A_\mathrm{obs}$, a
measurement of $A_\beta$ requires a precision measurement of the
degree of nuclear polarization, defined by:

\begin{equation}
\label{eq:nucpol}
\vec{P}=\frac{\langle\vec{I}\rangle}{I}.
\end{equation}

Here, $\vec{I}$ is the nuclear polarization vector and, $I=3/2$ is its
magnitude. Furthermore, since the nuclear spin is greater than $1/2$,
the atoms have additional internal degrees of freedom, proportional to
the next moment of the nuclear spin projection. We define the nuclear
alignment term as:

\begin{equation}
\label{eq:alipol}
T=\frac{I(I+1)-3\langle(\vec{I}\cdot\hat{i})^2\rangle}{I(2I-1)}
\end{equation}
where $\hat{i}$ is a unit vector in the direction of
$\vec{I}$. Although it does not contribute directly to the positron
asymmetry (see footnote 7 in reference~\cite{Jackson1957}), it does contribute
to angular correlations involving the neutrino momentum, which can be
inferred from the simultaneous measurement of the momentum of the
$\beta$ and nuclear recoil. Therefore, we include it here for
completeness.

\begin{figure*}[h!t]
\centering
  \includegraphics[width=0.55\textwidth,keepaspectratio]{cham_ag}
  \resizebox{0.44\textwidth}{!}{\input{chamber3.pstex_t}}
\caption{The main {\scshape Trinat}{} detection chamber. The red arrows on either
  panel indicate the direction of incoming light for both the MOT and
  OP lasers.  To polarize the atoms along the axis defined by
  scintillator and silicon strip detectors, which are opaque, the
  light is brought in at a $\SI{9.5}{\degree}$ angle with respect to
  normal incidence and reflected off of a thin mirror. These detectors
  are placed symmetrically along the vertical axis and are housed in a
  re-entrant flange which is separated from the vacuum by a thin Be
  foil.  Also visible are the water-cooled magnetic field coils which
  provide the Helmholtz (OP) and anti-Helmholtz (MOT) fields as well
  as the electrostatic hoops that generate a nearly uniform electric
  field.  The recoil microchannel plate detector (MCP) is at negative
  electric potential, while the electron MCP is at positive
  potential.}
\label{fig:chamber}
\end{figure*}

We have collected statistics for a measurement of $A_\mathrm{obs}$
with statistical uncertainty $\Delta A_\mathrm{obs}/A_\mathrm{obs} =
0.2\%$ and, therefore,
 must measure the nuclear polarization to a
similar level of precision so that the polarization measurement does
not dominate the final uncertainty.

To polarize the atoms, we optically pump them, with the MOT off, on
the $D_1$ transition with circularly polarized ($\sigma^\pm$) light.
This accumulates atoms in the state with $m_F=\pm F$ corresponding to
complete \emph{nuclear} polarization. Here, $\vec{F}=\vec{I}+\vec{J}$
where $\vec{J}$ is the atomic angular momentum.  To monitor the
polarization, we photoionize a small fraction of the atoms which have
been excited to the $P_{1/2}$ state by the optical pumping light. This
provides a cleaner signal with fewer trapped atoms compared to
monitoring the fluorescence. The total $P_{1/2}$ population is a
sensitive probe of the nuclear polarization because atoms must have
been excited from a partially polarized $S_{1/2}$ state with
$|m_F|<F$. Therefore, the $P_{1/2}$ population is related to the
number of partially polarized or unpolarized atoms. Finally, we fit a
numerical simulation of optical pumping to the photoion time spectrum
and deduce the nuclear polarization from the result.  A typical
simulation demonstrating the principle of the technique is shown in
figure~\ref{fig:principle}. In this paper, we present the results of
this nuclear polarization measurement, which is significantly more
precise than previous results with this
method~\cite{Melconian2007,Behling2015}.

\begin{figure}[h!t]
         \resizebox{\columnwidth}{!}{\input{polarization_principle.tex}}

         \caption{Simulated time evolution of optical pumping with
           $\sigma^{\pm}$ light on the $D_1$ transition.  The
           photoionization is observed and used to infer the nuclear
           polarization by comparing to a numerical simulation of
           optical pumping. As the rate of photoionization in the
           region $t\rightarrow\infty$ decreases, the degree of
           polarization increases towards unity.  The atoms are
           considered fully polarized after the optical pumping light
           has been on for $\SI{100}{\micro\second}$ (see
           section~\ref{results}). The parameter $s_3$ gives the
           degree of circular polarization and is defined in
           section~\ref{opticalpumpinglight}. The nuclear alignment
           term follows the same measurement strategy.}
        \label{fig:principle}

\end{figure}

\section{Experimental Methods}
\label{expmethods}
Nuclear $\beta$-decay and atom-trap experimental methods used by
{\scshape Trinat}{} are described in reference~\cite{Behr2014}.  Here, we will
describe the apparatus with a particular emphasis on the polarization
measurement.  First, we give a description of the entire apparatus in
section~\ref{generalDescription}. Following this,
sections~\ref{opticalpumpinglight} and~\ref{magneticfields} describe
the two depolarizing mechanisms that lead to $|P|<1$, and
section~\ref{photoionizationlight} describes the UV light used to
monitor the nuclear polarization.

\subsection{General Description}
\label{generalDescription}
Ions of the short-lived isotope \isotope[37]{K} are delivered from
ISAC, the radioactive beam facility at {\scshape Triumf}, and neutralized on a
hot zirconium foil~\cite{Melconian2005}.  The atoms are then collected
in a vapor-cell MOT in a preparation chamber with $0.1\%$
efficiency~\cite{Behr1997}. To suppress a background from untrapped
atoms, they are then transferred with $75\%$ efficiency by a
red-detuned pulsed laser ``push-beam'' to a second MOT where the
precision measurement takes place~\cite{Swanson1998}. The push beam is
controlled by a double-pass acousto-optic modulator (AOM) setup, is
turned on only briefly during atom transfers, and misses the second
trap by aiming the beam $\SI{1}{\centi\meter}$ above the measurement
trap's height except during atom transfers.

Since the MOT destroys any polarization, it must be turned off and on
rapidly so that there is sufficient time to optically pump the atoms
and collect polarized decay data while the previously confined atoms
expand ballistically.  The confining forces are then turned back on to
re-collect the atoms before the cloud's expansion causes a significant
loss of atoms from the trapping region.  The trapping beam itself is
switched off to less than $10^{-4}$ of its maximum value by turning
off the first-order diffracted beam from an AOM. Any remaining trap
light is from the tail of the zeroth-order beam,
$\SI{90}{\mega\hertz}$ (15 linewidths) off-resonance. The resulting
excitation is less than $2\times10^{-4}$ of the optical pumping light.

In order to rapidly eliminate the magnetic field used for trapping, an
AC-MOT is used~\cite{Harvey2008,Anholm2014}.  In this scheme, an AC
current is run through the anti-Helmholtz coils
(see figure~\ref{fig:chamber}) instead of the usual DC current.  The
resultant magnetic field produced by the coils varies sinusoidally in
time, as does the field that results from induced eddy currents in
nearby materials, though the two components differ in phase.  Then, in
order to minimize the residual magnetic field after shutting off the
MOT, the current through the coils is shut off when the combined
magnetic field is zero.  The optimal shutoff phase is a function of
chamber geometry and material, as well as the frequency of the AC
current~\cite{Harvey2008}.

In order to trap atoms in a sinusoidally varying magnetic field, it is
necessary to vary the polarization of the MOT's trapping beam as well.
This is achieved by the use of an electro-optic modulator, set to
adjust the trapping beam between two polarization states in phase with
the magnetic field, such that a confining force is produced at all
times.

Once the AC-MOT is off, the ballistically expanding atoms are
optically pumped with circularly polarized light on the $D_1$
($4S_{1/2}\rightarrow4P_{1/2}$) transition (see
figure~\ref{fig:levels}). Note that the atoms must be polarized along
the axis connecting a pair of opaque detectors as shown in
figure~\ref{fig:chamber}. In order to allow the light to propagate in
this direction, the light is brought in at a $\SI{19}{\degree}$ angle
with respect to the polarization axis and reflected off of a thin SiC
mirror before interacting with the atoms.

\begin{figure}[h!t]
\resizebox{\columnwidth}{!}{\input{energyLevels.pgf}}
\caption{The fine and hyperfine structure of \isotope[37]{K} showing
  the laser transitions relevant to optical pumping.  The natural
  linewidth of the $4P_{1/2}$ state is
  $\SI{6}{\mega\hertz}$. Circularly polarized light brought in along
  the vertical axis (see figure~\ref{fig:chamber}) and tuned to the
  $D_1$ transition pumps atoms into the $F=2$, $m_F=\pm2$ state,
  resulting in a very high cloud polarization.  The parameter $\Delta$
  gives the detuning from the $F=2\rightarrow F^\prime=2$ resonance
  and is different for the $\sigma^+$/$\sigma^-$ polarization states.
  The second frequency is detuned a fixed amount, $\Delta_{12}$, from
  this frequency and optically pumps atoms which occupy $F=1$ ground
  states.  Neither $\Delta$ nor $\Delta_{12}$ are shown to scale. The
  $\SI{355}{\nano\meter}$ light continually probes the excited state
  population by photoionizing atoms from the excited $P$ states, which
  are subsequently detected by the recoil MCP.}
\label{fig:levels}
\end{figure}

Furthermore, a static magnetic field, $B_z=\SI{2.3}{\gauss}$, is
applied along the quantization axis to break the degeneracy of the
Zeeman sublevels.  As a result of the optical pumping, atoms
accumulate in the
\mbox{$4S_{1/2}|F\!=\!2,m_F\!=\!\pm2\rangle$}
(fully stretched) state
depending on the sign of circular polarization.  This state
corresponds to complete atomic \emph{and} nuclear polarization.

To minimize systematic effects, the polarization state is reversed
every $\SI{16}{\second}$ and simultaneously a frequency shift of
$\Delta(\sigma^+)-\Delta(\sigma^-)=\SI{4}{\mega\hertz}$ (see
figure~\ref{fig:levels}) is applied. This is done in order to move
closer to the desired $m_F=\pm1\rightarrow m_{F^\prime}=\pm2$ transition
frequency while moving further from the unwanted
$m_F=\pm2\rightarrow m_{F^\prime}=\pm1$ transition, which can be excited by a
component of the optical pumping light circularly polarized with the
``wrong'' sign. Note that the sign of $B_z$ is not changed throughout
the experiment.

The nuclear polarization is measured by monitoring the total $P_{1/2}$
population of the atoms. Atoms that have been fully polarized are not
excited by the OP light and, therefore, remain in a fully stretched
$S_{1/2}$ ground state until the MOT light is switched back on. As
shown in figure~\ref{fig:principle}, observing a decrease in the
$P_{1/2}$ population implies an increase in $|P|$.

The $P_{1/2}$ population could be monitored by detecting the
fluorescence light as atoms de-excite to an $S_{1/2}$ state. However,
the collection efficiency of the first MOT as well as the flux of
\isotope[37]{K} delivered by ISAC limit the experiment to $\sim10^4$
atoms at a time. With this number of atoms, the time-resolved
fluorescence signal has a poor signal-to-noise ratio ($S/N$) and does
not provide a clean signal.

For this reason, we photoionize a small fraction of the atoms in the
$P_{1/2}$ state using UV light at $\SI{355}{\nano\meter}$ and pulsed
at a $\SI{10}{\kilo\hertz}$ repetition rate. The UV photons do not have
the energy necessary to photoionize atoms in the $S_{1/2}$ ground
state so that photoions are generated only from atoms that have been
excited to the $P_{1/2}$ state by the OP laser.

A uniform electric field generated by the series of electrostatic
hoops shown in figure~\ref{fig:chamber} sweeps the photoions onto the
microchannel plate (MCP) detector at negative electric potential where
they are observed in coincidence with the UV light. The MCP detector
is backed by a delay-line for position sensitivity. As a result, the
photoion spectrum shown in figure~\ref{fig:photoionspectra} is clean:
the photoions are well resolved both spatially and in time-of-flight.

\begin{figure}[h!t]
  \includegraphics[width=\textwidth,keepaspectratio]{photoion_spectrum_final}
  \caption{Photoion position and time-of-flight spectrum demonstrating
    the clean signal. The left panel shows events gated on the central
    time-of-flight peak while the right panel shows events requiring
    that the position is in the dense region at the center of the
    plate. The region of the plate nearly devoid of events has lower
    detection efficiency, but it does not affect the polarization
    measurement.}

\label{fig:photoionspectra}
\end{figure}

However, the photoionization rate has no sensitivity to the
distribution of the partially polarized atoms throughout the Zeeman
sublevels with $|m_F|<F$.  Although this population can be made quite
small, the precision measurement described here requires knowledge of
its distribution. There have been methods developed to probe this
directly~\cite{Wang2007,Smith2006,Julsgaard2004}, but the specific
constraints of our experiment, including the relatively low number of
trapped atoms, make these impractical.  Additionally, the polarization
measurement must be non-destructive, preserving the polarization of
the atoms in order to observe the $\beta$-asymmetry in the nuclear
decay of the same atoms.

To satisfy these requirements, we adopt the method of monitoring the
$P_{1/2}$ population with photoionization as described above and
modeling the sublevel distribution of the partially polarized atoms as
presented in section~\ref{Model}.  We emphasize that the $P_{1/2}$
population, inferred from the photoionization measurement, is directly
proportional to the total partially polarized population, and the
theoretical model must only determine the sublevel distribution of
this relatively small population.

In addition to the polarization described above, a measurement of
$A_\beta$ requires a simultaneous determination of the
$\beta$-asymmetry, $A_\mathrm{obs}$. The $\beta$-asymmetry is
measured by a pair of $\beta$-telescopes placed along the vertical
polarization axis. Although this arrangement requires an extra
reflection of the OP light, it allows the measurement of
$A_\mathrm{obs}$ to have the highest sensitivity. Each
$\beta$-telescope consists of a thin Si-strip detector backed by a
thick plastic scintillator.  The scintillator fully stops the
positrons from the \isotope[37]{K} decay
($Q_{\mathrm{EC}}=\SI{6.1}{\mega\electronvolt\per\clight\squared}$) and
records their full energy.  The Si-strip detector provides position
information and, due to its low efficiency for detecting
$\gamma$-rays, suppresses the background from
$\SI{511}{\kilo\electronvolt}$ annihilation radiation.  To identify
decays that occurred within the region of optical pumping, we detect
the low energy shake-off $e^-$ by sweeping it with an electric field
towards a microchannel plate detector and observing it in coincidence
with the $\beta^+$. This combination of detectors provides an
exceptionally clean signal, almost entirely free from backgrounds.

\medskip
Having described the apparatus generally, we now give a detailed
description of the elements necessary to produce highly polarized
nuclei and measure the degree of
polarization.

\subsection{Optical Pumping Light}
\label{opticalpumpinglight}

To obtain the highest polarization, both the $F=1$ and $F=2$ ground
states must be optically pumped. The two frequencies needed to
accomplish this are created by RF power injected directly into the
diode laser with the frequency close to the ground state hyperfine
splitting. We apply this standard technique~\cite{Kobayashi1982} at
relatively low RF power levels that produce light at about $1/2$ the
power of the carrier frequency and split from the carrier frequency by
the RF frequency.  This frequency is easily adjusted from the
hyperfine splitting of \isotope[41]{K} ($\SI{254}{\mega\hertz}$) to
\isotope[37]{K} ($\SI{240}{\mega\hertz}$) without changing the
alignment or beam spatial quality. The optical sideband strength is
monitored with a Fabry-Perot cavity and is stable in power to about
$10\%$.

The saturation spectroscopy and double-pass AOM setup shown in
figure~\ref{fig:diodelockswitch} allows frequency locking for either
\isotope[41]{K} or \isotope[37]{K}. The light is also detuned
$\SI{1}{\mega\hertz}$ with respect to the ground-state hyperfine
splitting to completely destroy dark state coherences~\cite{Gu2003}
(see section~\ref{cpt}). Following this, the light is divided into two
beams and injected into polarization-maintaining optical fibers. The
remainder of the optical path after exiting these optical fibers is
shown in figure~\ref{fig:optics}.

\begin{figure}[h!t]
\includegraphics[width=\textwidth,keepaspectratio]{diodelockswitch}
\caption{Optical pumping light and frequency locking scheme to maintain constant
  light profile with different isotopes.  The
  $F=2\rightarrow F^\prime=2$ and $F=1\rightarrow F^\prime=1$
  frequencies are generated by RF modulation of the diode laser
  current.  The OP light is turned on and off by changing the RF input
  frequency of an AOM (A), whose first-order diffraction is steered on
  and off an optical fiber (B).  That scheme, unlike turning the RF
  power on and off, keeps the AOM at near-constant temperature,
  avoiding steering and light profile distortion as the light is
  injected into the optical fiber; thus the light power is switched
  well from zero to full value without transients.  $10\%$ of the
  light is diverted to lock the laser frequency (C).  The light is
  shifted in frequency by a tunable double-pass AOM (D) before going
  to a vapor cell of potassium (E), allowing frequency locking either
  for naturally occurring \isotope[41]{K}, or for accelerator-produced
  \isotope[37]{K}, by referencing to Doppler-free Zeeman-dithered
  saturation absorption peaks of stable isotopes~\cite{Tanaka1992}.}
\label{fig:diodelockswitch}
\end{figure}

\begin{figure}[h!t]
\includegraphics[width=\textwidth,keepaspectratio]{poloptics}
\caption{Optical elements creating the circularly polarized $D_1$
  light. This arrangement is repeated for both OP arms. The
   liquid crystal variable retarder (LCVR) is used to control the sign
  of the circular polarization of the OP light and the LL01-780
  laser line filter is used to combine the OP and MOT
  light along the same optical path.}

\label{fig:optics}
\end{figure}

After exiting the optical fiber, the OP light passes through a
polarizing beam-splitter and contrast $5\times10^4$,
$\SI{25}{\milli\meter}$ diameter suspended silver nanoparticle linear
polarizer (CODIXX ColorPol VIS 700 BC4). This is shown in
figure~\ref{fig:optics}. Next, the polarization state is determined by
the voltage applied to a liquid crystal variable retarder which either
maintains the linear polarization or rotates it $\SI{90}{\degree}$.

Since the OP and MOT light must travel the same path through the
vacuum chamber, they are combined by an angle-tuned laser line
filter. This Semrock LL01-780 nominally transmits $98\%$ of
$\SI{766.49}{\nano\meter}$ OP light while reflecting $98\%$ of the
$\SI{769.9}{\nano\meter}$ MOT light at $\SI{20}{\degree}$
incidence. The transmission of OP light changes by $4\%$ between the
linear polarization states. The output of this feeds a high-quality
$1/4$-wave plate before being injected into the vacuum chamber. Note
that there are no lenses in the path after the polarizer, avoiding
position-dependent birefringence.

The quality of circular polarization is critical to the final nuclear
polarization achieved. Any component of the light with the ``wrong''
polarization removes atoms from the fully polarized state and drives
$|P|<1$. We parametrize the quality of circular polarization with the
normalized Stokes parameter:

\begin{equation}
\label{eq:s3}
s_{3}=\frac{\mathcal{I}_+-\mathcal{I}_-}{\mathcal{I}_++\mathcal{I}_-}
\end{equation}
where $\mathcal{I}_+$ ($\mathcal{I}_-$) is the laser intensity in the $\sigma^+$
($\sigma^-$) state.

The degree of linear polarization is measured in each polarization
state along both OP arms immediately before passing through the
atom-trap viewports and $s_3$ is determined for each case. However,
stress-induced birefringence in the viewport glass can change the
light ellipticity.  We characterize this birefringence by its effect
on $s_3$ as the light passes through the viewport. If the $s_3$
parameter of the incoming light is denoted $s_3^\mathrm{in}$, then this same
parameter for the light after it has passed through the viewport is
given by~\cite{Warner2014}:

\begin{equation}
\label{eq:deltaN}
s_3^\mathrm{out}=\sin(\arcsin(s_3^\mathrm{in})+\Delta n\,kL)
\end{equation}
where $\Delta n$ parametrizes the effect of the viewport, $k$ is the
wave number of the light and $L$ is the thickness of the viewport
glass.

We developed viewports to minimize $\Delta n$, replacing the elastomer
in a commercial viewport with PCTFE, which is compatible with
UHV~\cite{Warner2014}. We obtain $\Delta n=(-6\pm2)\times10^{-6}$ and
$(-2\pm1)\times10^{-6}$ for the two arms respectively. Although this
measurement is done with the viewports in air, we have measured the
cumulative effect of both viewports on $s_3$ both in air as well as
with the viewports under vacuum and observe no difference. This is
consistent with the pressure on the viewports having no effect on
$\Delta n$. The measured values for $s_3$ both before and after the
viewport are shown in table~\ref{tab:s3}.

\begin{table}\centering
\caption{Results of the measurement of the OP light polarization.The
  direct measurement of $s_{3}^\mathrm{in}$ is done before the
  viewport, and the value after the
 viewport ($s_3^\mathrm{out}$)
  includes a calculation of the effect of the
 birefringence in each
  viewport.}
\label{tab:s3}
  \input{s3_2014_table.tex}
\end{table}

After entering the vacuum chamber, the light must be reflected once as
shown in figure~\ref{fig:chamber}. The mirror used for this purpose is
coated with a commercial dielectric stack with $99.5\%$
reflectivity. We observe a change in the outgoing light's ellipticity
$|s_3^\prime-s_3|<10^{-4}$ at $\SI{9.5}{\degree}$ incidence.

The alignment of the optical pumping light, which defines the
polarization axis, is done at the two viewports; the mirror is fixed
with mechanical precision. The result is that the light is aligned to
$\Delta\theta=\SI{1}{\milli\radian}$ with respect to the vacuum
chamber and, therefore, to the detection axis. Since the
$\beta$-asymmetry (see~\cite{Jackson1957}, equation 2) is
proportional to $\cos\theta$, this produces a negligible error of
$5\times10^{-7}$.

\subsection{Magnetic Fields}
\label{magneticfields}
A second mechanism that can drive $|P|<1$ is a magnetic field
transverse to the optical pumping axis ($B_x$) that causes Larmor
precession out of the stretched state. We have carefully designed the
apparatus to minimize eddy currents once the AC-MOT is turned off,
which in turn produce a magnetic field. Non-magnetic materials such as
316L and 316LN grade stainless steel and titanium were used wherever
possible and the chamber welds were kept thin to minimize their
magnetic permeability. We measured the relative permeability of the
welds to be $<1.25$. The vacuum chamber has a large ($\SI{12}{\inch}$)
diameter to place potentially magnetic materials as far away from the
trapped atoms as reasonably possible. The nearest material to the atoms is the
set of electrostatic hoops shown in figure~\ref{fig:chamber} which
direct the photoions onto the MCP. These are made from SIGRADUR G
grade glassy carbon, a semiconductor with resistivity
$\SI{4500}{\micro\ohm\centi\meter}$, two orders of magnitude better
than stainless steel.

In order to cancel out magnetic fields that are constant on the time
scale of optical pumping, we arranged two pairs of magnetic trim coils
exterior to the vacuum chamber. By varying the DC current in these
coils, we were able to apply a transverse magnetic field to cancel
stray fields at the location of the trapped atoms.

To optimize these settings, we optically pumped \isotope[41]{K}, which
can be trapped in large numbers and has a similar hyperfine structure
to \isotope[37]{K}. We used the same system described in this section
except that we monitored the fluorescence directly rather than the
photoionization. Keeping all the laser parameters fixed, we scanned
the trim-coil current and observed the residual fluorescence after
optical pumping. The minimum residual fluorescence corresponds to the
optimal current setting which was also used for the \isotope[37]{K}
experiment.

Additionally, the AC quadrupole magnetic field is switched off before
the optical pumping begins but induces eddy currents in the
surrounding material, which in turn produce a magnetic field. Although
the purpose of using an AC-MOT is to reduce these eddy currents by
turning off the magnetic field when it is nearly zero, we used a Hall
probe to measure an initial residual field of
$\sim\SI{103}{\milli\gauss\per\centi\meter}$, which decays to a final
value of $\sim\SI{22}{\milli\gauss\per\centi\meter}$ with a time
constant of $\sim\SI{130}{\micro\second}$. Although this measurement
was done with one vacuum flange removed, it demonstrates both the
approximate size of this effect as well as the need, described in
section~\ref{results}, to wait until this field has completely decayed
away before starting optical pumping.

\subsection{Photoionization Light}
\label{photoionizationlight}
The $\SI{355}{\nano\meter}$ UV light that photoionizes the excited
atoms is circularly polarized and has a near-$\mathrm{TEM}_{00}$ mode
with a $1/e^2$ diameter of $\SI{12}{\milli\meter}$. It comes from a
commercial diode-pumped solid-state pulsed laser making
$\SI{0.5}{\nano\second}$ pulses at $\SI{10}{\kilo\hertz}$ repetition
rate. The light propagates at $\SI{35}{\degree}$ with respect to the
optical pumping axis. After interacting with the atoms, the UV light
is reflected along the same path in order to provide a second
opportunity to interact with the atoms with $\sim90\%$ of the original
intensity. Next the sign of the polarization is reversed, and the
light again interacts with the atoms twice, although with the third
pass now at $41\%$ of the original intensity. In total, the UV light
photoionizes about $1/10^6$ atoms per pulse. The effects of the UV
light polarization on the photoionization signal are discussed in
section~\ref{Model}.

Note that the cross-section of photoionization is on the order of
$\SI{1}{\mega\barn}$, while Rayleigh scattering has a cross-section
$10^6$ lower. Therefore, the $\SI{355}{\nano\meter}$ light is
effectively a passive probe that does not disturb the system. It
either photoionizes the atom, removing it from the population so its
subsequent less-polarized $\beta$-decay is not observed, or has
negligible probability of disturbing the polarization.

\section{Theoretical Model}
\label{Model}

Having described the experimental setup, we now describe the model
used to calculate the sublevel distribution of the small fraction of
atoms that are \emph{not} fully polarized. Although this population is
small, at the current level of precision, its distribution can impact
the nuclear polarization achieved.

Our theoretical optical pumping calculation is based on a
semi-classical approach using the density operator formalism, i.e.,
the standard optical Bloch equations with the phenomenological
spontaneous decay term $R(t)$

\begin{equation}
\label{eq:liouville}
\frac{d\rho(t)}{dt}=\frac{1}{i\hbar}[\mathcal{H}(t),\rho(t)]+R(t).
\end{equation}

We use the expressions of Tremblay and Jacques~\cite{Tremblay1990} and
extend their expressions to include the effects of two
counter-propagating beams. Because both of our frequencies come from
one laser, then are frequency shifted by an independent RF source
into two frequencies, we assume as in~\cite{Gu2003} that the
contribution of the laser linewidth to the ground-state relaxation
rate vanishes. We observed short timescale jitter of several hundred
Hertz in the RF sources and have, therefore, included a
$\SI{500}{\hertz}$ linewidth from RF sources in the ground-state
relaxation rate (see~\cite{Tremblay1990}, equation 2.37).  The external
$B$ field is included in Zeeman shifts of the magnetic sublevels.
Primarily, we consider an isotropic initial ground-state distribution,
but also consider an initial anisotropy as a systematic
uncertainty. The calculation was carried out by numerically solving
the density matrix equations, i.e., the 128 complex coupled
differential equations of the 16-level system of
figure~\ref{fig:levels}.  Additionally, an arbitrary transverse
magnetic field $B_x$, which can drive transitions with $\Delta F=0$,
$\Delta m_F=\pm1$, is included using the expressions
in~\cite{Renzoni2001}.

This model includes the two depolarizing mechanisms, discussed in
sections~\ref{opticalpumpinglight} and~\ref{magneticfields}, that lead
to $|P|<1$: ellipticity in the OP light and a transverse magnetic
field ($B_x$) which causes Larmor precession out of the stretched
state. Note that since we are pumping both ground state hyperfine
levels to prevent losses to the $F=1$ state, these are the \emph{only}
two depolarizing mechanisms. We also consider the false polarization
signal potentially produced by coherent population trapping (CPT) states
in section~\ref{cpt}.

The transverse magnetic field and $s_3$ are highly correlated when
observing the photoionization rate: both lead to a larger fraction of
unpolarized atoms and an increase in the photoionization rate.  When
this model is fit to the experimental data as described in
section~\ref{results}, either of these mechanisms, or any combination
of them, can equally well account for the observed steady-state
photoionization and are therefore highly correlated
($>98\%$). However, the unpolarized population is distributed
differently among the $|m_F|<F$ sublevels depending on the relative
importance of the light ellipticity and the transverse magnetic field
in driving atoms out of the stretched state. Since the unpolarized
atoms make a significant contribution to the average nuclear
polarization, the relative significance of these two depolarizing
mechanisms must be considered.

In order to correctly interpret the photoionization signal as a probe
of the total $P_{1/2}$ population, we must consider the relative
photoionization cross-sections of the magnetic
sublevels. Photoionization from the $P_{1/2}$ state populates outgoing
$s$- and $d$-wave photoelectrons with the cross-section proportional
to the square of radial ($\mathcal{R}$) and angular portions of the
matrix element connecting a pair of final and initial states. Since
the angular part does not depend on the details of the central
potential, it is well known. Using a single-electron model with a
parametric central potential, Aymar, Luc-Koenig, and Combet Farnoux
calculate the total cross-section for $s$- and $d$- wave
photoelectrons and their results are
$\mathcal{R}_d/\mathcal{R}_s\approx1.7$ at
$E_\gamma=\SI{760}{\milli\electronvolt}$~\cite{Aymar1976}.

Considering the off-axis propagation as well as the multiple passes of
the UV light (see section~\ref{photoionizationlight}), the total
photoionization cross-section changes by no more than $4\%$ in our
setup compared to the assumption that all states have an equal
probability to be photoionized. The polarization results change by
$<10^{-5}$ assuming a $50\%$ uncertainty on the ratio
$\mathcal{R}_d/\mathcal{R}_s$.

\section{Coherent Population Trapping}
\label{cpt}
The multi-level system of figure~\ref{fig:levels} can support coherent
population trapping (CPT) states on three distinct sets of
$\lambda$-atomic systems ($m_F=-1,0,1$). These states are especially
problematic for this measurement as atoms in these states are not
available to be photoionized and detected, exactly mimicking our
experimental signature for good polarization, while simultaneously
having $|P| < 1$. Although CPT states are adequately described by the
model of section~\ref{Model}, we describe both how their formation is
eliminated in our setup as well as the steps that we have taken to
verify this.

First, the OP light is retroreflected such that it interacts with the
atomic cloud twice: first propagating along $+\hat{z}$ and second
along $-\hat{z}$. Since these relative velocities are different for
the two passes, the relative Doppler shift of the light frequency
between the first and second pass greatly reduces the CPT effect in
all but the coldest atoms.

To verify that they are destroyed, we performed measurements with
\isotope[41]{K}. We measure the magnitude of the CPT state similarly
to~\cite{Gu2003} by optically pumping the atoms with $\Delta_{12}$ set
to intentionally create CPT states (see
figure~\ref{fig:levels}). After the atoms are optically pumped, we
switch the frequency of the $F=1 \rightarrow F^\prime=2$ laser away
from this resonance, destroying the CPT state and allowing the atoms
that had been trapped in this state to be optically pumped to the
$m_{F}=\pm F$ state, creating a second burst of photoionization. The
relative size of the two photoionization bursts is a measurement of
the CPT fraction.

We scan the OP frequency around the $m_F=0$ ground-state hyperfine
resonance as shown in figure~\ref{fig:cpt_bz} and observe that the CPT
resonance in our system has a FWHM of only
$\SI{19\pm4}{\kilo\hertz}$. We avoid this narrow resonance, as well as
the $m_F=\pm1$ resonances, during the polarization measurement by
setting $\Delta_{12}$ to be $\SI{1.1}{\mega\hertz}$ from the
ground-state hyperfine splitting. Simultaneously, since the resonant
CPT frequency is equal to the energy difference between the two
$m_F=0$ ground states, we use this to determine the aligned magnetic
field from the second-order Zeeman shift:
$B_z=\SI{2.339(10)}{\gauss}$.

\begin{figure}[h!t]
\includegraphics[width=\columnwidth]{bz_from_cpt}
\caption{The difference between the two OP frequencies, $\Delta_{12}$, is
  scanned near the ground state hyperfine resonance. The resonant
  frequency is $\SI{85}{\kilo\hertz}$ from the ground-state hyperfine
  splitting, providing a clean measurement of the aligned ($\hat{z}$)
  magnetic field. The width of this resonance is only
  $\SI{19}{\kilo\hertz}$ and, along with the $m_F=\pm1$ resonances, is
  carefully avoided during the \isotope[37]{K} experiment.}
\label{fig:cpt_bz}
\end{figure}

\section{Results}
\subsection{Photoionization Fits}
\label{results}
Figure~\ref{fig:withzoom} shows a typical photoionization curve
recorded during the experiment.  The MOT magnetic field and lasers are
switched off at $t=0$.  There was no MOT or OP light interacting with
the atoms until the OP light was turned on at
$t=t_{\mathrm{OP}}=\SI{332}{\micro\second}$. This was done in order
for the MOT magnetic field to die away as it would spoil the final
polarization as well as to give a long enough light-free region that
we use to measure backgrounds.

The atoms are fully polarized after $\SI{100}{\micro\second} $ and are
re-trapped by the MOT at $t=\SI{1906}{\micro\second}$ after expansion
from $\SI{2.0}{\milli\meter}$ to $\SI{4.5}{\milli\meter}$ FWHM.
Separate photoionization curves were recorded for the two polarization
states. This histogram is fit to the optical pumping calculation, and
the best-fit values are used to calculate the nuclear polarization and
alignment according to equations~\ref{eq:nucpol} and~\ref{eq:alipol}.

\begin{figure}[h!t]
\resizebox{\textwidth}{!}{\input{op_withZoom.pstex_t}}
\caption{Result of the fit to the $\sigma^-$ polarization state with
  $t_{\mathrm{OP}}=\SI{332}{\micro\second}$ and
  $E=\SI{535}{\volt\per\centi\meter}$.  The data is shown as the blue
  histogram and overlaid with the fit result in red. The nuclear
  polarization is shown in dark green and quickly approaches one as
  atoms accumulate in the stretched state.}
\label{fig:withzoom}
\end{figure}

We include a constant background rate in the fitting function.  In
order to separate this background from the residual photoionization
that results from unpolarized atoms, we extend the fitting region to
begin at $t=\SI{150}{\micro\second}$, before the optical pumping has
begun.  At this point, there is no light, either from the MOT or the
OP light, illuminating the atoms. Therefore, all events between this
point and $t_\mathrm{OP}$, when the optical pumping is turned on, are
considered background.  The primary source of background events are
random coincidences between the UV pulse and the $\beta$-decay of a
\isotope[37]{K} atom, delayed by the photoion time-of-flight.

Also at this point, the MOT magnetic field has mostly decayed away
while still leaving enough time before the OP light is turned on to
achieve a good statistical sensitivity on the background level.  We
also observed a defect in the event timing system which caused the
recorded time to be distributed around the actual event time with a
width of $\SI{1.0}{\micro\second}$ and include this in the fit.

The variable fitting parameters were a constant background rate
described above, which is parameterized by the average signal-to-noise
ratio ($S/N$) and the OP laser intensity in each polarization state
($\mathcal{I}^{\pm}$). Additionally, the constant transverse magnetic
field ($B_x$) and one parameter describing the laser frequencies were
used as free fitting parameters.  Of these, the nuclear polarization
depends strongly only on $B_x$. The light ellipticity also strongly
influences $P$, but this is not a free fitting parameter; it is fixed
to the values of $s_3^\mathrm{out}$ shown in table~\ref{tab:s3}. Although the
transverse magnetic field is minimized in the experiment by a pair of
orthogonal magnetic field coils, its absolute value at the atoms'
position has a complicated dependence on eddy currents in the vacuum
chamber and is difficult to determine reliably.  Therefore, it is best
fit directly to the experimental data as is done here.

Other parameters, including the laser frequencies, were held constant
during the fit.  Note that both
$\Delta(\sigma^-)-\Delta(\sigma^+)=\SI{4.0}{\mega\hertz}$ and
$\Delta_{12}=\SI{239.2}{\mega\hertz}$ are well defined experimentally
and the laser linewidth is $\SI{0.2}{\mega\hertz}$. Therefore, only one
overall parameter is required to describe the laser frequencies. We
determine this overall frequency by fixing the laser intensity in the
two polarization states such that $\mathcal{I}^+=\mathcal{I}^-$ and fitting the
photoionization data to obtain the best-fit value of
$\Delta(\sigma^-)=\SI{-2.8\pm0.2}{\mega\hertz}$, which is consistent
with the direct resonance
measurement~\cite{Behr1997,Rossi2015}. Finally, the magnetic field
($B_z$) is taken from the CPT resonance measurement described in
section~\ref{cpt}.

Throughout the data collection, we varied the time at which we turned
on the OP light
($t_{\mathrm{OP}}=$~\SIlist[list-units=single]{332;432;732}{\micro\second})
as well as the strength of the uniform electric field to collect
photoions
(\SIlist[list-units=single]{395;415;535}{\volt\per\centi\meter}).
This resulted in five distinct datasets (not every combination was
used).  Each dataset was independently fit with the binned maximum
likelihood method, this time \emph{not} requiring that
$\mathcal{I}^+=\mathcal{I}^-$, and the results for the nuclear
polarization calculated using the best-fit parameters are shown in
figure~\ref{fig:setbyset}.  The differences in statistical sensitivity
are a result of spending different amounts of time collecting data at
the various conditions. Since there is no significant difference among
datasets, we conclude that the polarization remained constant
throughout the roughly two weeks of data taking.

\begin{figure}[h!t]
\includegraphics[width=\columnwidth,keepaspectratio]{polarization_bySet}
\caption{The polarization we find as a result of fitting each set of
  data independently.  The cross-hatched region shows the $1\sigma$
  uncertainty on the polarization when combining the results of
  fitting each dataset this way. Note that the two polarization
states are not independent as the transverse magnetic field is the
same in both cases. Since there is no difference between sets, the
final result is fit to all datasets simultaneously.}
\label{fig:setbyset}
\end{figure}

Taking this into account, we performed the final analysis by fitting
each dataset simultaneously to one set of optical pumping
parameters. Since the gain of the recoil MCP detector fluctuated
throughout the run, each set was fit with an independent
signal-to-noise ratio representing a constant background in the
detector for a total of eight free fitting parameters ($\mathcal{I}^\pm$,
$B_x$, and $(S/N)_{A-E}$). The results are shown graphically in
figure~\ref{fig:global} and summarized in
table~\ref{tab:centralvalue}.

\begin{figure*}[h!t]
\includegraphics[width=\textwidth,keepaspectratio]{polarization_globalFit-bx}
\caption{Global fit result including a consistent set of
  parameters. The Stokes parameter, $s_3$, was fixed at its
  experimentally determined value.  A single transverse magnetic
  field, $B_x$,
 along with separate laser intensities for each
  polarization state were fit to the entire dataset.  The
  signal-to-noise ratios ($S/N_{\mathrm{A-E}}$) were allowed to vary
  independently for each of the five datasets.  Other parameters were
  fixed as indicated in the text. The binning for each dataset was
  chosen to be as fine as possible while producing roughly equal peak
  bin contents in each set. The effects of using a uniform binning are
  discussed in section~\ref{sysuncert}. The datasets shown here from
  top to bottom correspond to the conditions shown in
  figure~\ref{fig:setbyset} from left to right.}
\label{fig:global}
\end{figure*}

\begin{table*}\centering

  \caption{Results from the global polarization fit shown in
    figure~\ref{fig:global}.  The uncertainties listed here are purely
    statistical; the result of propagating the systematic
    uncertainties are discussed in the text. \label{tab:centralvalue}}
  \input{polarization_results.tex}
\end{table*}

The photoion spectra of figure~\ref{fig:global} indicate a slight
decrease in the partially polarized population even after the atoms
are considered fully polarized. This is a result of the AC-MOT
quadrupole field, and the eddy currents it creates, slowing decreasing
with time. The polarization results dividing the time when the atoms
are fully polarized into quadrants are shown in
figure~\ref{fig:timedepend}. All of the data collected with
$t_{\mathrm{OP}}=\SI{332}{\micro\second}$ is shown as this has the most
sensitivity to this effect. This figure indicates that the
polarization is improving even after $\SI{100}{\micro\second}$ of
optical pumping, although the magnitude of this effect is only
$\sim1\sigma$.  Keeping this in mind, we reiterate that the results shown
throughout this article represent the average polarization from
$t_{\mathrm{OP}}+\SI{100}{\micro\second}\rightarrow\SI{1906}{\micro\second}$.

\begin{figure}[h!t]
\resizebox{\textwidth}{!}{\input{polarization_timeDepend.tex}}
\caption{Time dependence of the nuclear polarization in the $\sigma^+$
  state. The shaded region shows the result with all the data
  considered while each point considers only the data in the indicated
  range in addition to the initial OP peak. The polarization is seen
  to slightly improve with time, indicating a gradual decrease in
  $B_x$.}
\label{fig:timedepend}
\end{figure}

\subsection{Systematic Uncertainties}
\label{sysuncert}
In this section, we discuss the systematic uncertainties in the
fitting routine and lay out the procedure we used to quantify them.

The final results are determined by performing a global fit to all
datasets at once.  However, it is also possible to find the weighted
average of results in figure~\ref{fig:setbyset} where each dataset is
fit independently.  The difference between these two analysis choices
gives a systematic uncertainty of $2\times10^{-4}$. Note that if we
fit each dataset independently, there are a total of twenty fitting
parameters: $\mathcal{I}^\pm$, $B_x$, and a $(S/N)$ for each of the
five datasets.  Therefore, the global fit is preferred simply because
it captures the same physics with fewer fitting parameters.

The uncertainty on the $s_3$ parameter is propagated to the final
result by varying the input $s_3^\mathrm{out}$ value by $\pm1\sigma$ and
comparing the results.  Although we do not expect the light's
polarization to be correlated in the two polarization states, we
conservatively treat them as though they are. This procedure gives the
most variation in the relative strengths of the two depolarizing
mechanisms, resulting in the largest difference in average nuclear
polarization.  Even with this conservative approach, the systematic
uncertainty is at most $2\times10^{-4}$ and does not limit the
measurement.

Next, the dependence of the results on the binning of the data is
studied by fitting the data with bins of width
\SIlist[list-units=single]{1;2;5;10;20}{\micro\second}. The central
value is taken from the fit with varying bin widths shown in
figure~\ref{fig:global} and we take the largest difference between any
choice of binning and this value as the systematic uncertainty.

As described in section~\ref{results}, we determine one overall
frequency by fitting the photoionization data with the requirement
that $\mathcal{I}^+=\mathcal{I}^-$. Since this requirement is only
approximately true, we relax this requirement when determining the
final results. However, we conservatively treat this condition as a
systematic uncertainty.

The magnetic field ($B_z$) has been measured by two independent
methods: the Hall probe technique described in
section~\ref{magneticfields} and the CPT field measurement described
in section~\ref{cpt}. Because the Hall probe measurement was performed
in air with one vacuum flange removed and without the presence of the
electrostatic hoops or MCP assembly, it is expected to be less
reliable than the CPT measurement. The results of these two
measurements differ by $\SI{180\pm20}{\milli\gauss}$, which is
significantly larger than the uncertainty of the CPT measurement
itself. Conservatively, this difference is treated as a systematic
uncertainty rather than propagating the smaller uncertainty on the CPT
measurement.

Finally, we allow a possible anisotropy in the initial ground-state sublevel
distribution of the atoms and characterize this by an initial
polarization $P_0$ and alignment $T_0$. We measure $P_0$ by observing
the $\beta$-asymmetry of the positrons emitted in the \isotope[37]{K}
decay before the optical pumping light is turned on. Comparing this to
the expected asymmetry ($A_\beta=-0.5706$), we conservatively measure
an initial polarization $|P_0|<0.022$. Including an initial population
distribution with this distribution produces a systematic uncertainty
of $1\times10^{-5}$.

However, $T_0$ does not produce a signal in the nuclear decay that we
can measure with the current setup.  In order to constrain this
possibility, we model the sublevel distribution of the MOT on the
$D_2$ ($F\!=\!2\rightarrow F^\prime\!=\!3$) transition. The vertically
($\hat{z}$) propagating beams combine to produce a linearly polarized standing
wave in the $x$-$y$ plane, while the orthogonal arms produce linearly
polarized standing waves in the $x$-$z$ and $y$-$z$ planes, which
represent a combination of linearly and circularly polarized light
along the $\hat{z}$ quantization axis. Since the atom velocities
are Doppler limited, their motion averages over the polarization
gradients of the resultant electric field. Each pair of $\sigma^{\pm}$
beams have equal power and the ratio of total power propagating along
$x\,$:$\,y\,$:$\,z$ is $2\,$:$\,2\,$:$\,1$ so that the effective
ratio of linearly to circularly polarized light is $3\,$:$\,2$.  Since
the AC-MOT is deliberately turned off with $B_z$ close to zero, we
adopt the value of $B_z=\SI{100}{\milli\gauss}$.  Since a transverse
magnetic field would only serve to decrease the anisotropy, we assume
that it is zero for this calculation. The resulting population
distribution has $T_0=0.03$.  Adopting a conservative uncertainty, we
constrain the maximum initial alignment to $T_0<0.06$ and compare the
results.  These systematic uncertainties are summarized in
table~\ref{tab:systematics}.

\begin{table*}\centering
\caption{Uncertainty budget for the nuclear polarization and alignment
  measurements. The largest systematic uncertainty arises from the potentially
  non-zero initial alignment ($T_0$) of the atoms, which we modeled as
  described in the text.  Also significant is the choice to perform a global
  fit rather than average the result of each dataset after a series
  of individual fits. The choice to prefer the global fit is justified
  by considering the lower number of fit parameters using this
  method.}
\label{tab:systematics}
\input{polarization_systematics.tex}
\end{table*}

At the current level of precision, the total systematic uncertainty is
of similar, but slightly smaller, magnitude as the statistical
uncertainty.  Since the model that is fit to the experimental data
only needs to account for the small contribution to the average
polarization from the unpolarized population, all of the uncertainties
as well as the statistical uncertainty can be reduced by improving
both the light polarization and further minimizing the transverse
magnetic field to reduce the unpolarized population that must be
modeled. The final results are:

\begin{equation}
\label{eq:finalfinal}
        \begin{aligned}
        P(\sigma^+)&=+0.9913(8) \\
        P(\sigma^-)&=-0.9912(8)
        \end{aligned}
        \qquad
        \begin{aligned}
        T(\sigma^+)&=-0.9770(22) \\
        T(\sigma^-)&=-0.9761(27)
        \end{aligned}
\end{equation}
 which represent an order of magnitude improvement compared to
 previous work~\cite{Melconian2007,Behling2015}.
\section{Discussion}

This nuclear polarization measurement is more precise than previous
measurements with the same technique and will not dominate the final
uncertainty on $A_\beta$ compared to the statistical uncertainty of
$\Delta A_\mathrm{obs}/A_\mathrm{obs} = 0.2\%$.  We note that the current
polarization measurement is limited primarily by statistics: the total
systematic uncertainty is only $5\times10^{-4}$. Therefore, we
conclude that future measurements can be sensitive enough to allow
correlation parameter measurements at the $0.1\%$ level without
significant changes to the techniques described here. In addition,
modest improvements to the apparatus will allow for an
even-more-precise measurement of the polarization in future
experiments.

Further increasing the light polarization and decreasing the
transverse magnetic field will both increase the average polarization
and decrease its uncertainty. With less unpolarized population to
model, the uncertainty about its distribution will lead to less
uncertainty on the nuclear polarization and alignment. Therefore, we
emphasize that improving the polarization will simultaneously improve
the precision that we can reach. Although we are continuing to
optimize the light polarization ($s_3^\mathrm{out}$), some optical elements,
particularly the liquid crystal variable retarder, preserve the
polarization better in one state than the other making it difficult to
optimize both polarization states simultaneously.  Improvements to the
trim coil system used to reduce the transverse magnetic field can also
reduce the polarization uncertainty. For example, if the magnetic
field is reduced to $1/2$ its current value and no other parameters
are changed, the statistical uncertainty is reduced by the same
factor. With careful measurements using \isotope[41]{K}, we
expect to be able to achieve this improvement.  In particular, there
is enough information from \isotope[41]{K} atoms to trim the gradient
of the magnetic field on each axis in addition to zeroing the average
value. Since we expect the systematic uncertainties to scale
similarly, it seems possible to achieve a polarization uncertainty of
$\sim0.04\%$ in upcoming measurements, allowing for an uncertainty of
$\sim0.1\%$ on the polarized correlation parameters.

\section{Conclusions}
In this paper, we have reported a precise \emph{in situ} measurement
of the nuclear polarization and alignment in optically pumped
\isotope[37]{K}. The same dataset used in these measurements contains
enough $\beta$-decay data to make a measurement of the
$\beta$-asymmetry ($A_\beta$) with an expected relative uncertainty of
$<0.5\%$. We will report these results in a future
publication. Furthermore, this work has demonstrated the capability to
measure the nuclear polarization to $<10^{-3}$, which motivates future
development towards measurements of polarized $\beta$-decay
correlations at this level of precision.

\section*{Acknowledgments}
We gratefully acknowledge the support staff of {\scshape Triumf}{} and ISAC and
thank S. Gensemer and A. Hatakeyama for alerting us to the effects of
CPT in optical pumping as well J. Zhang for help with weld
permeability measurements. This work was supported by the
U.S. Department of Energy under Grant No. DE-FG02-93ER40773 and Early
Career Award No. ER41747, by NSERC, by NRC through {\scshape Triumf}, and by the
Israel Science Foundation.


\section{Introduction}
Endowing machines with the ability to understand natural language
is a long-standing goal in NLP and holds the promise of
revolutionizing the way in which people interact with
machines and retrieve information. Richardson et al. \shortcite{richardson2013mctest}
proposed the task of machine comprehension, along with
MCTest, a question answering
dataset for evaluation. The ability of the machine to
understand text is evaluated
by posing a
series of questions, where the answer to each question can
be found only in the associated text. Solutions typically
focus on some semantic interpretation of the text, possibly
with some form of probabilistic or logic inference,
to answer the question. Despite intensive recent work
\cite{weston2014memory,weston2015towards,hermann2015teaching,sachan2015learning},
the problem is far from solved.

\begin{figure}[t]
\setlength{\abovecaptionskip}{-3pt}
\setlength{\belowcaptionskip}{-10pt}
\centering
\includegraphics[width=8.5cm]{IJCAI2016_example}
\caption{One example with 2 out of 4 questions in the MCTest. ``*" marks correct answer.} \label{fig:example}
\end{figure}

Machine comprehension is an open-domain question-answering
problem which contains factoid questions, but the answers
can be derived by extraction or induction of key
clues. Figure \ref{fig:example} shows one example in
MCTest. Each example consists of one document{}, four
associated questions; each question is followed by four
answer candidates in which only one is correct. Questions in
MCTest have two categories: ``one'' and ``multiple''. The
label means \emph{one} or \emph{multiple} sentences from
the document{} are required to answer this question. To
correctly answer the first question in the example, the two
blue sentences are required; for the second question
instead, only the red sentence can help.  The following
observations hold for the whole MCTest. (i) Most of the
sentences in the document{} are irrelavent for a given
question. It hints that we need to pay attention to just
some key regions. (ii) The answer candidates can be flexible
text in length and abstraction level, and probably do not
appear in the document{}. For example, candidate B for the
second question is ``outside'', which is one word and does
not exist in the document{}, while the answer candidates for
the first question are longer texts with some
auxiliary words like ``Because'' in the text. This requires
our system to handle flexible texts via \emph{extraction} as
well as \emph{abstraction}. (iii) Some questions require
multiple sentences to infer the answer, and those vital
sentences mostly appear close to each other (we call
them \emph{snippet{}}). Hence, our system should be able to
make a choice or compromise between potential
single-sentence clue and snippet{} clue.

Prior work on this task  is mostly based on feature engineering. This work, instead, takes the lead in presenting a deep neural network based approach without any linguistic features involved.

Concretely, we propose HABCNN, a hierarchical attention-based
convolutional neural network, to
address this task in two roadmaps. In the first one,
we
project the document
in two different ways, one based on question-attention,
one based on answer-attention and then compare the two projected document representations to
determine whether the answer matches the question.
In the second one, every question-answer pair is reformatted into a statement,
then the whole task is treated through textual entailment.

In both roadmaps, convolutional neural network (CNN) is
explored to model all types of text. As human beings usually
do for such a QA task, our model is expected to be able to
detect the key snippet{}s, key sentences, and key words or
phrases in the document{}. In order to detect those
informative parts required by questions, we explore an
attention mechanism to model the document{} so that its
representation contains required information intensively. In
practice, instead of imitating human beings in QA task
top-down, our system models the document{} bottom-up,
through accumulating the most relevant information from word
level to snippet{} level.

Our approach is novel in three aspects.
(i) A document{} is
modeled by a hierarchical CNN for different granularity,
from word to sentence level, then from sentence
to snippet{} level. The reason of choosing a CNN rather
than other sequence models like recurrent neural
network \cite{mikolov2010recurrent}, long short-term memory
unit (LSTM \cite{hochreiter1997long}), gated recurrent unit
(GRU \cite{cho2014learning}) etc, is that we argue CNNs are
more suitable to detect the key sentences within document{}s
and key phrases within sentences. Considering again the
second question in Figure \ref{fig:example}, the original
sentence ``They sat by the fire and talked about he
insects'' has more information than required, i.e, we do not
need to know ``they talked about the insects''. Sequence
modeling neural networks usually model the sentence meaning by
accumulating the whole sequence. CNNs, with
convolution-pooling steps, are supposed to detect some
prominent features no matter where the features come from.
(ii) In the example in Figure \ref{fig:example}, apparently
not all sentences are required given a question, and usually
different snippet{}s are required by different
questions. Hence, the same document{} should have different
representations based on what the question is. To this end,
attentions are incorporated into the hierarchical
CNN to guide the learning of dynamic document{}
representations which closely match the information
requirements by questions. (iii) Document representations
at sentence and snippet levels both are informative for
the question, a highway network is developed to combine
them, enabling our system to make a flexible tradeoff.

Overall, we make three contributions. (i) We
present a hierarchical attention-based CNN system
``HABCNN''. It is, to our knowledge, the first deep learning
based system for this MCTest task. (ii) Prior document{} modeling
systems based on deep neural networks mostly generate
generic representation, this work is the first to
incorporate attention so that document{} representation is
biased towards the question requirement. (iii) Our HABCNN
systems outperform other deep learning competitors by big
margins.

\begin{figure}[t]
\setlength{\abovecaptionskip}{-3pt}
\setlength{\belowcaptionskip}{-10pt}
\centering
\includegraphics[width=5.5cm]{IJCAI2016_routes}
\caption{Illustrations of HABCNN-QAP (top), HABCHH-QP (middle) and HABCNN-TE (bottom). Q, A, S: question, answer, statement; D: document{}} \label{fig:routes}
\end{figure}
\begin{figure*}[t]
\setlength{\abovecaptionskip}{-3pt}
\setlength{\belowcaptionskip}{-10pt}
\centering
\includegraphics[width=14cm]{IJCAI2016_MC}
\caption{HABCNN. Feature maps for phrase representations $\mathbf{p}_i$ and the max pooling steps that create sentence representations out of phrase representations are omitted for simplification. Each snippet covers three sentences in snippet-CNN. Symbols $\circ$ mean cosine similarity calculation.} \label{fig:abhcnn}
\end{figure*}

\section{Related Work}\label{sec:relatedwork}
Existing systems for MCTest task are mostly based on
manually engineered features. Representative work
includes \cite{narasimhan2015machine,sachan2015learning,wang2015machine,smith2015EMNLP}. In
these works, a common route is first to define a regularized
loss function based on assumed feature vectors, then the
effort focuses on designing effective features based on
various rules. Even though these researches are
groundbreaking for this task, their
flexibility and their capacity for generalization are limited.

Deep learning based approaches appeal to increasing interest
in analogous tasks. Weston et
al., \shortcite{weston2014memory} introduce memory networks
for factoid QA. Memory network framework is extended
in \cite{weston2015towards,kumar2015ask} for Facebook bAbI
dataset. Peng et al. \shortcite{PengLLW15}'s Neural
Reasoner infers over multiple supporting facts to
generate an entity answer for a given question and it
is also tested on bAbI. All of these works
deal with some short texts with simple-grammar, aiming
to \emph{generate} an answer which is restricted to be one
word denoting a location, a person etc.

Some works also tried over other kinds of QA tasks. For
example, Iyyer et al., \shortcite{iyyer2014neural} present
QANTA, a recursive neural network, to infer an entity based
on its description text. This task is basically a matching
between description and entity, no explicit question exist.
Another difference with us lies in that all the sentences in
the entity description actually contain partial information
about the entity, hence a description is supposed to have
only one representation. However in our task, the modeling
of document{} should be dynamically changed according to the
question analysis. Hermann et
al., \shortcite{hermann2015teaching} incorporate attention
mechanism into LSTM for a QA task over news text. Still,
their work does not handle some complex question types like
``Why...'', they merely aim to find the entity from
the document{} to fill the slot in the query so that the
completed query is true based on
the document{}. Nevertheless, it inspires us to treat
our task as a textual entailment problem by first
reformatting question-answer pairs into statements.

Some other deep learning systems are developed for answer
selection
task \cite{yu2014deep,yang2015wikiqa,severyn2015learning,shen2015question,wang2010modeling}. Differently,
this kind of question answering task does not
involve document{} comprehension. They only try to match the
question and answer candidate without any background
information. Instead, we treat machine comprehension in this work
as a question-answer matching problem under
background guidance.

Overall, for open-domain MCTest machine comprehension task, this work is the first to resort to deep neural networks.

\section{Model}\label{sec:model}
We investigate this task by three approaches, illustrated in
Figure \ref{fig:routes}. (i) We can  compute two
different document (D) representations in a common space,
one based on question (Q) attention, one based on answer (A)
attention, and compare them.
This architecture we name HABCNN-QAP. (ii)
We compute a  representation of D based on Q
attention (as before), but now we compare it directly with a
representation of A. We name this architecture HABCNN-QP.
(iii) We treat this QA task as textual entailment (TE), first
reformatting Q-A pair into a statement (S), then
matching S and D directly.
This architecture we name HABCNN-TE.
All three approaches are implemented in the common framework
HABCNN.

\subsection{HABCNN}
\seclabel{habcnn}
Recall that we use the abbreviations A (answer), Q
(question), S (statement), D (document).
HABCNN performs representation learning for triple (Q,
A, D) in HABCNN-QP and HABCNN-QAP, for tuple
(S, D) in HABCNN-TE. For convenience, we use
``query'' to refer to Q, A, or S uniformly. HABCNN, depicted
in Figure \ref{fig:abhcnn}, has the following phases.

\textbf{Input Layer.}
The input is (query,D). Query is two individual
sentences (for Q, A) or one single sentence (for S),
D is a sequence of sentences. Words are
initialized by $d$-dimensional pre-trained word embeddings. As
a result, each sentence is represented as a feature map with
dimensionality of $d \times s$ ($s$ is sentence length). In
Figure \ref{fig:abhcnn}, each sentence in the input layer is
depicted by a rectangle with multiple columns.

\textbf{Sentence-CNN.} Sentence-CNN is used for sentence representation learning from word level. Given a sentence  of length $s$ with a word sequence: $v_1,v_2,\ldots,v_s$,
let vector $\mathbf{c}_i\in\mathbb{R}^{wd}$ be the
concatenated embeddings of $w$ words
$v_{i-w+1},\ldots,v_{i}$ where $w$ is the filter width, $d$
is the dimensionality of word representations and $0< i
<s+w$.  Embeddings for words $v_i$, $i<1$ and $i>s$, are
zero padding.  We then generate the representation
$\mathbf{p}_i\in\mathbb{R}^{d_1}$ for the \textit{phrase}
$v_{i-w+1},\ldots,v_{i}$ using the convolution weights
$\mathbf{W}\in\mathbb{R}^{d_1\times wd}$:
\begin{equation}
\mathbf{p}_i=\mathrm{tanh}(\mathbf{W}\cdot\mathbf{c}_i+\mathbf{b})
\end{equation}
where bias $\mathbf{b}\in\mathbb{R}^{d_1}$. $d_1$ is called ``kernel size'' in CNN.

Note that the sentence-CNNs for query and all document sentences share the same weights, so that the resulting sentence representations are comparable.

\textbf{Sentence-Level Representation.}
The sentence-CNN generates a new feature map (omitted in
Figure \ref{fig:abhcnn}) for each input sentence, one
column in the feature map denotes a phrase representation
(i.e., $\mathbf{p}_i$ in Equation (1)).

For the query and each sentence of D, we do \emph{element-wise
1-max-pooling} (``max-pooling'' for
short) \cite{collobert2008unified} over phrase
representations to form their representations at this level.

We now treat D as a set of ``vital'' sentences
and ``noise'' sentences. We propose \emph{attention-pooling}
to learn the sentence-level representation of D as follows: first
identify vital sentences by computing attention for each
D's sentence  as the cosine similarity between the
its representation
and the query representation, then select the $k$
highest-attention sentences to do max-pooling over
them. Taking Figure \ref{fig:abhcnn} as an example, based on
the output of
sentence-CNN layer, $k=2$ important sentences with blue color are
combined by max-pooling as the sentence-level
representation $\mathbf{v}_s$ of D; the other -- white-color --
sentence representations are neglected as they have low
attentions. (If $k=all$, attention-pooling returns to the common
max-pooling in \cite{collobert2008unified}.) When the
query is (Q,A), this step will be repeated, once for Q, once
for A, to compute
representations of D at the sentence level
that are biased with respect to Q and A,
respectively.

\textbf{Snippet-CNN.}
As the example in Figure \ref{fig:example} shows, to answer
the first question ``why did Grandpa answer the door?'', it
does not suffice to compare this question
only to the  sentence ``Grandpa answered the door with a
smile and welcomed Jimmy inside''; instead, the snippet{}
``Finally, Jimmy arrived at Grandpa's house and
knocked. Grandpa answered the door with a smile and welcomed
Jimmy inside'' should be used to compare. To this end, it is
necessary to stack another CNN layer, \emph{snippet-CNN}, to learn representations
of snippet{}s, i.e., units of one or more sentences. Thus, the basic units
input to snippet-CNN (resp.\ sentence-CNN) are sentences (resp.\
words)
and the output is representations of snippets (resp.\ sentences).

Concretely, snippet-CNN puts all sentence representations
in column sequence as a feature map and
conducts another convolution operation over it. With
filter width $w$, this step generates
representation of snippet{} with $w$ consecutive
sentences. Similarly, we use the same CNN to learn
higher-abstraction query representations (just treating query as a document with only one sentence, so that the higher-abstraction query representation is in the same space with corresponding snippet representations).

\textbf{Snippet-Level Representation.}
For the output of snippet-CNN, each representation is
more abstract and denotes bigger granularity.
We apply the same attention-pooling process to snippet level representations:
attention values are computed as cosine similarities between query and snippets
and the snippets with the $k$ largest attentions are retained.
Max-pooling over the $k$ selected snippet representations
then creates the snippet-level  representation
$\mathbf{v}_t$ of D. Two
selected snippet{}s are shown  as red in Figure \ref{fig:abhcnn}.

\textbf{Overall Representation.}
Based on convolution layers at two different granularity, we
have derived query-biased representations of D at
sentence level (i.e., $\mathbf{v}_s$) as well as snippet{}
level (i.e., $\mathbf{v}_t$).  In order to create a flexible
choice for open Q/A, we develop a highway
network \cite{srivastava2015training} to combine the two
levels of representations as an overall
representation $\mathbf{v}_o$ of D:

\begin{equation}
\mathbf{v}_o=(1-\mathbf{h})\odot\mathbf{v}_s+\mathbf{h}\odot\mathbf{v}_t
\end{equation}
where highway network weights $\mathbf{h}$ are learned by
\begin{equation}
\mathbf{h}=\sigma(\mathbf{W}_h\mathbf{v}_s+\mathbf{b})
\end{equation}
where $\mathbf{W}_h\in\mathbb{R}^{d_1\times d_1}$. With the
same highway network, we can generate the overall query
representation, $\mathbf{r}_i$ in Figure \ref{fig:abhcnn},
by combining the two representations of the query at sentence
and snippet levels.

\subsection{HABCNN-QP \& HABCNN-QAP}
HABCNN-QP/QAP computes the representation of D as a
projection of D, either based on attention from Q or based on
attention from A. We hope that these two projections of the
document are close for a correct A and less close for an
incorrect A.
As we said in related
work, machine comprehension can be viewed as an answer
selection task \emph{using the document D as critical
background information}. Here, HABCNN-QP/QAP
do not compare Q and A directly, but they use Q and A to
filter the document differently, extracting what is critical
for the Q/A match by attention-pooling.
Then they match the two document representations in the
new space.

For ease of exposition, we have used the symbol
$\mathbf{v}_o$ so far, but
in HABCNN-QP/QAP we compute two different document representations:
$\mathbf{v}_{oq}$, for which
attention is computed with respect to Q; and
$\mathbf{v}_{oa}$ for which attention is computed with
respect to
A.
$\mathbf{r}_i$ also has two versions, one for Q: $\mathbf{r}_{iq}$, one for A: $\mathbf{r}_{ia}$.

HABCNN-QP and HABCNN-QAP make different use of
$\mathbf{v}_{oq}$.  HABCNN-QP compares $\mathbf{v}_{oq}$
with answer representation $\mathbf{r}_{ia}$.  HABCNN-QAP
compares $\mathbf{v}_{oq}$ with
$\mathbf{v}_{oa}$. HABCNN-QAP projects D twice, once based
on attention from Q, once based on attention from A
and compares the two
projected representations, shown in Figure \ref{fig:routes}
(top).  HABCNN-QP only utilizes the Q-based projection of D
and then compares the projected document with
the answer representation, shown in Figure \ref{fig:routes}
(middle).

\subsection{HABCNN-TE}
HABCNN-TE treats machine comprehension as \emph{textual
entailment}. We use the statements that are provided as part of
MCTest. Each statement corresponds to
a question-answer pair; e.g.,
the Q/A pair ``Why
did Grandpa answer the door?'' /
``Because he saw the insects''
(Figure \ref{fig:example})
is
reformatted into the statement ``Grandpa answered the door
because he saw the insects''.
The question answering task is then cast as: ``does the document
entail the statement?''

For HABCNN-TE, shown in Figure \ref{fig:routes} (bottom),
the input for Figure \ref{fig:abhcnn} is the pair (S,D).
HABCNN-TE tries to match the  S's representation $\mathbf{r}_i$ with the
D's representation  $\mathbf{v}_o$.

\section{Experiments}
\subsection{Dataset}
MCTest\footnote{http://requery.microsoft.com/mct} has two
subsets. MCTest-160 is a set of 160 items, each consisting
of a document, four questions followed by one correct anwer and three incorrect answers
(split into
70 train, 30 dev and 60 test) and  MCTest-500 a
set of 500  items (split into 300 train, 50 dev and 150
test).

\subsection{Training Setup and Tricks}
Our training objective is to minimize the following ranking loss function:
\begin{equation}
L(d,a^+, a^-)=\mathrm{max}(0, \alpha+S(d,a^-)-S(d,a^+))
\end{equation}
where $S(\cdot,\cdot)$ is a matching score between two representation vectors. Cosine similarity is used throughout. $\alpha$ is a constant.

For this common ranking loss, we also have two styles to utilize the data in view of each positive answer is accompanied with three negative answers. One is treating ($d$, $a^+$, $a^-_1$, $a^-_2$, $a^-_3$) as a training example, then our loss function can have three ``max()'' terms, each for a positive-negative pair; the other one is treating ($d$, $a^+$, $a^-_i$) as an individual training example. In practice, we find the second way works better. We conjecture that the second way has more training examples, and positive answers are repeatedly used to balance the amounts of positive and negative answers.

\textbf{Multitask learning}: Question typing is commonly
used and proved to be very helpful in QA tasks \cite{sachan2015learning}.
 Inspired, we stack a logistic regression layer over question representation $\mathbf{r}_{iq}$, with the purpose that this subtask can favor the parameter tuning of the whole system, and finally the question is better recognized and is able to find the answer more accurately.

To be specific, we classify questions into  12
classes: ``how'', ``how much'', ``how many'', ``what'',
``who'', ``where'', ``which'', ``when'', ``whose'', ``why'',
``will'' and ``\emph{other}''. The question label is
created by querying for the label keyword in the question.
If more than one keyword appears in a
question, we adopt the one appearing earlier and the more
specific one (e.g., ``how much'', not ``how''). In case
there is no match, the class
``\emph{other}'' is assigned.

We train with AdaGrad \cite{duchi2011adaptive} and
use 50-dimensional GloVe \cite{pennington2014glove} to
initialize word
representations,\footnote{http://nlp.stanford.edu/projects/glove/}
kept fixed during training.
Table \ref{tab:hyper} gives hyperparameter
values, tuned on dev.
\begin{table}[t]
\setlength{\abovecaptionskip}{0pt}
\setlength{\belowcaptionskip}{-5pt}
\begin{center}
\setlength{\tabcolsep}{2mm}
\begin{tabular}{ccccccc}
 $k$& lr & $d_1$ & bs & $w$ & $L_2$ & $\alpha$\\\hline
 [1,3]&0.05 & [90, 90] & 1 & [2,2] & 0.0065 & 0.2\\
\end{tabular}
\end{center}
\caption{Hyperparameters. $k$: top-$k$ in attention-pooling for both CNN layers; lr: learning rate; $d_1$: kernel size in CNN layers; bs: mini-batch size; $w$: filter width; $L_2$: $L_2$ normalization; $\alpha$: constant in loss function.}\label{tab:hyper}
\end{table}
\begin{table*}[htbp]
\setlength{\abovecaptionskip}{0pt}
\setlength{\belowcaptionskip}{-10pt}
\begin{center}
\begin{tabular}{ll|rrr|rrr|rrr|rrr}
&&\multicolumn{6}{c|}{MCTest-150}  & \multicolumn{6}{c}{MCTest-500} \\
\multicolumn{2}{c|}{method} &  \multicolumn{3}{c|}{acc}  & \multicolumn{3}{c|}{NDCG$_4$} &\multicolumn{3}{c|}{acc}  & \multicolumn{3}{c}{NDCG$_4$} \\
& & one&mul&all & one&mul&all&one&mul&all & one&mul&all\\\hline
\multirow{4}{*}{\rotatebox{90}{Baselines}}&Addition & 39.3 & 32.4 & 35.7 & 60.4 & 50.3 & 54.6 & 35.7 & 30.2 &32.9 & 56.6 & 55.2 & 55.8\\
&Addition-proj & 42.1 & 38.7  & 40.3 & 65.3 & 61.3 & 63.2 & 39.4 & 36.7& 38.0 & 63.3& 60.1& 61.7\\
&AR & 48.1  & 44.7 & 46.3 & 70.5 & 68.9 & 69.6 & 44.4 &39.5 &41.9 & 66.7 & 64.2&65.4\\
& NR & 48.4 & 46.8 & 47.6 & 70.7 & 68.2 & 69.7 &45.7 & 45.6 &45.6 & 71.9 & 69.5 &70.6\\\hline
\multicolumn{2}{c|}{Variant-I}&50.4 & 47.7 & 49.0& 73.9& 71.2& 72.5 & 45.4& 42.1&43.7&68.8&64.5&66.6\\
\multicolumn{2}{c|}{Variant-II}& 53.6 & 51.0 & 52.2 & 74.6 & 70.1 & 72.3&47.1 &46.0&46.5 &70.2&63.8&66.9\\\hline
\multicolumn{2}{c|}{HABCNN-QP}&57.9 & 53.7 & 55.7& 80.4& 80.0& 80.2 & 53.7& 46.7&50.1&75.4&72.7&74.0\\
\multicolumn{2}{c|}{HABCNN-QAP}& 59.0 & 57.9 & 58.4 & 81.5 & 79.9 & 80.6&54.0 &47.2&50.6 &75.7&72.6&74.1\\
\multicolumn{2}{c|}{HABCNN-TE}& \textbf{63.3} & \textbf{62.9}& \textbf{63.1}&\textbf{86.6} & \textbf{85.9}& \textbf{86.2}&\textbf{54.2}&\textbf{51.7}&\textbf{52.9}&\textbf{76.1}&\textbf{74.4}&\textbf{75.2}\\
\end{tabular}
\end{center}
\caption{Experimental results for one-sentence (one),
multiple-sentence (mul) and all cases.}\label{tab:overallresult}
\end{table*}

\begin{figure*}[t]
\setlength{\abovecaptionskip}{0pt}
\setlength{\belowcaptionskip}{-5pt}
\centering
\includegraphics[width=14cm]{IJCAI2016_visualization}
\caption{Attention visualization for statement ``Grandpa answered the door because Jimmy knocked'' in the example Figure \ref{fig:example}. Too long sentences are truncated with ``$\ldots$''. Left is attention weights for each single sentence after the sentence-CNN, right is attention weights for each snippet{} (two consecutive sentences as filter width $w=2$ in Table \ref{tab:hyper}) after the snippet-CNN. } \label{fig:visualization}
\end{figure*}

We consider two evaluation metrics: \emph{accuracy} (proportion of questions correctly answered) and \emph{NDCG$_4$} \cite{jarvelin2002cumulated}. Unlike accuracy which evaluates if the question is correctly answered or not, NDCG$_4$, being a measure of ranking quality, evaluates the position of the correct answer in our predicted ranking.

\subsection{Baseline Systems}
This work focuses on the comparison with systems about distributed representation learning and deep learning:

\textbf{Addition.} Directly compare question and answers without considering the D. Sentence representations are computed by element-wise addition over word representations.

\textbf{Addition-proj.} First compute sentence
representations for Q, A and all D sentences as the same way as \emph{Addition}, then match the two sentences in D which have highest similarity with Q and A respectively.

\textbf{NR.}
The Neural Reasoner \cite{PengLLW15} has an encoding layer,
multiple reasoning layers and a final answer layer. The
input for the encoding layer is a question and the sentences
of the document (called facts); each
sentence is encoded by a GRU into a vector. In each reasoning
layer, NR lets the question representation interact with
each fact representation as reasoning process. Finally,
all temporary reasoning clues are pooled as answer
representation.

\textbf{AR.}
The Attentive Reader \cite{hermann2015teaching} is
implemented by modeling the whole D as a word
sequence -- without specific sentence /
snippet{} representations -- using an LSTM.
Attention mechanism is
implemented at word representation level.

Overall, baselines Addition and Addition-proj do not involve complex
composition and inference. NR and AR represent the
top-performing deep neural networks in QA tasks.

\subsection{HABCNN Variants}
In addition to the main architectures described above, we
also explore
two variants of ABCHNN, inspired by \cite{PengLLW15} and  \cite{hermann2015teaching}, respectively.

\textbf{Variant-I}:
As RNNs are widely recognized as a competitor of CNNs in
sentence modeling, similar with \cite{PengLLW15}, we replace
the sentence-CNN in Figure \ref{fig:abhcnn}  by a GRU while keeping other parts unchanged.

\textbf{Variant-II}:
How to
model attention
at the granularity of \emph{words}
was shown in
\cite{hermann2015teaching}; see their paper for details.
We develop their attention idea and model attention at the
granularity of \emph{sentence and snippet}.
Our attention gives different
weights to  sentences/snippets (not words), then
computes the document representation as a weighted average
of all sentence/snippet representations.

\subsection{Results}
Table \ref{tab:overallresult} lists the performance of
baselines, HABCNN-TE variants, HABCNN systems in the first,
second and last block, respectively (we only report variants
for top-performing HABCNN-TE).  Consistently, our HABCNN
systems outperform all baselines, especially surpass the two
competitive deep learning based systems AR and NR.
The
margin between our best-performing ABHCNN-TE and NR is
15.6/16.5 (accuracy/NDCG) on MCTest-150 and
7.3/4.6 on
MCTest-500. This
demonstrates the promise of our architecture in this task.

As said before, both AR and NR systems aim to \emph{generate} answers in  entity form.  Their designs might not suit this machine comprehension task, in which the answers are openly-formed based on summarizing or abstracting the clues. To be more specific, AR models D always at word level, attentions are also paid to corresponding word representations, which is applicable for entity-style answers, but is less suitable for comprehension at sentence level or even snippet{} level.  NR contrarily models D  in sentence level always, neglecting the discovering of key phrases which however compose most of answers. In addition, the attention of AR system and the question-fact interaction  in NR system both bring large numbers of parameters, this potentially constrains their power in a dataset of limited size.

For Variant-I and Variant-II (second block of
Table \ref{tab:overallresult}), we can see that both
modifications do harm to the original HABCNN-TE
performance. The first variant, i.e, replacing the
sentence-CNN in Figure \ref{fig:abhcnn} as GRU module is not
helpful for this task. We suspect that this lies in the
fundamental function of CNN and GRU. The CNN models a
sentence without caring about the global word order
information, and max-pooling is supposed to extract the
features of key phrases in the sentence no matter where the
phrases are located. This property should be useful for
answer detection, as answers are usually formed by
discovering some key phrases, not all words in a sentence
should be considered. However, a GRU models a sentence by
reading the words sequentially, the importance of phrases is
less determined by the question requirement. The second
variant, using a more complicated attention scheme to model
biased D representations than simple cosine
similarity based attention used in our model, is less
effective to detect truly informative sentences
or snippet{}. We doubt such kind of attention scheme when
used in sentence sequences of large size. In training, the
attention weights after softmax normalization have actually
small difference across sentences, this means the system can
not distinguish key sentences from noise sentences effectively. Our
cosine similarity based attention-pooling, though pretty
simple, is able to filter noise sentences more effectively,
as we only pick top-$k$ pivotal sentences to form D
representation finally. This trick makes the system simple
while effective.

\subsection{Case Study and Error Analysis}
In Figure \ref{fig:visualization}, we visualize the attention distribution at sentence level as well as snippet{} level for the statement `` Grandpa answered the door because Jimmy knocked'' whose corresponding question requires multiple sentences to answer. From its left part, we can see that ``Grandpa answered the door with a smile and welcomed Jimmy inside'' has the highest attention weight. This meets the intuition that this sentence has semantic overlap with the statement. And yet this sentence does not contain the answer. Look further the right part, in which the CNN layer over sentence-level representations is supposed to extract high-level features of snippet{}s. In this level, the highest attention weight is cast to the best snippet{} ``Finally, Jimmy arrived...knocked. Grandpa answered the door...''. And the neighboring snippet{}s also get relatively higher attentions than other regions.  Recall that our system chooses the one sentence with top attention at left part and choose top-3 snippet{}s at right part (referring to $k$ value in Table \ref{tab:hyper}) to form D representations at different granularity, then uses a highway network to combine both representations as an overall D representation. This visualization hints that our architecture provides a good way for a question to compromise key information from different granularity.

We also do some preliminary error analysis. One big obstacle for our systems is the ``how many'' questions. For example, for question ``how many rooms did I say I checked?'' and the answer candidates are four digits ``5,4,3,2'' which never appear in the D, but require the counting of some locations. However, these digital answers can not be modeled well by distributed representations so far. In addition, digital answers also appear for ``what'' questions, like ``what time did...''. Another big limitation lies in ``why'' questions. This question type requires complex inference and long-distance dependencies. We observed that all deep lerning systems, including the two baselines, suffered somewhat from it.

\section{Conclusion}
This work takes the lead in presenting a CNN based neural network system for open-domain machine comprehension task. Our systems tried to solve this task in a document{} projection way as well as a textual entailment way. The latter one demonstrates slightly better performance. Overall, our architecture, modeling dynamic document representation by attention scheme from sentence level to snippet{} level, shows promising results in this task. In the future, more fine-grained representation learning approaches are expected to model complex answer types and question types.

\bibliographystyle{named}