private String chooseServerPrincipal(InputStream inStream, OutputStream outStream)
    throws IOException {
    Set<String> serverPrincipals = getServerPrincipals();
    if (serverPrincipals.size() == 1) {
      return serverPrincipals.iterator().next();
    }
    // this means we use kerberos authentication and there are multiple server principal candidates,
    // in this way we need to send a special preamble header to get server principal from server
    Call securityPreambleCall = createSecurityPreambleCall(r -> {
    });
    outStream.write(RpcClient.SECURITY_PREAMBLE_HEADER);
    readResponse(new DataInputStream(inStream), calls, securityPreambleCall, remoteExc -> {
      synchronized (this) {
        closeConn(remoteExc);
      }
    });
    if (securityPreambleCall.error != null) {
      LOG.debug("Error when trying to do a security preamble call to {}", remoteId.address,
        securityPreambleCall.error);
      if (ConnectionUtils.isUnexpectedPreambleHeaderException(securityPreambleCall.error)) {
        // this means we are connecting to an old server which does not support the security
        // preamble call, so we should fallback to randomly select a principal to use
        // TODO: find a way to reconnect without failing all the pending calls, for now, when we
        // reach here, shutdown should have already been scheduled
        throw securityPreambleCall.error;
      }
      if (IPCUtil.isSecurityNotEnabledException(securityPreambleCall.error)) {
        // server tells us security is not enabled, then we should check whether fallback to
        // simple is allowed, if so we just go without security, otherwise we should fail the
        // negotiation immediately
        if (rpcClient.fallbackAllowed) {
          // TODO: just change the preamble and skip the fallback to simple logic, for now, just
          // select the first principal can finish the connection setup, but waste one client
          // message
          return serverPrincipals.iterator().next();
        } else {
          throw new FallbackDisallowedException();
        }
      }
      return randomSelect(serverPrincipals);
    }
    return chooseServerPrincipal(serverPrincipals, securityPreambleCall);
  }
If multiple clients with the same principal try to connect to the same server at the same time,
the server assumes a replay attack is in progress. This is a feature of kerberos. In order to
work around this, what is done is that the client backs off randomly and tries to initiate the
connection again. The other problem is to do with ticket expiry. To handle that, a relogin is
attempted.
<p>
The retry logic is governed by the {@link SaslClientAuthenticationProvider#canRetry()} method.
Some providers have the ability to obtain new credentials and then re-attempt to authenticate
with HBase services. Other providers will continue to fail if they failed the first time -- for
those, we want to fail-fast.
</p>
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java