private void handleSaslConnectionFailure(final int currRetries, final int maxRetries,
    final Exception ex, final UserGroupInformation user, final String serverPrincipal)
    throws IOException, InterruptedException {
    closeSocket();
    user.doAs(new PrivilegedExceptionAction<Object>() {
      @Override
      public Object run() throws IOException, InterruptedException {
        // A provider which failed authentication, but doesn't have the ability to relogin with
        // some external system (e.g. username/password, the password either works or it doesn't)
        if (!provider.canRetry()) {
          LOG.warn("Exception encountered while connecting to the server " + remoteId.getAddress(),
            ex);
          if (ex instanceof RemoteException) {
            throw (RemoteException) ex;
          }
          if (ex instanceof SaslException) {
            String msg = "SASL authentication failed."
              + " The most likely cause is missing or invalid credentials.";
            throw new RuntimeException(msg, ex);
          }
          throw new IOException(ex);
        }

        // Other providers, like kerberos, could request a new ticket from a keytab. Let
        // them try again.
        if (currRetries < maxRetries) {
          LOG.debug("Exception encountered while connecting to the server " + remoteId.getAddress(),
            ex);

          // Invoke the provider to perform the relogin
          provider.relogin();

          // Get rid of any old state on the SaslClient
          disposeSasl();

          // have granularity of milliseconds
          // we are sleeping with the Connection lock held but since this
          // connection instance is being used for connecting to the server
          // in question, it is okay
          Thread.sleep(ThreadLocalRandom.current().nextInt(reloginMaxBackoff) + 1);
          return null;
        } else {
          String msg = "Failed to initiate connection for "
            + UserGroupInformation.getLoginUser().getUserName() + " to " + serverPrincipal;
          throw new IOException(msg, ex);
        }
      }
    });
  }
If multiple clients with the same principal try to connect to the same server at the same time,
the server assumes a replay attack is in progress. This is a feature of kerberos. In order to
work around this, what is done is that the client backs off randomly and tries to initiate the
connection again. The other problem is to do with ticket expiry. To handle that, a relogin is
attempted.
<p>
The retry logic is governed by the {@link SaslClientAuthenticationProvider#canRetry()} method.
Some providers have the ability to obtain new credentials and then re-attempt to authenticate
with HBase services. Other providers will continue to fail if they failed the first time -- for
those, we want to fail-fast.
</p>
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java