public static void testRecoveryAndDoubleExecution(
    final ProcedureExecutor<MasterProcedureEnv> procExec, final long procId, final int lastStep,
    final boolean expectExecRunning) throws Exception {
    ProcedureTestingUtility.waitProcedure(procExec, procId);
    assertEquals(false, procExec.isRunning());

    // Restart the executor and execute the step twice
    // execute step N - kill before store update
    // restart executor/store
    // execute step N - save on store
    // NOTE: currently we make assumption that states/ steps are sequential. There are already
    // instances of a procedures which skip (don't use) intermediate states/ steps. In future,
    // intermediate states/ steps can be added with ordinal greater than lastStep. If and when
    // that happens the states can not be treated as sequential steps and the condition in
    // following while loop needs to be changed. We can use euqals/ not equals operator to check
    // if the procedure has reached the user specified state. But there is a possibility that
    // while loop may not get the control back exaclty when the procedure is in lastStep. Proper
    // fix would be get all visited states by the procedure and then check if user speccified
    // state is in that list. Current assumption of sequential proregression of steps/ states is
    // made at multiple places so we can keep while condition below for simplicity.
    Procedure<?> proc = procExec.getProcedure(procId);
    int stepNum = proc instanceof StateMachineProcedure
      ? ((StateMachineProcedure) proc).getCurrentStateId()
      : 0;
    for (;;) {
      if (stepNum == lastStep) {
        break;
      }
      LOG.info("Restart " + stepNum + " exec state=" + proc);
      ProcedureTestingUtility.assertProcNotYetCompleted(procExec, procId);
      restartMasterProcedureExecutor(procExec);
      ProcedureTestingUtility.waitProcedure(procExec, procId);
      // Old proc object is stale, need to get the new one after ProcedureExecutor restart
      proc = procExec.getProcedure(procId);
      stepNum = proc instanceof StateMachineProcedure
        ? ((StateMachineProcedure) proc).getCurrentStateId()
        : stepNum + 1;
    }

    assertEquals(expectExecRunning, procExec.isRunning());
  }
Run through all procedure flow states TWICE while also restarting procedure executor at each
step; i.e force a reread of procedure store.
<p>
It does
<ol>
<li>Execute step N - kill the executor before store update
<li>Restart executor/store
<li>Execute step N - and then save to store
</ol>
<p>
This is a good test for finding state that needs persisting and steps that are not idempotent.
Use this version of the test when a procedure executes all flow steps from start to finish.
@see #testRecoveryAndDoubleExecution(ProcedureExecutor, long)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureTestingUtility.java