public static <TEnv> void testRecoveryAndDoubleExecution(final ProcedureExecutor<TEnv> procExec,
    final long procId, final boolean expectFailure, final Runnable customRestart) throws Exception {
    Procedure proc = procExec.getProcedure(procId);
    waitProcedure(procExec, procId);
    assertEquals(false, procExec.isRunning());
    for (int i = 0; !procExec.isFinished(procId); ++i) {
      proc = procExec.getProcedure(procId);
      LOG.info("Restart " + i + " exec state: " + proc);
      if (customRestart != null) {
        customRestart.run();
      } else {
        restart(procExec);
      }
      waitProcedure(procExec, procId);
    }

    assertEquals(true, procExec.isRunning());
    if (expectFailure) {
      assertProcFailed(procExec, procId);
    } else {
      assertProcNotFailed(procExec, procId);
    }
  }
Run through all procedure flow states TWICE while also restarting procedure executor at each
step; i.e force a reread of procedure store.
<p>
It does
<ol>
<li>Execute step N - kill the executor before store update
<li>Restart executor/store
<li>Execute step N - and then save to store
</ol>
<p>
This is a good test for finding state that needs persisting and steps that are not idempotent.
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java