@Override
  public void completionCleanup(final Procedure proc) {
    if (isTableProcedure(proc)) {
      TableProcedureInterface tableProc = (TableProcedureInterface) proc;
      if (shouldWaitBeforeEnqueuing(tableProc)) {
        schedLock();
        try {
          TableProcedureWaitingQueue waitingQueue =
            tableProcsWaitingEnqueue.get(tableProc.getTableName());
          if (waitingQueue != null) {
            Optional<Procedure<?>> nextProc = waitingQueue.procedureCompleted(proc);
            if (nextProc.isPresent()) {
              // enqueue it
              Procedure<?> next = nextProc.get();
              LOG.debug("{} completed, enqueue a new procedure {}", proc, next);
              doAdd(tableRunQueue, getTableQueue(tableProc.getTableName()), next, false);
            } else {
              if (waitingQueue.isEmpty()) {
                // there is no waiting procedures in it, remove
                tableProcsWaitingEnqueue.remove(tableProc.getTableName());
              }
            }
          } else {
            // this should not happen normally, warn it
            LOG.warn("no waiting queue while completing {}, which should not happen", proc);
          }
        } finally {
          schedUnlock();
        }
      }
      boolean tableDeleted;
      if (proc.hasException()) {
        Exception procEx = proc.getException().unwrapRemoteException();
        if (tableProc.getTableOperationType() == TableOperationType.CREATE) {
          // create failed because the table already exist
          tableDeleted = !(procEx instanceof TableExistsException);
        } else {
          // the operation failed because the table does not exist
          tableDeleted = (procEx instanceof TableNotFoundException);
        }
      } else {
        // the table was deleted
        tableDeleted = (tableProc.getTableOperationType() == TableOperationType.DELETE);
      }
      if (tableDeleted) {
        markTableAsDeleted(tableProc.getTableName(), proc);
      }
    } else if (proc instanceof PeerProcedureInterface) {
      tryCleanupPeerQueue(getPeerId(proc), proc);
    } else if (proc instanceof ServerProcedureInterface) {
      tryCleanupServerQueue(getServerName(proc), proc);
    } else if (proc instanceof GlobalProcedureInterface) {
      tryCleanupGlobalQueue(getGlobalId(proc), proc);
    } else {
      // No cleanup for other procedure types, yet.
      return;
    }
  }
ProcedureScheduler for the Master Procedures. This ProcedureScheduler tries to provide to the
ProcedureExecutor procedures that can be executed without having to wait on a lock. Most of the
master operations can be executed concurrently, if they are operating on different tables (e.g.
two create table procedures can be performed at the same time) or against two different servers;
say two servers that crashed at about the same time.
<p>
Each procedure should implement an Interface providing information for this queue. For example
table related procedures should implement TableProcedureInterface. Each procedure will be pushed
in its own queue, and based on the operation type we may make smarter decisions: e.g. we can
abort all the operations preceding a delete table, or similar.
<h4>Concurrency control</h4> Concurrent access to member variables (tableRunQueue,
serverRunQueue, locking, tableMap, serverBuckets) is controlled by schedLock(). This mainly
includes:<br>
<ul>
<li>{@link #push(Procedure, boolean, boolean)}: A push will add a Queue back to run-queue when:
<ol>
<li>Queue was empty before push (so must have been out of run-queue)</li>
<li>Child procedure is added (which means parent procedure holds exclusive lock, and it must have
moved Queue out of run-queue)</li>
</ol>
</li>
<li>{@link #poll(long)}: A poll will remove a Queue from run-queue when:
<ol>
<li>Queue becomes empty after poll</li>
<li>Exclusive lock is requested by polled procedure and lock is available (returns the
procedure)</li>
<li>Exclusive lock is requested but lock is not available (returns null)</li>
<li>Polled procedure is child of parent holding exclusive lock and the next procedure is not a
child</li>
</ol>
</li>
<li>Namespace/table/region locks: Queue is added back to run-queue when lock being released is:
<ol>
<li>Exclusive lock</li>
<li>Last shared lock (in case queue was removed because next procedure in queue required
exclusive lock)</li>
</ol>
</li>
</ul>
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java