@Test
  public void testMultipleMember() throws Exception {
    // 2 members
    List<String> members = new ArrayList<>();
    members.add("member1");
    members.add("member2");

    LatchedProcedure proc = new LatchedProcedure(coord, new ForeignExceptionDispatcher(), 100,
      Integer.MAX_VALUE, "op", null, members);
    final LatchedProcedure procspy = spy(proc);
    // start the barrier procedure
    new Thread() {
      @Override
      public void run() {
        procspy.call();
      }
    }.start();

    // coordinator: wait for the barrier to be acquired, then send start barrier
    procspy.startedAcquireBarrier.await();

    // we only know that {@link Procedure#sendStartBarrier()} was called, and others are blocked.
    verify(procspy).sendGlobalBarrierStart();
    verify(procspy, never()).sendGlobalBarrierReached();
    verify(procspy, never()).sendGlobalBarrierComplete();
    verify(procspy, never()).barrierAcquiredByMember(anyString()); // no externals

    // member0: [1/2] trigger global barrier acquisition.
    procspy.barrierAcquiredByMember(members.get(0));

    // coordinator not satisified.
    verify(procspy).sendGlobalBarrierStart();
    verify(procspy, never()).sendGlobalBarrierReached();
    verify(procspy, never()).sendGlobalBarrierComplete();

    // member 1: [2/2] trigger global barrier acquisition.
    procspy.barrierAcquiredByMember(members.get(1));

    // coordinator: wait for global barrier to be acquired.
    procspy.startedDuringBarrier.await();
    verify(procspy).sendGlobalBarrierStart(); // old news

    // member 1, 2: trigger global barrier release
    procspy.barrierReleasedByMember(members.get(0), new byte[0]);
    procspy.barrierReleasedByMember(members.get(1), new byte[0]);

    // coordinator wait for procedure to be completed
    procspy.completedProcedure.await();
    verify(procspy).sendGlobalBarrierReached();
    verify(procspy).sendGlobalBarrierComplete();
    verify(procspy, never()).receive(any());
  }
With a single member, verify ordered execution. The Coordinator side is run in a separate
thread so we can only trigger from members and wait for particular state latches.
hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedure.java