@Test
  public void testCanaryStopsScanningAfterTimeout() throws Exception {
    // Prepare a table with multiple regions, and close those regions on the regionserver.
    // Do not notify HMaster or META. CanaryTool will scan and receive NotServingRegionExceptions.
    final TableName tableName = TableName.valueOf(name.getMethodName());
    // Close the unused Table reference returned by createMultiRegionTable.
    testingUtility.createMultiRegionTable(tableName, new byte[][] { FAMILY }).close();
    List<RegionInfo> regions = testingUtility.getAdmin().getRegions(tableName);
    assertTrue("verify table has multiple regions", regions.size() > 1);
    HRegionServer regionserver = testingUtility.getMiniHBaseCluster().getRegionServer(0);
    for (RegionInfo region : regions) {
      closeRegion(testingUtility, regionserver, region);
    }

    // Run CanaryTool with 1 thread. This thread will attempt to scan the first region.
    // It will use default rpc retries and receive NotServingRegionExceptions for many seconds
    // according to HConstants.RETRY_BACKOFF. The CanaryTool timeout is set to 4 seconds, so it
    // will time out before the first region scan is complete.
    ExecutorService executor = new ScheduledThreadPoolExecutor(1);
    CanaryTool canary = new CanaryTool(executor);
    String[] args = { "-t", "4000", tableName.getNameAsString() };
    int retCode = ToolRunner.run(testingUtility.getConfiguration(), canary, args);
    executor.shutdown();
    try {
      if (!executor.awaitTermination(3, TimeUnit.SECONDS)) {
        executor.shutdownNow();
      }
    } catch (InterruptedException e) {
      executor.shutdownNow();
    }

    CanaryTool.Sink sink = canary.getActiveSink();
    assertEquals("verify canary timed out with TIMEOUT_ERROR_EXIT_CODE", 3, retCode);
    assertEquals("verify only the first region failed", 1, sink.getReadFailureCount());
    assertEquals("verify no successful reads", 0, sink.getReadSuccessCount());
    assertEquals("verify we were attempting to scan all regions", regions.size(),
      ((CanaryTool.RegionStdOutSink) sink).getTotalExpectedRegions());
  }
When CanaryTool times out, it should stop scanning and shutdown quickly and gracefully. This
test helps to confirm that threadpools do not continue executing work after the canary
finishes. It also verifies sink behavior and measures correct failure counts in the sink.
@throws Exception if it can't create a table, communicate with minicluster, or run the canary.
hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java