@Test
  public void testRecoveryWithOneServerPermanentlyDownAndOneRestarted() throws Exception {

    SERVER_ZONE_MAP = new HashMap<>();
    SERVER_ZONE_MAP.put(1, ZONE_A);
    SERVER_ZONE_MAP.put(2, ZONE_A);
    SERVER_ZONE_MAP.put(3, ZONE_B);
    SERVER_ZONE_MAP.put(4, ZONE_B);


    // Startup the servers
    for (Map.Entry<Integer, String> entry : SERVER_ZONE_MAP.entrySet()) {
      startServerInRedundancyZone(entry.getKey(), entry.getValue());
    }

    // Put data in the server regions
    clientPopulateServers();

    // Rebalance Server VM will initiate the rebalances in this test
    VM server2 = clusterStartupRule.getVM(2);

    // Take the server 1 offline
    clusterStartupRule.stop(1, false);

    // Baseline rebalance with everything up
    server2.invoke(() -> doRebalance(ClusterStartupRule.getCache().getResourceManager()));

    // Stop server 3 never to start again.
    clusterStartupRule.stop(3, true);

    // Restart the server 1
    startServerInRedundancyZone(1, SERVER_ZONE_MAP.get(1));

    // Rebalance with remaining servers up
    server2.invoke(() -> doRebalance(ClusterStartupRule.getCache().getResourceManager()));

    // print the bucket count on server 4 for debug. Should be 113 because server 3 is still down
    assertThat(getBucketCount(4)).isEqualTo(EXPECTED_BUCKET_COUNT);

    assertThat(getBucketCount(1)).isGreaterThanOrEqualTo(EXPECTED_BUCKET_COUNT / 2)
        .isLessThanOrEqualTo(EXPECTED_BUCKET_COUNT / 2 + 1);

    assertThat(getBucketCount(2)).isGreaterThanOrEqualTo(EXPECTED_BUCKET_COUNT / 2)
        .isLessThanOrEqualTo(EXPECTED_BUCKET_COUNT / 2 + 1);

    // Verify that all bucket counts add up to what they should
    int zoneABucketCount = getZoneABucketCount();
    assertThat(zoneABucketCount).isEqualTo(EXPECTED_BUCKET_COUNT);
  }
Test that we correctly use the redundancy-zone property to determine where to place redundant
copies of a buckets and doesn't allow cross redundancy zone deletes. This does not use a
rebalance once the servers are down.
geode-core/src/distributedTest/java/org/apache/geode/internal/cache/control/RebalanceOperationComplexPart2DistributedTest.java