private boolean getDownloadRobots() {
        return insertSitemaps || disallowSitesInRobotsFile;
    }
Returns true if either the property to include sitemaps is true or disallow sites in
robots.txt is true.
ache/src/main/java/achecrawler/link/frontier/FrontierManager.java