public boolean getDisallowSitesInRobotsFile() {
        return disallowSitesInRobotsFile;
    }
Returns true if the user wants the disallowed sites in robots.txt to be skipped

@return
ache/src/main/java/achecrawler/link/LinkStorageConfig.java