public boolean isDisallowedByRobots(LinkRelevance link) {
        String hostname = link.getURL().getHost();
        SimpleRobotRules rules = robotRulesMap.get(hostname);
        return rules != null && !rules.isAllowed(link.getURL().toString());
    }
Inserts the robot rules object into the HashMap

@param link
@param robotRules
@throws NullPointerException
            when either of the argument is null
ache/src/main/java/achecrawler/link/frontier/Frontier.java