public void insert(RobotsTxtHandler.RobotsData robotsData) {
        if (disallowSitesInRobotsTxt) {
            this.insertRobotRules(robotsData.link, robotsData.robotRules);
        }
        if (insertSiteMaps) {
            for (String sitemap : robotsData.robotRules.getSitemaps()) {
                try {
                    frontierManager.insert(LinkRelevance.createSitemap(sitemap, 299));
                } catch (Exception e) {
                    logger.error("Failed to insert sitemap from robot: " + sitemap);
                }
            }
        }
    }
This method inserts links from a given page into the frontier

@param obj
           Object - page containing links
@return Object
ache/src/main/java/achecrawler/link/LinkStorage.java