protected boolean noNeedToRemoveDuplicate(Request request) {
        return HttpConstant.Method.POST.equalsIgnoreCase(request.getMethod());
    }
Remove duplicate urls and only push urls which are not duplicate.<br><br>

@author code4crafer@gmail.com
@since 0.5.0
webmagic-core/src/main/java/us/codecraft/webmagic/scheduler/DuplicateRemovedScheduler.java