@Override
        public boolean equals(Object o) {
            if (this == o) {
                return true;
            }
            if (o == null || getClass() != o.getClass()) {
                return false;
            }

            TokenDFTF tokenDFTF = (TokenDFTF) o;

            if (df != tokenDFTF.df) {
                return false;
            }
            if (tf != tokenDFTF.tf) {
                return false;
            }
            return Objects.equals(token, tokenDFTF.token);
        }
Utility class that reads in a UTF-8 input file with one document per row
and outputs the 20000 tokens with the highest document frequencies.
<p>
The CommmonTokensAnalyzer intentionally drops tokens shorter than 4 characters,
but includes bigrams for cjk.
<p>
It also has a include list for __email__ and __url__ and a skip list
for common html markup terms.
tika-eval/tika-eval-app/src/main/java/org/apache/tika/eval/app/tools/TopCommonTokenCounter.java