private String stringify(Token[] tokens) {
		StringBuilder s = new StringBuilder();
		if (tokens == null) {
			s.append("null");
		}
		else {
			s.append("#").append(tokens.length).append(" ");
			for (Token token : tokens) {
				s.append(token).append(" ");
			}
		}
		return s.toString().trim();
	}
Check the tokenizer breaks strings up into correct token streams.

@author Andy Clement
spring-xd-dirt/src/test/java/org/springframework/xd/dirt/stream/dsl/TokenizerTests.java