public List<Token> getTokens() {
		return tokens;
	}
Lex some input data into a stream of tokens that can then then be parsed.

@author Andy Clement
spring-xd-dirt/src/main/java/org/springframework/xd/dirt/job/dsl/Tokenizer.java