"The word ‘tokenizer’ makes a lot more sense, but ‘lexer’ is so much fun to say that I use it anyway."

No wonders people think a lot of programming subjects are intimidating.

Weird lex, but okay

Couldn't this be considered the case where every lexer is a tokenizer, but not every tokenizer is a lexer? Because all lexemes need to mean something in relation to some lexical analysis, but not all tokens need to mean something when you tokenize a string. In short, lexing a nonsense string should result in an error but tokenizing a nonsense string only results into parts of that string extracted by some rule. Dunno, philosophical question :D

@Hazarth No idea, haven't done a lot of this. But just the way that sentence is expressed makes me a bit irritated.