The Lexer now only provides the tokens to the parser, while the
parser is responsible for determining which attributes are placed
on notes. This only needs to be done when the attributes are
actually needed, rather than for all tokens.
This removes the usedAttributes lexer option (and lexer options
entirely). The attributes are now enabled unconditionally. They
have less overhead now, and the need to explicitly enable them for
some use cases (e.g. formatting-preserving printing) doesn't seem
like a good tradeoff anymore.
There are some additional changes to the Lexer interface that
should be done after this, and the docs / upgrading guide haven't
been adjusted yet.
Kind specifies whether the number was formatted as decimal, octal,
binary or hex. The pretty printer reproduces the number kind (but
not necessarily the exact formatting).
Adding this as an option to avoid breaking people's tests.
Some of the test results show pretty clearly that we are incorrectly
assigning the same comment multiple times for nested nodes (mentioned
in #36).
