The Lexer now only provides the tokens to the parser, while the
parser is responsible for determining which attributes are placed
on notes. This only needs to be done when the attributes are
actually needed, rather than for all tokens.
This removes the usedAttributes lexer option (and lexer options
entirely). The attributes are now enabled unconditionally. They
have less overhead now, and the need to explicitly enable them for
some use cases (e.g. formatting-preserving printing) doesn't seem
like a good tradeoff anymore.
There are some additional changes to the Lexer interface that
should be done after this, and the docs / upgrading guide haven't
been adjusted yet.
Add ErrorHandler interface, as well as ErrorHandler\Throwing
and ErrorHandler\Collecting. The error handler is passed to
Parser::parse(). This supersedes the throwOnError option.
NameResolver now accepts an ErrorHandler in the ctor.
Lexer::startLexing() no longer throws, instead errors can be fetched
using Lexer::getErrors().
Lexer errors now also contain full line and position information.
As these are shared between Php5 and Php7 parsers they should be
in some common place, otherwise we'd have to always reference either
one or the other.
HHVM does not throw warnings from token_get_all()
for invalid chars and unterminated comments. This
is not really integral functionality, so I'm okay
to just skip it to get a passing HHVM build.
Also change endFilePos semantics to refer to the last character that
is *included* in the token, rather than one past the last character.
This ensures that all end* attributes have the same semantics.
The lexer can now optionally add startFilePos and endFilePos
attributes, which are offsets in to the lexed code string.
The end offset currently points one past the last character of
the token - this is pending further discussion.
The attributes are not added by default and have to be enabled
using the new 'usedAttributes' lexer option:
$lexer = new Lexer([
'usedAttributes' => [
'comments', 'startLine', 'endLine',
'startFilePos', 'endFilePos'
]
]);
