This doesn't make a lot of sense now that Lexer::tokenize() returns
the tokens.
The tokens for the last parse should be fetched via
Parser::getTokens() instead.
Types omitted in two places where we violate them currently:
Namespace_::$stmts can be null during parsing, and Enum_::$scalarType
can be a complex type for invalid programs.
The Lexer now only provides the tokens to the parser, while the
parser is responsible for determining which attributes are placed
on notes. This only needs to be done when the attributes are
actually needed, rather than for all tokens.
This removes the usedAttributes lexer option (and lexer options
entirely). The attributes are now enabled unconditionally. They
have less overhead now, and the need to explicitly enable them for
some use cases (e.g. formatting-preserving printing) doesn't seem
like a good tradeoff anymore.
There are some additional changes to the Lexer interface that
should be done after this, and the docs / upgrading guide haven't
been adjusted yet.
This makes us match the PHP 8.2 handling of readonly. Handling of
"readonly" functions is moved to the parser to allow distinguishing
them from readonly properties with DNF types. We have to uglify the
grammar to avoid some shift/reduce conflicts. Thank you WordPress.
The formatting in this project has become something of a mess,
because it changed over time. Add a CS fixer config and reformat
to the desired style, which is PSR-12, but with sane brace placement.
Only determine needed emulators based on PHP version once, and
add an adaptor that allows treating forward and reverse emulation
the same.
Previously the isEmulationNeeded() check was too conservative,
as it also considered emulators that are not relevant for the
version. Though possibly that check should just be dropped
altogether.
a) Don't check for !== false, preg_match() return 0 if there is
no match. This effectively means that the check was always true.
b) Check for hex characters only if there's an 0x prefix. 1_1 is
very likely a numeric separator, but a_b might well be part of
an identifier.
Fixes#639.
This was recently fixed in PHP via
310708845f
so we should fix it here as well.
I'm not adding a dedicated test as it will fail until new PHP versions
are released. This is indirectly tested through the php-src tester.
This only implements the token emulation part, it does not yet
handle the indentation stripping.
Emulating this feature on old PHP versions is really tough and is
not going to work perfectly, but hopefully this implementation is
"good enough".
