Release PHP-Parser 4.7.0

Like comment emulation, this is unconditional, as it is required
for core functionality.

.gitattributes

@@ -0,0 +1,9 @@
+/doc export-ignore
+/test export-ignore
+/test_old export-ignore
+.gitattributes export-ignore
+.gitignore export-ignore
+.travis.yml export-ignore
+CHANGELOG.md export-ignore
+phpunit.xml.dist export-ignore
+UPGRADE-*.md export-ignore

.gitignore

@@ -2,3 +2,4 @@ vendor/
 composer.lock
 grammar/kmyacc.exe
 grammar/y.output
+.phpunit.result.cache

.travis.yml

@@ -1,6 +1,5 @@
 language: php
 dist: xenial
-sudo: false
 
 cache:
   directories:
@@ -11,12 +10,17 @@ php:
   - 7.1
   - 7.2
   - 7.3
-  - 7.4snapshot
+  - 7.4
   - nightly
 
 install:
   - if [ $TRAVIS_PHP_VERSION = '7.0' ]; then composer require satooshi/php-coveralls '~1.0'; fi
-  - composer install --prefer-dist --ignore-platform-reqs
+  - |
+    if [ $TRAVIS_PHP_VERSION = 'nightly' ]; then
+      composer install --prefer-dist --ignore-platform-reqs;
+    else
+      composer install --prefer-dist;
+    fi
 
 matrix:
   allow_failures:

CHANGELOG.md

@@ -1,7 +1,108 @@
-Version 4.2.3-dev
+Version 4.7.1-dev
 -----------------
 
-Nothing yet.
+Version 4.7.0 (2020-07-25)
+--------------------------
+
+### Added
+
+* Add `ParentConnectingVisitor` and `NodeConnectingVisitor` classes.
+* [PHP 8.0] Added support for match expressions. These are represented using a new `Expr\Match_`
+  containing `MatchArm`s.
+* [PHP 8.0] Added support for trailing comma in closure use lists.
+
+### Fixed
+
+* Fixed missing error for unterminated comment with trailing newline (#688).
+* Compatibility with PHP 8.0 has been restored: Namespaced names are now always represented by
+  `T_NAME_*` tokens, using emulationg on older PHP versions. Full support for reserved keywords
+  in namespaced names is not yet present.
+
+Version 4.6.0 (2020-07-02)
+--------------------------
+
+### Added
+
+* [PHP 8.0] Added support for trailing commas in parameter lists.
+* [PHP 8.0] Added support for constructor promotion. The parameter visibility is stored in
+  `Node\Param::$flags`.
+
+### Fixed
+
+* Comment tokens now always follow the PHP 8 interpretation, and do not include trailing
+  whitespace.
+* As a result of the previous change, some whitespace issues when inserting a statement into a
+  method containing only a comment, and using the formatting-preserving pretty printer, have been
+  resolved.
+
+Version 4.5.0 (2020-06-03)
+--------------------------
+
+### Added
+
+* [PHP 8.0] Added support for the mixed type. This means `mixed` types are now parsed as an
+  `Identifier` rather than a `Name`.
+* [PHP 8.0] Added support for catching without capturing the exception. This means that
+  `Catch_::$var` may now be null.
+
+Version 4.4.0 (2020-04-10)
+--------------------------
+
+### Added
+
+* Added support for passing union types in builders.
+* Added end line, token position and file position information for comments.
+* Added `getProperty()` method to `ClassLike` nodes.
+
+### Fixed
+
+* Fixed generation of invalid code when using the formatting preserving pretty printer, and
+  inserting code next to certain nop statements. The formatting is still ugly though.
+* `getDocComment()` no longer requires that the very last comment before a node be a doc comment.
+  There may not be non-doc comments between the doc comment and the declaration.
+* Allowed arbitrary expressions in `isset()` and `list()`, rather than just variables.
+  In particular, this allows `isset(($x))`, which is legal PHP code.
+* [PHP 8.0] Add support for [variable syntax tweaks RFC](https://wiki.php.net/rfc/variable_syntax_tweaks).
+
+Version 4.3.0 (2019-11-08)
+--------------------------
+
+### Added
+
+* [PHP 8.0] Added support for union types using a new `UnionType` node.
+
+Version 4.2.5 (2019-10-25)
+--------------------------
+
+### Changed
+
+* Tests and documentation are no longer included in source archives. They can still be accessed
+  by cloning the repository.
+* php-yacc is now used to generate the parser. This has no impact on users of the library.
+
+Version 4.2.4 (2019-09-01)
+--------------------------
+
+### Added
+
+* Added getProperties(), getConstants() and getTraitUses() to ClassLike. (#629, #630)
+
+### Fixed
+
+* Fixed flexible heredoc emulation to check for digits after the end label. This synchronizes
+  behavior with the upcoming PHP 7.3.10 release.
+
+Version 4.2.3 (2019-08-12)
+--------------------------
+
+### Added
+
+* [PHP 7.4] Add support for numeric literal separators. (#615)
+
+### Fixed
+
+* Fixed resolution of return types for arrow functions. (#613)
+* Fixed compatibility with PHP 7.4.
 
 Version 4.2.2 (2019-05-25)
 --------------------------

LICENSE

@@ -1,31 +1,29 @@
-Copyright (c) 2011-2018 by Nikita Popov.
+BSD 3-Clause License
 
-Some rights reserved.
+Copyright (c) 2011, Nikita Popov
+All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are
+modification, are permitted provided that the following conditions are met:
-met:
 
-    * Redistributions of source code must retain the above copyright
+1. Redistributions of source code must retain the above copyright notice, this
-      notice, this list of conditions and the following disclaimer.
+   list of conditions and the following disclaimer.
 
-    * Redistributions in binary form must reproduce the above
+2. Redistributions in binary form must reproduce the above copyright notice,
-      copyright notice, this list of conditions and the following
+   this list of conditions and the following disclaimer in the documentation
-      disclaimer in the documentation and/or other materials provided
+   and/or other materials provided with the distribution.
-      with the distribution.
 
-    * The names of the contributors may not be used to endorse or
+3. Neither the name of the copyright holder nor the names of its
-      promote products derived from this software without specific
+   contributors may be used to endorse or promote products derived from
-      prior written permission.
+   this software without specific prior written permission.
 
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -3,10 +3,10 @@ PHP Parser
 
 [![Build Status](https://travis-ci.org/nikic/PHP-Parser.svg?branch=master)](https://travis-ci.org/nikic/PHP-Parser) [![Coverage Status](https://coveralls.io/repos/github/nikic/PHP-Parser/badge.svg?branch=master)](https://coveralls.io/github/nikic/PHP-Parser?branch=master)
 
-This is a PHP 5.2 to PHP 7.3 parser written in PHP. Its purpose is to simplify static code analysis and
+This is a PHP 5.2 to PHP 7.4 parser written in PHP. Its purpose is to simplify static code analysis and
 manipulation.
 
-[**Documentation for version 4.x**][doc_master] (stable; for running on PHP >= 7.0; for parsing PHP 5.2 to PHP 7.3).
+[**Documentation for version 4.x**][doc_master] (stable; for running on PHP >= 7.0; for parsing PHP 5.2 to PHP 7.4).
 
 [Documentation for version 3.x][doc_3_x] (unsupported; for running on PHP >= 5.5; for parsing PHP 5.2 to PHP 7.2).
 

composer.json

@@ -17,11 +17,12 @@
         "ext-tokenizer": "*"
     },
     "require-dev": {
-        "phpunit/phpunit": "^6.5 || ^7.0"
+        "phpunit/phpunit": "^6.5 || ^7.0 || ^8.0",
+        "ircmaxell/php-yacc": "0.0.5"
     },
     "extra": {
         "branch-alias": {
-            "dev-master": "4.2-dev"
+            "dev-master": "4.7-dev"
         }
     },
     "autoload": {

doc/0_Introduction.markdown

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-This project is a PHP 5.2 to PHP 7.3 parser **written in PHP itself**.
+This project is a PHP 5.2 to PHP 7.4 parser **written in PHP itself**.
 
 What is this for?
 -----------------
@@ -26,11 +26,11 @@ programmatic PHP code analysis are incidentally PHP developers, not C developers
 What can it parse?
 ------------------
 
-The parser supports parsing PHP 5.2-7.3.
+The parser supports parsing PHP 5.2-7.4.
 
 As the parser is based on the tokens returned by `token_get_all` (which is only able to lex the PHP
 version it runs on), additionally a wrapper for emulating tokens from newer versions is provided.
-This allows to parse PHP 7.3 source code running on PHP 7.0, for example. This emulation is somewhat
+This allows to parse PHP 7.4 source code running on PHP 7.0, for example. This emulation is somewhat
 hacky and not perfect, but it should work well on any sane code.
 
 What output does it produce?
@@ -56,7 +56,7 @@ array(
 ```
 
 This matches the structure of the code: An echo statement, which takes two strings as expressions,
-with the values `Hi` and `World!`.
+with the values `Hi` and `World`.
 
 You can also see that the AST does not contain any whitespace information (but most comments are saved).
 So using it for formatting analysis is not possible.

doc/2_Usage_of_basic_components.markdown

@@ -77,7 +77,7 @@ A parser instance can be reused to parse multiple files.
 Node dumping
 ------------
 
-To dump the abstact syntax tree in human readable form, a `NodeDumper` can be used:
+To dump the abstract syntax tree in human readable form, a `NodeDumper` can be used:
 
 ```php
 <?php

doc/component/FAQ.markdown

@@ -7,28 +7,21 @@ Frequently Asked Questions
 How can the parent of a node be obtained?
 -----
 
-The AST does not store parent nodes by default. However, it is easy to add a custom parent node
+The AST does not store parent nodes by default. However, the `ParentConnectingVisitor` can be used to achieve this:
-attribute using a custom node visitor:
 
 ```php
-use PhpParser\Node;
+use PhpParser\NodeTraverser;
-use PhpParser\NodeVisitorAbstract;
+use PhpParser\NodeVisitor\ParentConnectingVisitor;
+use PhpParser\ParserFactory;
 
-class ParentConnector extends NodeVisitorAbstract {
+$code = '...';
-    private $stack;
+
-    public function beforeTraverse(array $nodes) {
+$traverser = new NodeTraverser;
-        $this->stack = [];
+$traverser->addVisitor(new ParentConnectingVisitor);
-    }
+
-    public function enterNode(Node $node) {
+$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
-        if (!empty($this->stack)) {
+$ast    = $parser->parse($code);
-            $node->setAttribute('parent', $this->stack[count($this->stack)-1]);
+$ast    = $traverser->traverse($ast);
-        }
-        $this->stack[] = $node;
-    }
-    public function leaveNode(Node $node) {
-        array_pop($this->stack);
-    }
-}
 ```
 
 After running this visitor, the parent node can be obtained through `$node->getAttribute('parent')`.
@@ -36,33 +29,27 @@ After running this visitor, the parent node can be obtained through `$node->getA
 How can the next/previous sibling of a node be obtained?
 -----
 
-Again, siblings are not stored by default, but the visitor from the previous entry can be easily
+Again, siblings are not stored by default, but the `NodeConnectingVisitor` can be used to store
-extended to store the previous / next node with a common parent as well:
+the previous / next node with a common parent as well:
 
 ```php
-use PhpParser\Node;
+use PhpParser\NodeTraverser;
-use PhpParser\NodeVisitorAbstract;
+use PhpParser\NodeVisitor\NodeConnectingVisitor;
+use PhpParser\ParserFactory;
 
-class NodeConnector extends NodeVisitorAbstract {
+$code = '...';
-    private $stack;
+
-    private $prev;
+$traverser = new NodeTraverser;
-    public function beforeTraverse(array $nodes) {
+$traverser->addVisitor(new NodeConnectingVisitor);
-        $this->stack = [];
+
-        $this->prev = null;
+$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
-    }
+$ast    = $parser->parse($code);
-    public function enterNode(Node $node) {
+$ast    = $traverser->traverse($ast);
-        if (!empty($this->stack)) {
-            $node->setAttribute('parent', $this->stack[count($this->stack)-1]);
-        }
-        if ($this->prev && $this->prev->getAttribute('parent') == $node->getAttribute('parent')) {
-            $node->setAttribute('prev', $this->prev);
-            $this->prev->setAttribute('next', $node);
-        }
-        $this->stack[] = $node;
-    }
-    public function leaveNode(Node $node) {
-        $this->prev = $node;
-        array_pop($this->stack);
-    }
-}
 ```
+
+After running this visitor, the parent node can be obtained through `$node->getAttribute('parent')`,
+the previous node can be obtained through `$node->getAttribute('previous')`, and the next node can be
+obtained through `$node->getAttribute('next')`.
+
+`ParentConnectingVisitor` and `NodeConnectingVisitor` should not be used at the same time. The latter
+includes the functionality of the former.

doc/component/Walking_the_AST.markdown

@@ -295,7 +295,7 @@ Simple node finding
 
 While the node visitor mechanism is very flexible, creating a node visitor can be overly cumbersome
 for minor tasks. For this reason a `NodeFinder` is provided, which can find AST nodes that either
-satisfy a certain callback, or which are instanced of a certain node type. A couple of examples are
+satisfy a certain callback, or which are instances of a certain node type. A couple of examples are
 shown in the following:
 
 ```php
@@ -312,7 +312,7 @@ $extendingClasses = $nodeFinder->find($stmts, function(Node $node) {
         && $node->extends !== null;
 });
 
-// Find first class occuring in the AST. Returns null if no class exists.
+// Find first class occurring in the AST. Returns null if no class exists.
 $class = $nodeFinder->findFirstInstanceOf($stmts, Node\Stmt\Class_::class);
 
 // Find first class that has name $name

grammar/README.md

@@ -21,8 +21,10 @@ applied to it:
 Building the parser
 ===================
 
-In order to rebuild the parser, you need [moriyoshi's fork of kmyacc](https://github.com/moriyoshi/kmyacc-forked).
+Run `php grammar/rebuildParsers.php` to rebuild the parsers. Additional options:
-After you compiled/installed it, run the `rebuildParsers.php` script.
 
-By default only the `Parser.php` is built. If you want to additionally emit debug symbols and create `y.output`, run the
+ * The `KMYACC` environment variable can be used to specify an alternative `kmyacc` binary.
-script with `--debug`. If you want to retain the preprocessed grammar pass `--keep-tmp-grammar`.
+   By default the `phpyacc` dev dependency will be used. To use the original `kmyacc`, you
+   need to compile [moriyoshi's fork](https://github.com/moriyoshi/kmyacc-forked).
+ * The `--debug` option enables emission of debug symbols and creates the `y.output` file.
+ * The `--keep-tmp-grammar` option preserves the preprocessed grammar file.

grammar/php5.y

@@ -28,6 +28,7 @@ reserved_non_modifiers:
     | T_FUNCTION | T_CONST | T_RETURN | T_PRINT | T_YIELD | T_LIST | T_SWITCH | T_ENDSWITCH | T_CASE | T_DEFAULT
     | T_BREAK | T_ARRAY | T_CALLABLE | T_EXTENDS | T_IMPLEMENTS | T_NAMESPACE | T_TRAIT | T_INTERFACE | T_CLASS
     | T_CLASS_C | T_TRAIT_C | T_FUNC_C | T_METHOD_C | T_LINE | T_FILE | T_DIR | T_NS_C | T_HALT_COMPILER | T_FN
+    | T_MATCH
 ;
 
 semi_reserved:
@@ -48,13 +49,14 @@ reserved_non_modifiers_identifier:
       reserved_non_modifiers                                { $$ = Node\Identifier[$1]; }
 ;
 
-namespace_name_parts:
+namespace_name:
-      T_STRING                                              { init($1); }
+      T_STRING                                              { $$ = Name[$1]; }
-    | namespace_name_parts T_NS_SEPARATOR T_STRING          { push($1, $3); }
+    | T_NAME_QUALIFIED                                      { $$ = Name[$1]; }
 ;
 
-namespace_name:
+legacy_namespace_name:
-      namespace_name_parts                                  { $$ = Name[$1]; }
+      namespace_name                                        { $$ = $1; }
+    | T_NAME_FULLY_QUALIFIED                                { $$ = Name[substr($1, 1)]; }
 ;
 
 plain_variable:
@@ -90,16 +92,11 @@ use_type:
     | T_CONST                                               { $$ = Stmt\Use_::TYPE_CONSTANT; }
 ;
 
-/* Using namespace_name_parts here to avoid s/r conflict on T_NS_SEPARATOR */
 group_use_declaration:
-      T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+      T_USE use_type legacy_namespace_name T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
-          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, $2]; }
+          { $$ = Stmt\GroupUse[$3, $6, $2]; }
-    | T_USE use_type T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+    | T_USE legacy_namespace_name T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[new Name($4, stackAttributes(#4)), $7, $2]; }
+          { $$ = Stmt\GroupUse[$2, $5, Stmt\Use_::TYPE_UNKNOWN]; }
-    | T_USE namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[new Name($2, stackAttributes(#2)), $5, Stmt\Use_::TYPE_UNKNOWN]; }
-    | T_USE T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, Stmt\Use_::TYPE_UNKNOWN]; }
 ;
 
 unprefixed_use_declarations:
@@ -126,8 +123,10 @@ unprefixed_use_declaration:
 ;
 
 use_declaration:
-      unprefixed_use_declaration                            { $$ = $1; }
+      legacy_namespace_name
-    | T_NS_SEPARATOR unprefixed_use_declaration             { $$ = $2; }
+          { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #1); }
+    | legacy_namespace_name T_AS identifier
+          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #3); }
 ;
 
 inline_use_declaration:
@@ -733,9 +732,10 @@ class_name:
 ;
 
 name:
-      namespace_name_parts                                  { $$ = Name[$1]; }
+      T_STRING                                              { $$ = Name[$1]; }
-    | T_NS_SEPARATOR namespace_name_parts                   { $$ = Name\FullyQualified[$2]; }
+    | T_NAME_QUALIFIED                                      { $$ = Name[$1]; }
-    | T_NAMESPACE T_NS_SEPARATOR namespace_name_parts       { $$ = Name\Relative[$3]; }
+    | T_NAME_FULLY_QUALIFIED                                { $$ = Name\FullyQualified[substr($1, 1)]; }
+    | T_NAME_RELATIVE                                       { $$ = Name\Relative[substr($1, 10)]; }
 ;
 
 class_name_reference:

grammar/php7.y

@@ -28,6 +28,7 @@ reserved_non_modifiers:
     | T_FUNCTION | T_CONST | T_RETURN | T_PRINT | T_YIELD | T_LIST | T_SWITCH | T_ENDSWITCH | T_CASE | T_DEFAULT
     | T_BREAK | T_ARRAY | T_CALLABLE | T_EXTENDS | T_IMPLEMENTS | T_NAMESPACE | T_TRAIT | T_INTERFACE | T_CLASS
     | T_CLASS_C | T_TRAIT_C | T_FUNC_C | T_METHOD_C | T_LINE | T_FILE | T_DIR | T_NS_C | T_HALT_COMPILER | T_FN
+    | T_MATCH
 ;
 
 semi_reserved:
@@ -48,13 +49,14 @@ reserved_non_modifiers_identifier:
       reserved_non_modifiers                                { $$ = Node\Identifier[$1]; }
 ;
 
-namespace_name_parts:
+namespace_name:
-      T_STRING                                              { init($1); }
+      T_STRING                                              { $$ = Name[$1]; }
-    | namespace_name_parts T_NS_SEPARATOR T_STRING          { push($1, $3); }
+    | T_NAME_QUALIFIED                                      { $$ = Name[$1]; }
 ;
 
-namespace_name:
+legacy_namespace_name:
-      namespace_name_parts                                  { $$ = Name[$1]; }
+      namespace_name                                        { $$ = $1; }
+    | T_NAME_FULLY_QUALIFIED                                { $$ = Name[substr($1, 1)]; }
 ;
 
 plain_variable:
@@ -104,16 +106,11 @@ use_type:
     | T_CONST                                               { $$ = Stmt\Use_::TYPE_CONSTANT; }
 ;
 
-/* Using namespace_name_parts here to avoid s/r conflict on T_NS_SEPARATOR */
 group_use_declaration:
-      T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+      T_USE use_type legacy_namespace_name T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
-          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, $2]; }
+          { $$ = Stmt\GroupUse[$3, $6, $2]; }
-    | T_USE use_type T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+    | T_USE legacy_namespace_name T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[new Name($4, stackAttributes(#4)), $7, $2]; }
+          { $$ = Stmt\GroupUse[$2, $5, Stmt\Use_::TYPE_UNKNOWN]; }
-    | T_USE namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[new Name($2, stackAttributes(#2)), $5, Stmt\Use_::TYPE_UNKNOWN]; }
-    | T_USE T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, Stmt\Use_::TYPE_UNKNOWN]; }
 ;
 
 unprefixed_use_declarations:
@@ -153,8 +150,10 @@ unprefixed_use_declaration:
 ;
 
 use_declaration:
-      unprefixed_use_declaration                            { $$ = $1; }
+      legacy_namespace_name
-    | T_NS_SEPARATOR unprefixed_use_declaration             { $$ = $2; }
+          { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #1); }
+    | legacy_namespace_name T_AS identifier
+          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #3); }
 ;
 
 inline_use_declaration:
@@ -232,7 +231,7 @@ non_empty_statement:
     | T_RETURN optional_expr semi                           { $$ = Stmt\Return_[$2]; }
     | T_GLOBAL global_var_list semi                         { $$ = Stmt\Global_[$2]; }
     | T_STATIC static_var_list semi                         { $$ = Stmt\Static_[$2]; }
-    | T_ECHO expr_list semi                                 { $$ = Stmt\Echo_[$2]; }
+    | T_ECHO expr_list_forbid_comma semi                    { $$ = Stmt\Echo_[$2]; }
     | T_INLINE_HTML                                         { $$ = Stmt\InlineHTML[$1]; }
     | expr semi                                             { $$ = Stmt\Expression[$1]; }
     | T_UNSET '(' variables_list ')' semi                   { $$ = Stmt\Unset_[$3]; }
@@ -269,7 +268,7 @@ name_union:
 ;
 
 catch:
-    T_CATCH '(' name_union plain_variable ')' '{' inner_statement_list '}'
+    T_CATCH '(' name_union optional_plain_variable ')' '{' inner_statement_list '}'
         { $$ = Stmt\Catch_[$3, $4, $7]; }
 ;
 
@@ -399,6 +398,25 @@ case_separator:
     | ';'
 ;
 
+match:
+      T_MATCH '(' expr ')' '{' match_arm_list '}'           { $$ = Expr\Match_[$3, $6]; }
+;
+
+match_arm_list:
+      /* empty */                                           { $$ = []; }
+    | non_empty_match_arm_list optional_comma               { $$ = $1; }
+;
+
+non_empty_match_arm_list:
+      match_arm                                             { init($1); }
+    | non_empty_match_arm_list ',' match_arm                { push($1, $3); }
+;
+
+match_arm:
+      expr_list_allow_comma T_DOUBLE_ARROW expr             { $$ = Node\MatchArm[$1, $3]; }
+    | T_DEFAULT optional_comma T_DOUBLE_ARROW expr          { $$ = Node\MatchArm[null, $4]; }
+;
+
 while_statement:
       statement                                             { $$ = toArray($1); }
     | ':' inner_statement_list T_ENDWHILE ';'               { $$ = $2; }
@@ -440,7 +458,7 @@ foreach_variable:
 ;
 
 parameter_list:
-      non_empty_parameter_list no_comma                     { $$ = $1; }
+      non_empty_parameter_list optional_comma               { $$ = $1; }
     | /* empty */                                           { $$ = array(); }
 ;
 
@@ -449,18 +467,28 @@ non_empty_parameter_list:
     | non_empty_parameter_list ',' parameter                { push($1, $3); }
 ;
 
+optional_visibility_modifier:
+      /* empty */               { $$ = 0; }
+    | T_PUBLIC                  { $$ = Stmt\Class_::MODIFIER_PUBLIC; }
+    | T_PROTECTED               { $$ = Stmt\Class_::MODIFIER_PROTECTED; }
+    | T_PRIVATE                 { $$ = Stmt\Class_::MODIFIER_PRIVATE; }
+;
+
 parameter:
-      optional_type optional_ref optional_ellipsis plain_variable
+      optional_visibility_modifier optional_type optional_ref optional_ellipsis plain_variable
-          { $$ = Node\Param[$4, null, $1, $2, $3]; $this->checkParam($$); }
+          { $$ = new Node\Param($5, null, $2, $3, $4, attributes(), $1);
-    | optional_type optional_ref optional_ellipsis plain_variable '=' expr
+            $this->checkParam($$); }
-          { $$ = Node\Param[$4, $6, $1, $2, $3]; $this->checkParam($$); }
+    | optional_visibility_modifier optional_type optional_ref optional_ellipsis plain_variable '=' expr
-    | optional_type optional_ref optional_ellipsis error
+          { $$ = new Node\Param($5, $7, $2, $3, $4, attributes(), $1);
-          { $$ = Node\Param[Expr\Error[], null, $1, $2, $3]; }
+            $this->checkParam($$); }
+    | optional_visibility_modifier optional_type optional_ref optional_ellipsis error
+          { $$ = new Node\Param(Expr\Error[], null, $2, $3, $4, attributes(), $1); }
 ;
 
 type_expr:
       type                                                  { $$ = $1; }
     | '?' type                                              { $$ = Node\NullableType[$2]; }
+    | union_type                                            { $$ = Node\UnionType[$1]; }
 ;
 
 type:
@@ -469,6 +497,11 @@ type:
     | T_CALLABLE                                            { $$ = Node\Identifier['callable']; }
 ;
 
+union_type:
+      type '|' type                                         { init($1, $3); }
+    | union_type '|' type                                   { push($1, $3); }
+;
+
 optional_type:
       /* empty */                                           { $$ = null; }
     | type_expr                                             { $$ = $1; }
@@ -626,10 +659,14 @@ property_declaration:
     | property_decl_name '=' expr                           { $$ = Stmt\PropertyProperty[$1, $3]; }
 ;
 
-expr_list:
+expr_list_forbid_comma:
       non_empty_expr_list no_comma                          { $$ = $1; }
 ;
 
+expr_list_allow_comma:
+      non_empty_expr_list optional_comma                    { $$ = $1; }
+;
+
 non_empty_expr_list:
       non_empty_expr_list ',' expr                          { push($1, $3); }
     | expr                                                  { init($1); }
@@ -637,7 +674,7 @@ non_empty_expr_list:
 
 for_expr:
       /* empty */                                           { $$ = array(); }
-    | expr_list                                             { $$ = $1; }
+    | expr_list_forbid_comma                                { $$ = $1; }
 ;
 
 expr:
@@ -647,6 +684,7 @@ expr:
     | variable '=' expr                                     { $$ = Expr\Assign[$1, $3]; }
     | variable '=' '&' variable                             { $$ = Expr\AssignRef[$1, $4]; }
     | new_expr                                              { $$ = $1; }
+    | match                                                 { $$ = $1; }
     | T_CLONE expr                                          { $$ = Expr\Clone_[$2]; }
     | variable T_PLUS_EQUAL expr                            { $$ = Expr\AssignOp\Plus      [$1, $3]; }
     | variable T_MINUS_EQUAL expr                           { $$ = Expr\AssignOp\Minus     [$1, $3]; }
@@ -700,7 +738,7 @@ expr:
     | expr '?' expr ':' expr                                { $$ = Expr\Ternary[$1, $3,   $5]; }
     | expr '?' ':' expr                                     { $$ = Expr\Ternary[$1, null, $4]; }
     | expr T_COALESCE expr                                  { $$ = Expr\BinaryOp\Coalesce[$1, $3]; }
-    | T_ISSET '(' variables_list ')'                        { $$ = Expr\Isset_[$3]; }
+    | T_ISSET '(' expr_list_allow_comma ')'                 { $$ = Expr\Isset_[$3]; }
     | T_EMPTY '(' expr ')'                                  { $$ = Expr\Empty_[$3]; }
     | T_INCLUDE expr                                        { $$ = Expr\Include_[$2, Expr\Include_::TYPE_INCLUDE]; }
     | T_INCLUDE_ONCE expr                                   { $$ = Expr\Include_[$2, Expr\Include_::TYPE_INCLUDE_ONCE]; }
@@ -761,7 +799,7 @@ lexical_vars:
 ;
 
 lexical_var_list:
-      non_empty_lexical_var_list no_comma                   { $$ = $1; }
+      non_empty_lexical_var_list optional_comma             { $$ = $1; }
 ;
 
 non_empty_lexical_var_list:
@@ -786,20 +824,22 @@ class_name:
 ;
 
 name:
-      namespace_name_parts                                  { $$ = Name[$1]; }
+      T_STRING                                              { $$ = Name[$1]; }
-    | T_NS_SEPARATOR namespace_name_parts                   { $$ = Name\FullyQualified[$2]; }
+    | T_NAME_QUALIFIED                                      { $$ = Name[$1]; }
-    | T_NAMESPACE T_NS_SEPARATOR namespace_name_parts       { $$ = Name\Relative[$3]; }
+    | T_NAME_FULLY_QUALIFIED                                { $$ = Name\FullyQualified[substr($1, 1)]; }
+    | T_NAME_RELATIVE                                       { $$ = Name\Relative[substr($1, 10)]; }
 ;
 
 class_name_reference:
       class_name                                            { $$ = $1; }
     | new_variable                                          { $$ = $1; }
+    | '(' expr ')'                                          { $$ = $2; }
     | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 class_name_or_var:
       class_name                                            { $$ = $1; }
-    | dereferencable                                        { $$ = $1; }
+    | fully_dereferencable                                  { $$ = $1; }
 ;
 
 exit_expr:
@@ -821,9 +861,20 @@ ctor_arguments:
 
 constant:
       name                                                  { $$ = Expr\ConstFetch[$1]; }
-    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM identifier_ex
+    | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
+    | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
+    | T_DIR                                                 { $$ = Scalar\MagicConst\Dir[]; }
+    | T_CLASS_C                                             { $$ = Scalar\MagicConst\Class_[]; }
+    | T_TRAIT_C                                             { $$ = Scalar\MagicConst\Trait_[]; }
+    | T_METHOD_C                                            { $$ = Scalar\MagicConst\Method[]; }
+    | T_FUNC_C                                              { $$ = Scalar\MagicConst\Function_[]; }
+    | T_NS_C                                                { $$ = Scalar\MagicConst\Namespace_[]; }
+;
+
+class_constant:
+      class_name_or_var T_PAAMAYIM_NEKUDOTAYIM identifier_ex
           { $$ = Expr\ClassConstFetch[$1, $3]; }
-    /* We interpret and isolated FOO:: as an unfinished class constant fetch. It could also be
+    /* We interpret an isolated FOO:: as an unfinished class constant fetch. It could also be
        an unfinished static property fetch or unfinished scoped call. */
     | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM error
           { $$ = Expr\ClassConstFetch[$1, new Expr\Error(stackAttributes(#3))]; $this->errorState = 2; }
@@ -843,28 +894,21 @@ dereferencable_scalar:
     | T_CONSTANT_ENCAPSED_STRING
           { $attrs = attributes(); $attrs['kind'] = strKind($1);
             $$ = new Scalar\String_(Scalar\String_::parse($1), $attrs); }
+    | '"' encaps_list '"'
+          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
+            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
 ;
 
 scalar:
       T_LNUMBER                                             { $$ = $this->parseLNumber($1, attributes()); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
-    | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
-    | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
-    | T_DIR                                                 { $$ = Scalar\MagicConst\Dir[]; }
-    | T_CLASS_C                                             { $$ = Scalar\MagicConst\Class_[]; }
-    | T_TRAIT_C                                             { $$ = Scalar\MagicConst\Trait_[]; }
-    | T_METHOD_C                                            { $$ = Scalar\MagicConst\Method[]; }
-    | T_FUNC_C                                              { $$ = Scalar\MagicConst\Function_[]; }
-    | T_NS_C                                                { $$ = Scalar\MagicConst\Namespace_[]; }
     | dereferencable_scalar                                 { $$ = $1; }
     | constant                                              { $$ = $1; }
+    | class_constant                                        { $$ = $1; }
     | T_START_HEREDOC T_ENCAPSED_AND_WHITESPACE T_END_HEREDOC
           { $$ = $this->parseDocString($1, $2, $3, attributes(), stackAttributes(#3), true); }
     | T_START_HEREDOC T_END_HEREDOC
           { $$ = $this->parseDocString($1, '', $2, attributes(), stackAttributes(#2), true); }
-    | '"' encaps_list '"'
-          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
-            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
     | T_START_HEREDOC encaps_list T_END_HEREDOC
           { $$ = $this->parseDocString($1, $2, $3, attributes(), stackAttributes(#3), true); }
 ;
@@ -874,10 +918,16 @@ optional_expr:
     | expr                                                  { $$ = $1; }
 ;
 
-dereferencable:
+fully_dereferencable:
       variable                                              { $$ = $1; }
     | '(' expr ')'                                          { $$ = $2; }
     | dereferencable_scalar                                 { $$ = $1; }
+    | class_constant                                        { $$ = $1; }
+;
+
+array_object_dereferencable:
+      fully_dereferencable                                  { $$ = $1; }
+    | constant                                              { $$ = $1; }
 ;
 
 callable_expr:
@@ -888,18 +938,23 @@ callable_expr:
 
 callable_variable:
       simple_variable                                       { $$ = Expr\Variable[$1]; }
-    | dereferencable '[' optional_expr ']'                  { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | array_object_dereferencable '[' optional_expr ']'     { $$ = Expr\ArrayDimFetch[$1, $3]; }
-    | constant '[' optional_expr ']'                        { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | array_object_dereferencable '{' expr '}'              { $$ = Expr\ArrayDimFetch[$1, $3]; }
-    | dereferencable '{' expr '}'                           { $$ = Expr\ArrayDimFetch[$1, $3]; }
     | function_call                                         { $$ = $1; }
-    | dereferencable T_OBJECT_OPERATOR property_name argument_list
+    | array_object_dereferencable T_OBJECT_OPERATOR property_name argument_list
           { $$ = Expr\MethodCall[$1, $3, $4]; }
 ;
 
+optional_plain_variable:
+      /* empty */                                           { $$ = null; }
+    | plain_variable                                        { $$ = $1; }
+;
+
 variable:
       callable_variable                                     { $$ = $1; }
     | static_member                                         { $$ = $1; }
-    | dereferencable T_OBJECT_OPERATOR property_name        { $$ = Expr\PropertyFetch[$1, $3]; }
+    | array_object_dereferencable T_OBJECT_OPERATOR property_name
+          { $$ = Expr\PropertyFetch[$1, $3]; }
 ;
 
 simple_variable:
@@ -944,22 +999,7 @@ property_name:
 ;
 
 list_expr:
-      T_LIST '(' list_expr_elements ')'                     { $$ = Expr\List_[$3]; }
+      T_LIST '(' inner_array_pair_list ')'                  { $$ = Expr\List_[$3]; }
-;
-
-list_expr_elements:
-      list_expr_elements ',' list_expr_element              { push($1, $3); }
-    | list_expr_element                                     { init($1); }
-;
-
-list_expr_element:
-      variable                                              { $$ = Expr\ArrayItem[$1, null, false]; }
-    | '&' variable                                          { $$ = Expr\ArrayItem[$2, null, true]; }
-    | list_expr                                             { $$ = Expr\ArrayItem[$1, null, false]; }
-    | expr T_DOUBLE_ARROW variable                          { $$ = Expr\ArrayItem[$3, $1, false]; }
-    | expr T_DOUBLE_ARROW '&' variable                      { $$ = Expr\ArrayItem[$4, $1, true]; }
-    | expr T_DOUBLE_ARROW list_expr                         { $$ = Expr\ArrayItem[$3, $1, false]; }
-    | /* empty */                                           { $$ = null; }
 ;
 
 array_pair_list:
@@ -979,10 +1019,12 @@ inner_array_pair_list:
 ;
 
 array_pair:
-      expr T_DOUBLE_ARROW expr                              { $$ = Expr\ArrayItem[$3, $1,   false]; }
+      expr                                                  { $$ = Expr\ArrayItem[$1, null, false]; }
-    | expr                                                  { $$ = Expr\ArrayItem[$1, null, false]; }
-    | expr T_DOUBLE_ARROW '&' variable                      { $$ = Expr\ArrayItem[$4, $1,   true]; }
     | '&' variable                                          { $$ = Expr\ArrayItem[$2, null, true]; }
+    | list_expr                                             { $$ = Expr\ArrayItem[$1, null, false]; }
+    | expr T_DOUBLE_ARROW expr                              { $$ = Expr\ArrayItem[$3, $1,   false]; }
+    | expr T_DOUBLE_ARROW '&' variable                      { $$ = Expr\ArrayItem[$4, $1,   true]; }
+    | expr T_DOUBLE_ARROW list_expr                         { $$ = Expr\ArrayItem[$3, $1,   false]; }
     | T_ELLIPSIS expr                                       { $$ = Expr\ArrayItem[$2, null, false, attributes(), true]; }
     | /* empty */                                           { $$ = null; }
 ;

grammar/rebuildParsers.php

@@ -13,13 +13,10 @@ $tmpResultFile  = __DIR__ . '/tmp_parser.php';
 $resultDir = __DIR__ . '/../lib/PhpParser/Parser';
 $tokensResultsFile = $resultDir . '/Tokens.php';
 
-// check for kmyacc binary in this directory, otherwise fall back to global name
+$kmyacc = getenv('KMYACC');
-if (file_exists(__DIR__ . '/kmyacc.exe')) {
+if (!$kmyacc) {
-    $kmyacc = __DIR__ . '/kmyacc.exe';
+    // Use phpyacc from dev dependencies by default.
-} else if (file_exists(__DIR__ . '/kmyacc')) {
+    $kmyacc = __DIR__ . '/../vendor/bin/phpyacc';
-    $kmyacc = __DIR__ . '/kmyacc';
-} else {
-    $kmyacc = 'kmyacc';
 }
 
 $options = array_flip($argv);
@@ -62,8 +59,7 @@ foreach ($grammarFileToName as $grammarFile => $name) {
     $additionalArgs = $optionDebug ? '-t -v' : '';
 
     echo "Building $name parser.\n";
-    $output = trim(shell_exec("$kmyacc $additionalArgs -l -m $skeletonFile -p $name $tmpGrammarFile 2>&1"));
+    $output = execCmd("$kmyacc $additionalArgs -m $skeletonFile -p $name $tmpGrammarFile");
-    echo "Output: \"$output\"\n";
 
     $resultCode = file_get_contents($tmpResultFile);
     $resultCode = removeTrailingWhitespace($resultCode);
@@ -73,8 +69,7 @@ foreach ($grammarFileToName as $grammarFile => $name) {
     unlink($tmpResultFile);
 
     echo "Building token definition.\n";
-    $output = trim(shell_exec("$kmyacc -l -m $tokensTemplate $tmpGrammarFile 2>&1"));
+    $output = execCmd("$kmyacc -m $tokensTemplate $tmpGrammarFile");
-    assert($output === '');
     rename($tmpResultFile, $tokensResultsFile);
 
     if (!$optionKeepTmpGrammar) {
@@ -183,7 +178,7 @@ function resolveMacros($code) {
 
                 return '$startAttributes = ' . $args[1] . ';'
                     . ' if (isset($startAttributes[\'comments\']))'
-                    . ' { ' . $args[0] . ' = new Stmt\Nop($this->createZeroLengthAttributes($startAttributes)); }'
+                    . ' { ' . $args[0] . ' = new Stmt\Nop($this->createCommentNopAttributes($startAttributes[\'comments\'])); }'
                     . ' else { ' . $args[0] . ' = null; }';
             }
 
@@ -234,6 +229,15 @@ function ensureDirExists($dir) {
     }
 }
 
+function execCmd($cmd) {
+    $output = trim(shell_exec("$cmd 2>&1"));
+    if ($output !== "") {
+        echo "> " . $cmd . "\n";
+        echo $output;
+    }
+    return $output;
+}
+
 //////////////////////////////
 /// Regex helper functions ///
 //////////////////////////////

grammar/tokens.y

@@ -57,6 +57,7 @@
 %token T_ENDDECLARE
 %token T_AS
 %token T_SWITCH
+%token T_MATCH
 %token T_ENDSWITCH
 %token T_CASE
 %token T_DEFAULT
@@ -112,3 +113,6 @@
 %token T_DIR
 %token T_NS_SEPARATOR
 %token T_ELLIPSIS
+%token T_NAME_FULLY_QUALIFIED
+%token T_NAME_QUALIFIED
+%token T_NAME_RELATIVE

lib/PhpParser/Builder/Param.php

@@ -12,7 +12,7 @@ class Param implements PhpParser\Builder
 
     protected $default = null;
 
-    /** @var string|Node\Name|Node\NullableType|null */
+    /** @var Node\Identifier|Node\Name|Node\NullableType|null */
     protected $type = null;
 
     protected $byRef = false;
@@ -44,7 +44,7 @@ class Param implements PhpParser\Builder
     /**
      * Sets type for the parameter.
      *
-     * @param string|Node\Name|Node\NullableType $type Parameter type
+     * @param string|Node\Name|Node\NullableType|Node\UnionType $type Parameter type
      *
      * @return $this The builder instance (for fluid interface)
      */
@@ -60,7 +60,7 @@ class Param implements PhpParser\Builder
     /**
      * Sets type for the parameter.
      *
-     * @param string|Node\Name|Node\NullableType $type Parameter type
+     * @param string|Node\Name|Node\NullableType|Node\UnionType $type Parameter type
      *
      * @return $this The builder instance (for fluid interface)
      *

lib/PhpParser/BuilderHelpers.php

@@ -8,6 +8,7 @@ use PhpParser\Node\Name;
 use PhpParser\Node\NullableType;
 use PhpParser\Node\Scalar;
 use PhpParser\Node\Stmt;
+use PhpParser\Node\UnionType;
 
 /**
  * This class defines helpers used in the implementation of builders. Don't use it directly.
@@ -158,16 +159,19 @@ final class BuilderHelpers
      * In particular, builtin types become Identifiers, custom types become Names and nullables
      * are wrapped in NullableType nodes.
      *
-     * @param string|Name|Identifier|NullableType $type The type to normalize
+     * @param string|Name|Identifier|NullableType|UnionType $type The type to normalize
      *
-     * @return Name|Identifier|NullableType The normalized type
+     * @return Name|Identifier|NullableType|UnionType The normalized type
      */
     public static function normalizeType($type) {
         if (!is_string($type)) {
-            if (!$type instanceof Name && !$type instanceof Identifier
+            if (
-                    && !$type instanceof NullableType) {
+                !$type instanceof Name && !$type instanceof Identifier &&
+                !$type instanceof NullableType && !$type instanceof UnionType
+            ) {
                 throw new \LogicException(
-                    'Type must be a string, or an instance of Name, Identifier or NullableType');
+                    'Type must be a string, or an instance of Name, Identifier, NullableType or UnionType'
+                );
             }
             return $type;
         }
@@ -179,7 +183,7 @@ final class BuilderHelpers
         }
 
         $builtinTypes = [
-            'array', 'callable', 'string', 'int', 'float', 'bool', 'iterable', 'void', 'object'
+            'array', 'callable', 'string', 'int', 'float', 'bool', 'iterable', 'void', 'object', 'mixed'
         ];
 
         $lowerType = strtolower($type);
@@ -193,7 +197,11 @@ final class BuilderHelpers
             throw new \LogicException('void type cannot be nullable');
         }
 
-        return $nullable ? new Node\NullableType($type) : $type;
+        if ($nullable && (string) $type === 'mixed') {
+            throw new \LogicException('mixed type cannot be nullable');
+        }
+
+        return $nullable ? new NullableType($type) : $type;
     }
 
     /**

lib/PhpParser/Comment.php

@@ -5,9 +5,12 @@ namespace PhpParser;
 class Comment implements \JsonSerializable
 {
     protected $text;
-    protected $line;
+    protected $startLine;
-    protected $filePos;
+    protected $startFilePos;
-    protected $tokenPos;
+    protected $startTokenPos;
+    protected $endLine;
+    protected $endFilePos;
+    protected $endTokenPos;
 
     /**
      * Constructs a comment node.
@@ -18,12 +21,17 @@ class Comment implements \JsonSerializable
      * @param int    $startTokenPos Token offset the comment started on
      */
     public function __construct(
-        string $text, int $startLine = -1, int $startFilePos = -1, int $startTokenPos = -1
+        string $text,
+        int $startLine = -1, int $startFilePos = -1, int $startTokenPos = -1,
+        int $endLine = -1, int $endFilePos = -1, int $endTokenPos = -1
     ) {
         $this->text = $text;
-        $this->line = $startLine;
+        $this->startLine = $startLine;
-        $this->filePos = $startFilePos;
+        $this->startFilePos = $startFilePos;
-        $this->tokenPos = $startTokenPos;
+        $this->startTokenPos = $startTokenPos;
+        $this->endLine = $endLine;
+        $this->endFilePos = $endFilePos;
+        $this->endTokenPos = $endTokenPos;
     }
 
     /**
@@ -38,28 +46,88 @@ class Comment implements \JsonSerializable
     /**
      * Gets the line number the comment started on.
      *
-     * @return int Line number
+     * @return int Line number (or -1 if not available)
      */
-    public function getLine() : int {
+    public function getStartLine() : int {
-        return $this->line;
+        return $this->startLine;
     }
 
     /**
      * Gets the file offset the comment started on.
      *
-     * @return int File offset
+     * @return int File offset (or -1 if not available)
      */
-    public function getFilePos() : int {
+    public function getStartFilePos() : int {
-        return $this->filePos;
+        return $this->startFilePos;
     }
 
     /**
      * Gets the token offset the comment started on.
      *
+     * @return int Token offset (or -1 if not available)
+     */
+    public function getStartTokenPos() : int {
+        return $this->startTokenPos;
+    }
+
+    /**
+     * Gets the line number the comment ends on.
+     *
+     * @return int Line number (or -1 if not available)
+     */
+    public function getEndLine() : int {
+        return $this->endLine;
+    }
+
+    /**
+     * Gets the file offset the comment ends on.
+     *
+     * @return int File offset (or -1 if not available)
+     */
+    public function getEndFilePos() : int {
+        return $this->endFilePos;
+    }
+
+    /**
+     * Gets the token offset the comment ends on.
+     *
+     * @return int Token offset (or -1 if not available)
+     */
+    public function getEndTokenPos() : int {
+        return $this->endTokenPos;
+    }
+
+    /**
+     * Gets the line number the comment started on.
+     *
+     * @deprecated Use getStartLine() instead
+     *
+     * @return int Line number
+     */
+    public function getLine() : int {
+        return $this->startLine;
+    }
+
+    /**
+     * Gets the file offset the comment started on.
+     *
+     * @deprecated Use getStartFilePos() instead
+     *
+     * @return int File offset
+     */
+    public function getFilePos() : int {
+        return $this->startFilePos;
+    }
+
+    /**
+     * Gets the token offset the comment started on.
+     *
+     * @deprecated Use getStartTokenPos() instead
+     *
      * @return int Token offset
      */
     public function getTokenPos() : int {
-        return $this->tokenPos;
+        return $this->startTokenPos;
     }
 
     /**
@@ -159,9 +227,13 @@ class Comment implements \JsonSerializable
         return [
             'nodeType' => $type,
             'text' => $this->text,
-            'line' => $this->line,
+            // TODO: Rename these to include "start".
-            'filePos' => $this->filePos,
+            'line' => $this->startLine,
-            'tokenPos' => $this->tokenPos,
+            'filePos' => $this->startFilePos,
+            'tokenPos' => $this->startTokenPos,
+            'endLine' => $this->endLine,
+            'endFilePos' => $this->endFilePos,
+            'endTokenPos' => $this->endTokenPos,
         ];
     }
 }

lib/PhpParser/JsonDecoder.php

@@ -73,7 +73,9 @@ class JsonDecoder
         }
 
         return new $className(
-            $value['text'], $value['line'] ?? -1, $value['filePos'] ?? -1, $value['tokenPos'] ?? -1
+            $value['text'],
+            $value['line'] ?? -1, $value['filePos'] ?? -1, $value['tokenPos'] ?? -1,
+            $value['endLine'] ?? -1, $value['endFilePos'] ?? -1, $value['endTokenPos'] ?? -1
         );
     }
 

lib/PhpParser/Lexer.php

@@ -34,13 +34,29 @@ class Lexer
      *                       first three. For more info see getNextToken() docs.
      */
     public function __construct(array $options = []) {
-        // map from internal tokens to PhpParser tokens
+        // Compatibility define for PHP < 7.4.
+        if (!defined('T_BAD_CHARACTER')) {
+            \define('T_BAD_CHARACTER', -1);
+        }
+
+        // Compatibility defines for PHP < 8.0.
+        if (!defined('T_NAME_QUALIFIED')) {
+            \define('T_NAME_QUALIFIED', -2);
+        }
+        if (!defined('T_NAME_FULLY_QUALIFIED')) {
+            \define('T_NAME_FULLY_QUALIFIED', -3);
+        }
+        if (!defined('T_NAME_RELATIVE')) {
+            \define('T_NAME_RELATIVE', -4);
+        }
+
+        // Create Map from internal tokens to PhpParser tokens.
         $this->tokenMap = $this->createTokenMap();
 
         // map of tokens to drop while lexing (the map is only used for isset lookup,
         // that's why the value is simply set to 1; the value is never actually used.)
         $this->dropTokens = array_fill_keys(
-            [\T_WHITESPACE, \T_OPEN_TAG, \T_COMMENT, \T_DOC_COMMENT], 1
+            [\T_WHITESPACE, \T_OPEN_TAG, \T_COMMENT, \T_DOC_COMMENT, \T_BAD_CHARACTER], 1
         );
 
         $defaultAttributes = ['comments', 'startLine', 'endLine'];
@@ -84,7 +100,7 @@ class Lexer
 
         error_clear_last();
         $this->tokens = @token_get_all($code);
-        $this->handleErrors($errorHandler);
+        $this->postprocessTokens($errorHandler);
 
         if (false !== $scream) {
             ini_set('xdebug.scream', $scream);
@@ -92,13 +108,9 @@ class Lexer
     }
 
     private function handleInvalidCharacterRange($start, $end, $line, ErrorHandler $errorHandler) {
+        $tokens = [];
         for ($i = $start; $i < $end; $i++) {
             $chr = $this->code[$i];
-            if ($chr === 'b' || $chr === 'B') {
-                // HHVM does not treat b" tokens correctly, so ignore these
-                continue;
-            }
-
             if ($chr === "\0") {
                 // PHP cuts error message after null byte, so need special case
                 $errorMsg = 'Unexpected null byte';
@@ -108,6 +120,7 @@ class Lexer
                 );
             }
 
+            $tokens[] = [\T_BAD_CHARACTER, $chr, $line];
             $errorHandler->handleError(new Error($errorMsg, [
                 'startLine' => $line,
                 'endLine' => $line,
@@ -115,6 +128,7 @@ class Lexer
                 'endFilePos' => $i,
             ]));
         }
+        return $tokens;
     }
 
     /**
@@ -128,43 +142,100 @@ class Lexer
             && substr($token[1], -2) !== '*/';
     }
 
-    /**
+    protected function postprocessTokens(ErrorHandler $errorHandler) {
-     * Check whether an error *may* have occurred during tokenization.
-     *
-     * @return bool
-     */
-    private function errorMayHaveOccurred() : bool {
-        if (defined('HHVM_VERSION')) {
-            // In HHVM token_get_all() does not throw warnings, so we need to conservatively
-            // assume that an error occurred
-            return true;
-        }
-
-        return null !== error_get_last();
-    }
-
-    protected function handleErrors(ErrorHandler $errorHandler) {
-        if (!$this->errorMayHaveOccurred()) {
-            return;
-        }
-
         // PHP's error handling for token_get_all() is rather bad, so if we want detailed
         // error information we need to compute it ourselves. Invalid character errors are
         // detected by finding "gaps" in the token array. Unterminated comments are detected
         // by checking if a trailing comment has a "*/" at the end.
+        //
+        // Additionally, we canonicalize to the PHP 8 comment format here, which does not include
+        // the trailing whitespace anymore.
+        //
+        // We also canonicalize to the PHP 8 T_NAME_* tokens.
 
         $filePos = 0;
         $line = 1;
-        foreach ($this->tokens as $token) {
+        $numTokens = \count($this->tokens);
+        for ($i = 0; $i < $numTokens; $i++) {
+            $token = $this->tokens[$i];
+
+            // Since PHP 7.4 invalid characters are represented by a T_BAD_CHARACTER token.
+            // In this case we only need to emit an error.
+            if ($token[0] === \T_BAD_CHARACTER) {
+                $this->handleInvalidCharacterRange($filePos, $filePos + 1, $line, $errorHandler);
+            }
+
+            if ($token[0] === \T_COMMENT && substr($token[1], 0, 2) !== '/*'
+                    && preg_match('/(\r\n|\n|\r)$/D', $token[1], $matches)) {
+                $trailingNewline = $matches[0];
+                $token[1] = substr($token[1], 0, -strlen($trailingNewline));
+                $this->tokens[$i] = $token;
+                if (isset($this->tokens[$i + 1]) && $this->tokens[$i + 1][0] === \T_WHITESPACE) {
+                    // Move trailing newline into following T_WHITESPACE token, if it already exists.
+                    $this->tokens[$i + 1][1] = $trailingNewline . $this->tokens[$i + 1][1];
+                    $this->tokens[$i + 1][2]--;
+                } else {
+                    // Otherwise, we need to create a new T_WHITESPACE token.
+                    array_splice($this->tokens, $i + 1, 0, [
+                        [\T_WHITESPACE, $trailingNewline, $line],
+                    ]);
+                    $numTokens++;
+                }
+            }
+
+            // Emulate PHP 8 T_NAME_* tokens, by combining sequences of T_NS_SEPARATOR and T_STRING
+            // into a single token.
+            // TODO: Also handle reserved keywords in namespaced names.
+            if (\is_array($token)
+                    && ($token[0] === \T_NS_SEPARATOR || $token[0] === \T_STRING || $token[0] === \T_NAMESPACE)) {
+                $lastWasSeparator = $token[0] === \T_NS_SEPARATOR;
+                $text = $token[1];
+                for ($j = $i + 1; isset($this->tokens[$j]); $j++) {
+                    if ($lastWasSeparator) {
+                        if ($this->tokens[$j][0] !== \T_STRING) {
+                            break;
+                        }
+                        $lastWasSeparator = false;
+                    } else {
+                        if ($this->tokens[$j][0] !== \T_NS_SEPARATOR) {
+                            break;
+                        }
+                        $lastWasSeparator = true;
+                    }
+                    $text .= $this->tokens[$j][1];
+                }
+                if ($lastWasSeparator) {
+                    // Trailing separator is not part of the name.
+                    $j--;
+                    $text = substr($text, 0, -1);
+                }
+                if ($j > $i + 1) {
+                    if ($token[0] === \T_NS_SEPARATOR) {
+                        $type = \T_NAME_FULLY_QUALIFIED;
+                    } else if ($token[0] === \T_NAMESPACE) {
+                        $type = \T_NAME_RELATIVE;
+                    } else {
+                        $type = \T_NAME_QUALIFIED;
+                    }
+                    $token = [$type, $text, $line];
+                    array_splice($this->tokens, $i, $j - $i, [$token]);
+                    $numTokens -= $j - $i - 1;
+                }
+            }
+
             $tokenValue = \is_string($token) ? $token : $token[1];
             $tokenLen = \strlen($tokenValue);
 
             if (substr($this->code, $filePos, $tokenLen) !== $tokenValue) {
                 // Something is missing, must be an invalid character
                 $nextFilePos = strpos($this->code, $tokenValue, $filePos);
-                $this->handleInvalidCharacterRange(
+                $badCharTokens = $this->handleInvalidCharacterRange(
                     $filePos, $nextFilePos, $line, $errorHandler);
                 $filePos = (int) $nextFilePos;
+
+                array_splice($this->tokens, $i, 0, $badCharTokens);
+                $numTokens += \count($badCharTokens);
+                $i += \count($badCharTokens);
             }
 
             $filePos += $tokenLen;
@@ -187,8 +258,9 @@ class Lexer
                 $this->tokens[] = [$isDocComment ? \T_DOC_COMMENT : \T_COMMENT, $comment, $line];
             } else {
                 // Invalid characters at the end of the input
-                $this->handleInvalidCharacterRange(
+                $badCharTokens = $this->handleInvalidCharacterRange(
                     $filePos, \strlen($this->code), $line, $errorHandler);
+                $this->tokens = array_merge($this->tokens, $badCharTokens);
             }
             return;
         }
@@ -273,17 +345,23 @@ class Lexer
                 $this->line += substr_count($value, "\n");
                 $this->filePos += \strlen($value);
             } else {
+                $origLine = $this->line;
+                $origFilePos = $this->filePos;
+                $this->line += substr_count($token[1], "\n");
+                $this->filePos += \strlen($token[1]);
+
                 if (\T_COMMENT === $token[0] || \T_DOC_COMMENT === $token[0]) {
                     if ($this->attributeCommentsUsed) {
                         $comment = \T_DOC_COMMENT === $token[0]
-                            ? new Comment\Doc($token[1], $this->line, $this->filePos, $this->pos)
+                            ? new Comment\Doc($token[1],
-                            : new Comment($token[1], $this->line, $this->filePos, $this->pos);
+                                $origLine, $origFilePos, $this->pos,
+                                $this->line, $this->filePos - 1, $this->pos)
+                            : new Comment($token[1],
+                                $origLine, $origFilePos, $this->pos,
+                                $this->line, $this->filePos - 1, $this->pos);
                         $startAttributes['comments'][] = $comment;
                     }
                 }
-
-                $this->line += substr_count($token[1], "\n");
-                $this->filePos += \strlen($token[1]);
                 continue;
             }
 
@@ -384,6 +462,11 @@ class Lexer
             $tokenMap[\T_COMPILER_HALT_OFFSET] = Tokens::T_STRING;
         }
 
+        // Assign tokens for which we define compatibility constants, as token_name() does not know them.
+        $tokenMap[\T_NAME_QUALIFIED] = Tokens::T_NAME_QUALIFIED;
+        $tokenMap[\T_NAME_FULLY_QUALIFIED] = Tokens::T_NAME_FULLY_QUALIFIED;
+        $tokenMap[\T_NAME_RELATIVE] = Tokens::T_NAME_RELATIVE;
+
         return $tokenMap;
     }
 }

lib/PhpParser/Lexer/Emulative.php

@@ -7,17 +7,25 @@ use PhpParser\ErrorHandler;
 use PhpParser\Lexer;
 use PhpParser\Lexer\TokenEmulator\CoaleseEqualTokenEmulator;
 use PhpParser\Lexer\TokenEmulator\FnTokenEmulator;
+use PhpParser\Lexer\TokenEmulator\MatchTokenEmulator;
+use PhpParser\Lexer\TokenEmulator\NumericLiteralSeparatorEmulator;
 use PhpParser\Lexer\TokenEmulator\TokenEmulatorInterface;
+use PhpParser\Parser\Tokens;
 
 class Emulative extends Lexer
 {
     const PHP_7_3 = '7.3.0dev';
     const PHP_7_4 = '7.4.0dev';
+    const PHP_8_0 = '8.0.0dev';
+
+    const T_COALESCE_EQUAL = 1007;
+    const T_FN = 1008;
+    const T_MATCH = 1009;
 
     const FLEXIBLE_DOC_STRING_REGEX = <<<'REGEX'
 /<<<[ \t]*(['"]?)([a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*)\1\r?\n
 (?:.*\r?\n)*?
-(?<indentation>\h*)\2(?![a-zA-Z_\x80-\xff])(?<separator>(?:;?[\r\n])?)/x
+(?<indentation>\h*)\2(?![a-zA-Z0-9_\x80-\xff])(?<separator>(?:;?[\r\n])?)/x
 REGEX;
 
     /** @var mixed[] Patches used to reverse changes introduced in the code */
@@ -33,14 +41,14 @@ REGEX;
     {
         parent::__construct($options);
 
-        // prepare token emulators
         $this->tokenEmulators[] = new FnTokenEmulator();
+        $this->tokenEmulators[] = new MatchTokenEmulator();
         $this->tokenEmulators[] = new CoaleseEqualTokenEmulator();
+        $this->tokenEmulators[] = new NumericLiteralSeparatorEmulator();
 
-        // add emulated tokens here
+        $this->tokenMap[self::T_COALESCE_EQUAL] = Tokens::T_COALESCE_EQUAL;
-        foreach ($this->tokenEmulators as $emulativeToken) {
+        $this->tokenMap[self::T_FN] = Tokens::T_FN;
-            $this->tokenMap[$emulativeToken->getTokenId()] = $emulativeToken->getParserTokenId();
+        $this->tokenMap[self::T_MATCH] = Tokens::T_MATCH;
-        }
     }
 
     public function startLexing(string $code, ErrorHandler $errorHandler = null) {
@@ -57,14 +65,6 @@ REGEX;
         // 1. emulation of heredoc and nowdoc new syntax
         $preparedCode = $this->processHeredocNowdoc($code);
         parent::startLexing($preparedCode, $collector);
-
-        // add token emulation
-        foreach ($this->tokenEmulators as $emulativeToken) {
-            if ($emulativeToken->isEmulationNeeded($code)) {
-                $this->tokens = $emulativeToken->emulate($code, $this->tokens);
-            }
-        }
-
         $this->fixupTokens();
 
         $errors = $collector->getErrors();
@@ -74,6 +74,13 @@ REGEX;
                 $errorHandler->handleError($error);
             }
         }
+
+        // add token emulation
+        foreach ($this->tokenEmulators as $emulativeToken) {
+            if ($emulativeToken->isEmulationNeeded($code)) {
+                $this->tokens = $emulativeToken->emulate($code, $this->tokens);
+            }
+        }
     }
 
     private function isHeredocNowdocEmulationNeeded(string $code): bool

lib/PhpParser/Lexer/TokenEmulator/CoaleseEqualTokenEmulator.php

@@ -3,22 +3,9 @@
 namespace PhpParser\Lexer\TokenEmulator;
 
 use PhpParser\Lexer\Emulative;
-use PhpParser\Parser\Tokens;
 
 final class CoaleseEqualTokenEmulator implements TokenEmulatorInterface
 {
-    const T_COALESCE_EQUAL = 1007;
-
-    public function getTokenId(): int
-    {
-        return self::T_COALESCE_EQUAL;
-    }
-
-    public function getParserTokenId(): int
-    {
-        return Tokens::T_COALESCE_EQUAL;
-    }
-
     public function isEmulationNeeded(string $code) : bool
     {
         // skip version where this is supported
@@ -38,7 +25,7 @@ final class CoaleseEqualTokenEmulator implements TokenEmulatorInterface
             if (isset($tokens[$i + 1])) {
                 if ($tokens[$i][0] === T_COALESCE && $tokens[$i + 1] === '=') {
                     array_splice($tokens, $i, 2, [
-                        [self::T_COALESCE_EQUAL, '??=', $line]
+                        [Emulative::T_COALESCE_EQUAL, '??=', $line]
                     ]);
                     $c--;
                     continue;

lib/PhpParser/Lexer/TokenEmulator/FnTokenEmulator.php

@@ -3,22 +3,9 @@
 namespace PhpParser\Lexer\TokenEmulator;
 
 use PhpParser\Lexer\Emulative;
-use PhpParser\Parser\Tokens;
 
 final class FnTokenEmulator implements TokenEmulatorInterface
 {
-    const T_FN = 1008;
-
-    public function getTokenId(): int
-    {
-        return self::T_FN;
-    }
-
-    public function getParserTokenId(): int
-    {
-        return Tokens::T_FN;
-    }
-
     public function isEmulationNeeded(string $code) : bool
     {
         // skip version where this is supported
@@ -31,16 +18,14 @@ final class FnTokenEmulator implements TokenEmulatorInterface
 
     public function emulate(string $code, array $tokens): array
     {
-        // We need to manually iterate and manage a count because we'll change
-        // the tokens array on the way
         foreach ($tokens as $i => $token) {
-            if ($token[0] === T_STRING && $token[1] === 'fn') {
+            if ($token[0] === T_STRING && strtolower($token[1]) === 'fn') {
                 $previousNonSpaceToken = $this->getPreviousNonSpaceToken($tokens, $i);
                 if ($previousNonSpaceToken !== null && $previousNonSpaceToken[0] === T_OBJECT_OPERATOR) {
                     continue;
                 }
 
-                $tokens[$i][0] = self::T_FN;
+                $tokens[$i][0] = Emulative::T_FN;
             }
         }
 

lib/PhpParser/Lexer/TokenEmulator/MatchTokenEmulator.php

@@ -0,0 +1,51 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\Lexer\TokenEmulator;
+
+use PhpParser\Lexer\Emulative;
+
+final class MatchTokenEmulator implements TokenEmulatorInterface
+{
+    public function isEmulationNeeded(string $code) : bool
+    {
+        // skip version where this is supported
+        if (version_compare(\PHP_VERSION, Emulative::PHP_8_0, '>=')) {
+            return false;
+        }
+
+        return strpos($code, 'match') !== false;
+    }
+
+    public function emulate(string $code, array $tokens): array
+    {
+        foreach ($tokens as $i => $token) {
+            if ($token[0] === T_STRING && strtolower($token[1]) === 'match') {
+                $previousNonSpaceToken = $this->getPreviousNonSpaceToken($tokens, $i);
+                if ($previousNonSpaceToken !== null && $previousNonSpaceToken[0] === T_OBJECT_OPERATOR) {
+                    continue;
+                }
+
+                $tokens[$i][0] = Emulative::T_MATCH;
+            }
+        }
+
+        return $tokens;
+    }
+
+    /**
+     * @param mixed[] $tokens
+     * @return mixed[]|null
+     */
+    private function getPreviousNonSpaceToken(array $tokens, int $start)
+    {
+        for ($i = $start - 1; $i >= 0; --$i) {
+            if ($tokens[$i][0] === T_WHITESPACE) {
+                continue;
+            }
+
+            return $tokens[$i];
+        }
+
+        return null;
+    }
+}

lib/PhpParser/Lexer/TokenEmulator/NumericLiteralSeparatorEmulator.php

@@ -0,0 +1,99 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\Lexer\TokenEmulator;
+
+use PhpParser\Lexer\Emulative;
+
+final class NumericLiteralSeparatorEmulator implements TokenEmulatorInterface
+{
+    const BIN = '(?:0b[01]+(?:_[01]+)*)';
+    const HEX = '(?:0x[0-9a-f]+(?:_[0-9a-f]+)*)';
+    const DEC = '(?:[0-9]+(?:_[0-9]+)*)';
+    const SIMPLE_FLOAT = '(?:' . self::DEC . '\.' . self::DEC . '?|\.' . self::DEC . ')';
+    const EXP = '(?:e[+-]?' . self::DEC . ')';
+    const FLOAT = '(?:' . self::SIMPLE_FLOAT . self::EXP . '?|' . self::DEC . self::EXP . ')';
+    const NUMBER = '~' . self::FLOAT . '|' . self::BIN . '|' . self::HEX . '|' . self::DEC . '~iA';
+
+    public function isEmulationNeeded(string $code) : bool
+    {
+        // skip version where this is supported
+        if (version_compare(\PHP_VERSION, Emulative::PHP_7_4, '>=')) {
+            return false;
+        }
+
+        return preg_match('~[0-9]_[0-9]~', $code)
+            || preg_match('~0x[0-9a-f]+_[0-9a-f]~i', $code);
+    }
+
+    public function emulate(string $code, array $tokens): array
+    {
+        // We need to manually iterate and manage a count because we'll change
+        // the tokens array on the way
+        $codeOffset = 0;
+        for ($i = 0, $c = count($tokens); $i < $c; ++$i) {
+            $token = $tokens[$i];
+            $tokenLen = \strlen(\is_array($token) ? $token[1] : $token);
+
+            if ($token[0] !== T_LNUMBER && $token[0] !== T_DNUMBER) {
+                $codeOffset += $tokenLen;
+                continue;
+            }
+
+            $res = preg_match(self::NUMBER, $code, $matches, 0, $codeOffset);
+            assert($res, "No number at number token position");
+
+            $match = $matches[0];
+            $matchLen = \strlen($match);
+            if ($matchLen === $tokenLen) {
+                // Original token already holds the full number.
+                $codeOffset += $tokenLen;
+                continue;
+            }
+
+            $tokenKind = $this->resolveIntegerOrFloatToken($match);
+            $newTokens = [[$tokenKind, $match, $token[2]]];
+
+            $numTokens = 1;
+            $len = $tokenLen;
+            while ($matchLen > $len) {
+                $nextToken = $tokens[$i + $numTokens];
+                $nextTokenText = \is_array($nextToken) ? $nextToken[1] : $nextToken;
+                $nextTokenLen = \strlen($nextTokenText);
+
+                $numTokens++;
+                if ($matchLen < $len + $nextTokenLen) {
+                    // Split trailing characters into a partial token.
+                    assert(is_array($nextToken), "Partial token should be an array token");
+                    $partialText = substr($nextTokenText, $matchLen - $len);
+                    $newTokens[] = [$nextToken[0], $partialText, $nextToken[2]];
+                    break;
+                }
+
+                $len += $nextTokenLen;
+            }
+
+            array_splice($tokens, $i, $numTokens, $newTokens);
+            $c -= $numTokens - \count($newTokens);
+            $codeOffset += $matchLen;
+        }
+
+        return $tokens;
+    }
+
+    private function resolveIntegerOrFloatToken(string $str): int
+    {
+        $str = str_replace('_', '', $str);
+
+        if (stripos($str, '0b') === 0) {
+            $num = bindec($str);
+        } elseif (stripos($str, '0x') === 0) {
+            $num = hexdec($str);
+        } elseif (stripos($str, '0') === 0 && ctype_digit($str)) {
+            $num = octdec($str);
+        } else {
+            $num = +$str;
+        }
+
+        return is_float($num) ? T_DNUMBER : T_LNUMBER;
+    }
+}

lib/PhpParser/Lexer/TokenEmulator/TokenEmulatorInterface.php

@@ -2,12 +2,9 @@
 
 namespace PhpParser\Lexer\TokenEmulator;
 
+/** @internal */
 interface TokenEmulatorInterface
 {
-    public function getTokenId(): int;
-
-    public function getParserTokenId(): int;
-
     public function isEmulationNeeded(string $code): bool;
 
     /**

lib/PhpParser/Node.php

@@ -95,8 +95,6 @@ interface Node
     /**
      * Gets the doc comment of the node.
      *
-     * The doc comment has to be the last comment associated with the node.
-     *
      * @return null|Comment\Doc Doc comment object or null
      */
     public function getDocComment();

lib/PhpParser/Node/Expr/ArrowFunction.php

@@ -17,7 +17,7 @@ class ArrowFunction extends Expr implements FunctionLike
     /** @var Node\Param[] */
     public $params = [];
 
-    /** @var null|Node\Identifier|Node\Name|Node\NullableType */
+    /** @var null|Node\Identifier|Node\Name|Node\NullableType|Node\UnionType */
     public $returnType;
 
     /** @var Expr */

lib/PhpParser/Node/Expr/Closure.php

@@ -16,7 +16,7 @@ class Closure extends Expr implements FunctionLike
     public $params;
     /** @var ClosureUse[] use()s */
     public $uses;
-    /** @var null|Node\Identifier|Node\Name|Node\NullableType Return type */
+    /** @var null|Node\Identifier|Node\Name|Node\NullableType|Node\UnionType Return type */
     public $returnType;
     /** @var Node\Stmt[] Statements */
     public $stmts;

lib/PhpParser/Node/Expr/Match_.php

@@ -0,0 +1,31 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\Node\Expr;
+
+use PhpParser\Node;
+use PhpParser\Node\MatchArm;
+
+class Match_ extends Node\Expr
+{
+    /** @var Node\Expr */
+    public $cond;
+    /** @var MatchArm[] */
+    public $arms;
+
+    /**
+     * @param MatchArm[] $arms
+     */
+    public function __construct(Node\Expr $cond, array $arms = [], array $attributes = []) {
+        $this->attributes = $attributes;
+        $this->cond = $cond;
+        $this->arms = $arms;
+    }
+
+    public function getSubNodeNames() : array {
+        return ['cond', 'arms'];
+    }
+
+    public function getType() : string {
+        return 'Expr_Match';
+    }
+}

lib/PhpParser/Node/FunctionLike.php

@@ -23,7 +23,7 @@ interface FunctionLike extends Node
     /**
      * Get the declared return type or null
      *
-     * @return null|Identifier|Node\Name|Node\NullableType
+     * @return null|Identifier|Node\Name|Node\NullableType|Node\UnionType
      */
     public function getReturnType();
 

lib/PhpParser/Node/MatchArm.php

@@ -0,0 +1,31 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\Node;
+
+use PhpParser\Node;
+use PhpParser\NodeAbstract;
+
+class MatchArm extends NodeAbstract
+{
+    /** @var null|Node\Expr[] */
+    public $conds;
+    /** @var Node\Expr */
+    public $body;
+
+    /**
+     * @param null|Node\Expr[] $conds
+     */
+    public function __construct($conds, Node\Expr $body, array $attributes = []) {
+        $this->conds = $conds;
+        $this->body = $body;
+        $this->attributes = $attributes;
+    }
+
+    public function getSubNodeNames() : array {
+        return ['conds', 'body'];
+    }
+
+    public function getType() : string {
+        return 'MatchArm';
+    }
+}

lib/PhpParser/Node/Param.php

@@ -6,7 +6,7 @@ use PhpParser\NodeAbstract;
 
 class Param extends NodeAbstract
 {
-    /** @var null|Identifier|Name|NullableType Type declaration */
+    /** @var null|Identifier|Name|NullableType|UnionType Type declaration */
     public $type;
     /** @var bool Whether parameter is passed by reference */
     public $byRef;
@@ -16,20 +16,25 @@ class Param extends NodeAbstract
     public $var;
     /** @var null|Expr Default value */
     public $default;
+    /** @var int */
+    public $flags;
 
     /**
      * Constructs a parameter node.
      *
      * @param Expr\Variable|Expr\Error                           $var        Parameter variable
      * @param null|Expr                                          $default    Default value
-     * @param null|string|Identifier|Name|NullableType $type       Type declaration
+     * @param null|string|Identifier|Name|NullableType|UnionType $type       Type declaration
      * @param bool                                               $byRef      Whether is passed by reference
      * @param bool                                               $variadic   Whether this is a variadic argument
+     * @param array                                              $flags      Optional visibility flags
      * @param array                                              $attributes Additional attributes
      */
     public function __construct(
         $var, Expr $default = null, $type = null,
-        bool $byRef = false, bool $variadic = false, array $attributes = []
+        bool $byRef = false, bool $variadic = false,
+        array $attributes = [],
+        int $flags = 0
     ) {
         $this->attributes = $attributes;
         $this->type = \is_string($type) ? new Identifier($type) : $type;
@@ -37,10 +42,11 @@ class Param extends NodeAbstract
         $this->variadic = $variadic;
         $this->var = $var;
         $this->default = $default;
+        $this->flags = $flags;
     }
 
     public function getSubNodeNames() : array {
-        return ['type', 'byRef', 'variadic', 'var', 'default'];
+        return ['flags', 'type', 'byRef', 'variadic', 'var', 'default'];
     }
 
     public function getType() : string {

lib/PhpParser/Node/Scalar/DNumber.php

@@ -34,6 +34,8 @@ class DNumber extends Scalar
      * @return float The parsed number
      */
     public static function parse(string $str) : float {
+        $str = str_replace('_', '', $str);
+
         // if string contains any of .eE just cast it to float
         if (false !== strpbrk($str, '.eE')) {
             return (float) $str;

lib/PhpParser/Node/Scalar/LNumber.php

@@ -41,6 +41,8 @@ class LNumber extends Scalar
      * @return LNumber The constructed LNumber, including kind attribute
      */
     public static function fromString(string $str, array $attributes = [], bool $allowInvalidOctal = false) : LNumber {
+        $str = str_replace('_', '', $str);
+
         if ('0' !== $str[0] || '0' === $str) {
             $attributes['kind'] = LNumber::KIND_DEC;
             return new LNumber((int) $str, $attributes);

lib/PhpParser/Node/Scalar/String_.php

@@ -100,7 +100,7 @@ class String_ extends Scalar
                 if (isset(self::$replacements[$str])) {
                     return self::$replacements[$str];
                 } elseif ('x' === $str[0] || 'X' === $str[0]) {
-                    return chr(hexdec($str));
+                    return chr(hexdec(substr($str, 1)));
                 } elseif ('u' === $str[0]) {
                     return self::codePointToUtf8(hexdec($matches[2]));
                 } else {

lib/PhpParser/Node/Stmt/Catch_.php

@@ -9,7 +9,7 @@ class Catch_ extends Node\Stmt
 {
     /** @var Node\Name[] Types of exceptions to catch */
     public $types;
-    /** @var Expr\Variable Variable for exception */
+    /** @var Expr\Variable|null Variable for exception */
     public $var;
     /** @var Node\Stmt[] Statements */
     public $stmts;
@@ -18,12 +18,12 @@ class Catch_ extends Node\Stmt
      * Constructs a catch node.
      *
      * @param Node\Name[]           $types      Types of exceptions to catch
-     * @param Expr\Variable $var        Variable for exception
+     * @param Expr\Variable|null    $var        Variable for exception
      * @param Node\Stmt[]           $stmts      Statements
      * @param array                 $attributes Additional attributes
      */
     public function __construct(
-        array $types, Expr\Variable $var, array $stmts = [], array $attributes = []
+        array $types, Expr\Variable $var = null, array $stmts = [], array $attributes = []
     ) {
         $this->attributes = $attributes;
         $this->types = $types;

lib/PhpParser/Node/Stmt/ClassLike.php

@@ -14,6 +14,65 @@ abstract class ClassLike extends Node\Stmt
     /** @var Node\Stmt[] Statements */
     public $stmts;
 
+    /**
+     * @return TraitUse[]
+     */
+    public function getTraitUses() : array {
+        $traitUses = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof TraitUse) {
+                $traitUses[] = $stmt;
+            }
+        }
+        return $traitUses;
+    }
+
+    /**
+     * @return ClassConst[]
+     */
+    public function getConstants() : array {
+        $constants = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof ClassConst) {
+                $constants[] = $stmt;
+            }
+        }
+        return $constants;
+    }
+
+    /**
+     * @return Property[]
+     */
+    public function getProperties() : array {
+        $properties = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof Property) {
+                $properties[] = $stmt;
+            }
+        }
+        return $properties;
+    }
+
+    /**
+     * Gets property with the given name defined directly in this class/interface/trait.
+     *
+     * @param string $name Name of the property
+     *
+     * @return Property|null Property node or null if the property does not exist
+     */
+    public function getProperty(string $name) {
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof Property) {
+                foreach ($stmt->props as $prop) {
+                    if ($prop instanceof PropertyProperty && $name === $prop->name->toString()) {
+                        return $stmt;
+                    }
+                }
+            }
+        }
+        return null;
+    }
+
     /**
      * Gets all methods defined directly in this class/interface/trait
      *

lib/PhpParser/Node/Stmt/ClassMethod.php

@@ -15,7 +15,7 @@ class ClassMethod extends Node\Stmt implements FunctionLike
     public $name;
     /** @var Node\Param[] Parameters */
     public $params;
-    /** @var null|Node\Identifier|Node\Name|Node\NullableType Return type */
+    /** @var null|Node\Identifier|Node\Name|Node\NullableType|Node\UnionType Return type */
     public $returnType;
     /** @var Node\Stmt[]|null Statements */
     public $stmts;

lib/PhpParser/Node/Stmt/Function_.php

@@ -16,7 +16,7 @@ class Function_ extends Node\Stmt implements FunctionLike
     public $name;
     /** @var Node\Param[] Parameters */
     public $params;
-    /** @var null|Node\Identifier|Node\Name|Node\NullableType Return type */
+    /** @var null|Node\Identifier|Node\Name|Node\NullableType|Node\UnionType Return type */
     public $returnType;
     /** @var Node\Stmt[] Statements */
     public $stmts;

lib/PhpParser/Node/Stmt/Property.php

@@ -6,6 +6,7 @@ use PhpParser\Node;
 use PhpParser\Node\Identifier;
 use PhpParser\Node\Name;
 use PhpParser\Node\NullableType;
+use PhpParser\Node\UnionType;
 
 class Property extends Node\Stmt
 {
@@ -13,7 +14,7 @@ class Property extends Node\Stmt
     public $flags;
     /** @var PropertyProperty[] Properties */
     public $props;
-    /** @var null|Identifier|Name|NullableType Type declaration */
+    /** @var null|Identifier|Name|NullableType|UnionType Type declaration */
     public $type;
 
     /**
@@ -22,7 +23,7 @@ class Property extends Node\Stmt
      * @param int                                                $flags      Modifiers
      * @param PropertyProperty[]                                 $props      Properties
      * @param array                                              $attributes Additional attributes
-     * @param null|string|Identifier|Name|NullableType $type       Type declaration
+     * @param null|string|Identifier|Name|NullableType|UnionType $type       Type declaration
      */
     public function __construct(int $flags, array $props, array $attributes = [], $type = null) {
         $this->attributes = $attributes;

lib/PhpParser/Node/Stmt/TryCatch.php

@@ -18,7 +18,7 @@ class TryCatch extends Node\Stmt
      *
      * @param Node\Stmt[]   $stmts      Statements
      * @param Catch_[]      $catches    Catches
-     * @param null|Finally_ $finally    Optionaly finally node
+     * @param null|Finally_ $finally    Optional finally node
      * @param array         $attributes Additional attributes
      */
     public function __construct(array $stmts, array $catches, Finally_ $finally = null, array $attributes = []) {

lib/PhpParser/Node/UnionType.php

@@ -0,0 +1,30 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\Node;
+
+use PhpParser\NodeAbstract;
+
+class UnionType extends NodeAbstract
+{
+    /** @var (Identifier|Name)[] Types */
+    public $types;
+
+    /**
+     * Constructs a union type.
+     *
+     * @param (Identifier|Name)[] $types      Types
+     * @param array               $attributes Additional attributes
+     */
+    public function __construct(array $types, array $attributes = []) {
+        $this->attributes = $attributes;
+        $this->types = $types;
+    }
+
+    public function getSubNodeNames() : array {
+        return ['types'];
+    }
+    
+    public function getType() : string {
+        return 'UnionType';
+    }
+}

lib/PhpParser/NodeAbstract.php

@@ -108,24 +108,20 @@ abstract class NodeAbstract implements Node, \JsonSerializable
     /**
      * Gets the doc comment of the node.
      *
-     * The doc comment has to be the last comment associated with the node.
-     *
      * @return null|Comment\Doc Doc comment object or null
      */
     public function getDocComment() {
         $comments = $this->getComments();
-        if (!$comments) {
+        for ($i = count($comments) - 1; $i >= 0; $i--) {
-            return null;
+            $comment = $comments[$i];
+            if ($comment instanceof Comment\Doc) {
+                return $comment;
+            }
         }
 
-        $lastComment = $comments[count($comments) - 1];
-        if (!$lastComment instanceof Comment\Doc) {
         return null;
     }
 
-        return $lastComment;
-    }
-
     /**
      * Sets the doc comment of the node.
      *
@@ -135,16 +131,17 @@ abstract class NodeAbstract implements Node, \JsonSerializable
      */
     public function setDocComment(Comment\Doc $docComment) {
         $comments = $this->getComments();
-
+        for ($i = count($comments) - 1; $i >= 0; $i--) {
-        $numComments = count($comments);
+            if ($comments[$i] instanceof Comment\Doc) {
-        if ($numComments > 0 && $comments[$numComments - 1] instanceof Comment\Doc) {
+                // Replace existing doc comment.
-            // Replace existing doc comment
+                $comments[$i] = $docComment;
-            $comments[$numComments - 1] = $docComment;
+                $this->setAttribute('comments', $comments);
-        } else {
+                return;
-            // Append new comment
+            }
-            $comments[] = $docComment;
         }
 
+        // Append new doc comment.
+        $comments[] = $docComment;
         $this->setAttribute('comments', $comments);
     }
 

lib/PhpParser/NodeVisitor/NameResolver.php

@@ -91,6 +91,7 @@ class NameResolver extends NodeVisitorAbstract
             $this->resolveSignature($node);
         } elseif ($node instanceof Stmt\ClassMethod
                   || $node instanceof Expr\Closure
+                  || $node instanceof Expr\ArrowFunction
         ) {
             $this->resolveSignature($node);
         } elseif ($node instanceof Stmt\Property) {
@@ -161,12 +162,18 @@ class NameResolver extends NodeVisitorAbstract
     }
 
     private function resolveType($node) {
+        if ($node instanceof Name) {
+            return $this->resolveClassName($node);
+        }
         if ($node instanceof Node\NullableType) {
             $node->type = $this->resolveType($node->type);
             return $node;
         }
-        if ($node instanceof Name) {
+        if ($node instanceof Node\UnionType) {
-            return $this->resolveClassName($node);
+            foreach ($node->types as &$type) {
+                $type = $this->resolveType($type);
+            }
+            return $node;
         }
         return $node;
     }

lib/PhpParser/NodeVisitor/NodeConnectingVisitor.php

@@ -0,0 +1,52 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\NodeVisitor;
+
+use PhpParser\Node;
+use PhpParser\NodeVisitorAbstract;
+
+/**
+ * Visitor that connects a child node to its parent node
+ * as well as its sibling nodes.
+ *
+ * On the child node, the parent node can be accessed through
+ * <code>$node->getAttribute('parent')</code>, the previous
+ * node can be accessed through <code>$node->getAttribute('previous')</code>,
+ * and the next node can be accessed through <code>$node->getAttribute('next')</code>.
+ */
+final class NodeConnectingVisitor extends NodeVisitorAbstract
+{
+    /**
+     * @var Node[]
+     */
+    private $stack = [];
+
+    /**
+     * @var ?Node
+     */
+    private $previous;
+
+    public function beforeTraverse(array $nodes) {
+        $this->stack    = [];
+        $this->previous = null;
+    }
+
+    public function enterNode(Node $node) {
+        if (!empty($this->stack)) {
+            $node->setAttribute('parent', $this->stack[count($this->stack) - 1]);
+        }
+
+        if ($this->previous !== null && $this->previous->getAttribute('parent') === $node->getAttribute('parent')) {
+            $node->setAttribute('previous', $this->previous);
+            $this->previous->setAttribute('next', $node);
+        }
+
+        $this->stack[] = $node;
+    }
+
+    public function leaveNode(Node $node) {
+        $this->previous = $node;
+
+        array_pop($this->stack);
+    }
+}

lib/PhpParser/NodeVisitor/ParentConnectingVisitor.php

@@ -0,0 +1,41 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\NodeVisitor;
+
+use function array_pop;
+use function count;
+use PhpParser\Node;
+use PhpParser\NodeVisitorAbstract;
+
+/**
+ * Visitor that connects a child node to its parent node.
+ *
+ * On the child node, the parent node can be accessed through
+ * <code>$node->getAttribute('parent')</code>.
+ */
+final class ParentConnectingVisitor extends NodeVisitorAbstract
+{
+    /**
+     * @var Node[]
+     */
+    private $stack = [];
+
+    public function beforeTraverse(array $nodes)
+    {
+        $this->stack = [];
+    }
+
+    public function enterNode(Node $node)
+    {
+        if (!empty($this->stack)) {
+            $node->setAttribute('parent', $this->stack[count($this->stack) - 1]);
+        }
+
+        $this->stack[] = $node;
+    }
+
+    public function leaveNode(Node $node)
+    {
+        array_pop($this->stack);
+    }
+}

lib/PhpParser/Parser/Tokens.php

@@ -84,63 +84,67 @@ final class Tokens
     const T_ENDDECLARE = 332;
     const T_AS = 333;
     const T_SWITCH = 334;
-    const T_ENDSWITCH = 335;
+    const T_MATCH = 335;
-    const T_CASE = 336;
+    const T_ENDSWITCH = 336;
-    const T_DEFAULT = 337;
+    const T_CASE = 337;
-    const T_BREAK = 338;
+    const T_DEFAULT = 338;
-    const T_CONTINUE = 339;
+    const T_BREAK = 339;
-    const T_GOTO = 340;
+    const T_CONTINUE = 340;
-    const T_FUNCTION = 341;
+    const T_GOTO = 341;
-    const T_FN = 342;
+    const T_FUNCTION = 342;
-    const T_CONST = 343;
+    const T_FN = 343;
-    const T_RETURN = 344;
+    const T_CONST = 344;
-    const T_TRY = 345;
+    const T_RETURN = 345;
-    const T_CATCH = 346;
+    const T_TRY = 346;
-    const T_FINALLY = 347;
+    const T_CATCH = 347;
-    const T_THROW = 348;
+    const T_FINALLY = 348;
-    const T_USE = 349;
+    const T_THROW = 349;
-    const T_INSTEADOF = 350;
+    const T_USE = 350;
-    const T_GLOBAL = 351;
+    const T_INSTEADOF = 351;
-    const T_STATIC = 352;
+    const T_GLOBAL = 352;
-    const T_ABSTRACT = 353;
+    const T_STATIC = 353;
-    const T_FINAL = 354;
+    const T_ABSTRACT = 354;
-    const T_PRIVATE = 355;
+    const T_FINAL = 355;
-    const T_PROTECTED = 356;
+    const T_PRIVATE = 356;
-    const T_PUBLIC = 357;
+    const T_PROTECTED = 357;
-    const T_VAR = 358;
+    const T_PUBLIC = 358;
-    const T_UNSET = 359;
+    const T_VAR = 359;
-    const T_ISSET = 360;
+    const T_UNSET = 360;
-    const T_EMPTY = 361;
+    const T_ISSET = 361;
-    const T_HALT_COMPILER = 362;
+    const T_EMPTY = 362;
-    const T_CLASS = 363;
+    const T_HALT_COMPILER = 363;
-    const T_TRAIT = 364;
+    const T_CLASS = 364;
-    const T_INTERFACE = 365;
+    const T_TRAIT = 365;
-    const T_EXTENDS = 366;
+    const T_INTERFACE = 366;
-    const T_IMPLEMENTS = 367;
+    const T_EXTENDS = 367;
-    const T_OBJECT_OPERATOR = 368;
+    const T_IMPLEMENTS = 368;
-    const T_LIST = 369;
+    const T_OBJECT_OPERATOR = 369;
-    const T_ARRAY = 370;
+    const T_LIST = 370;
-    const T_CALLABLE = 371;
+    const T_ARRAY = 371;
-    const T_CLASS_C = 372;
+    const T_CALLABLE = 372;
-    const T_TRAIT_C = 373;
+    const T_CLASS_C = 373;
-    const T_METHOD_C = 374;
+    const T_TRAIT_C = 374;
-    const T_FUNC_C = 375;
+    const T_METHOD_C = 375;
-    const T_LINE = 376;
+    const T_FUNC_C = 376;
-    const T_FILE = 377;
+    const T_LINE = 377;
-    const T_COMMENT = 378;
+    const T_FILE = 378;
-    const T_DOC_COMMENT = 379;
+    const T_COMMENT = 379;
-    const T_OPEN_TAG = 380;
+    const T_DOC_COMMENT = 380;
-    const T_OPEN_TAG_WITH_ECHO = 381;
+    const T_OPEN_TAG = 381;
-    const T_CLOSE_TAG = 382;
+    const T_OPEN_TAG_WITH_ECHO = 382;
-    const T_WHITESPACE = 383;
+    const T_CLOSE_TAG = 383;
-    const T_START_HEREDOC = 384;
+    const T_WHITESPACE = 384;
-    const T_END_HEREDOC = 385;
+    const T_START_HEREDOC = 385;
-    const T_DOLLAR_OPEN_CURLY_BRACES = 386;
+    const T_END_HEREDOC = 386;
-    const T_CURLY_OPEN = 387;
+    const T_DOLLAR_OPEN_CURLY_BRACES = 387;
-    const T_PAAMAYIM_NEKUDOTAYIM = 388;
+    const T_CURLY_OPEN = 388;
-    const T_NAMESPACE = 389;
+    const T_PAAMAYIM_NEKUDOTAYIM = 389;
-    const T_NS_C = 390;
+    const T_NAMESPACE = 390;
-    const T_DIR = 391;
+    const T_NS_C = 391;
-    const T_NS_SEPARATOR = 392;
+    const T_DIR = 392;
-    const T_ELLIPSIS = 393;
+    const T_NS_SEPARATOR = 393;
+    const T_ELLIPSIS = 394;
+    const T_NAME_FULLY_QUALIFIED = 395;
+    const T_NAME_QUALIFIED = 396;
+    const T_NAME_RELATIVE = 397;
 }

lib/PhpParser/ParserAbstract.php

@@ -648,7 +648,7 @@ abstract class ParserAbstract implements Parser
     }
 
     protected function handleBuiltinTypes(Name $name) {
-        $scalarTypes = [
+        $builtinTypes = [
             'bool'     => true,
             'int'      => true,
             'float'    => true,
@@ -656,6 +656,9 @@ abstract class ParserAbstract implements Parser
             'iterable' => true,
             'void'     => true,
             'object'   => true,
+            'null'     => true,
+            'false'    => true,
+            'mixed'    => true,
         ];
 
         if (!$name->isUnqualified()) {
@@ -663,7 +666,7 @@ abstract class ParserAbstract implements Parser
         }
 
         $lowerName = $name->toLowerString();
-        if (!isset($scalarTypes[$lowerName])) {
+        if (!isset($builtinTypes[$lowerName])) {
             return $name;
         }
 
@@ -840,21 +843,29 @@ abstract class ParserAbstract implements Parser
     }
 
     /**
-     * Create attributes for a zero-length node with the given start attributes.
+     * Create attributes for a zero-length common-capturing nop.
      *
-     * @param array $startAttributes
+     * @param Comment[] $comments
      * @return array
      */
-    protected function createZeroLengthAttributes(array $startAttributes) {
+    protected function createCommentNopAttributes(array $comments) {
-        $attributes = $startAttributes;
+        $comment = $comments[count($comments) - 1];
-        if (isset($startAttributes['startLine'])) {
+        $commentEndLine = $comment->getEndLine();
-            $attributes['endLine'] = $startAttributes['startLine'];
+        $commentEndFilePos = $comment->getEndFilePos();
+        $commentEndTokenPos = $comment->getEndTokenPos();
+
+        $attributes = ['comments' => $comments];
+        if (-1 !== $commentEndLine) {
+            $attributes['startLine'] = $commentEndLine;
+            $attributes['endLine'] = $commentEndLine;
         }
-        if (isset($startAttributes['startTokenPos'])) {
+        if (-1 !== $commentEndFilePos) {
-            $attributes['endTokenPos'] = $startAttributes['startTokenPos'] - 1;
+            $attributes['startFilePos'] = $commentEndFilePos + 1;
+            $attributes['endFilePos'] = $commentEndFilePos;
         }
-        if (isset($startAttributes['startFilePos'])) {
+        if (-1 !== $commentEndTokenPos) {
-            $attributes['endFilePos'] = $startAttributes['startFilePos'] - 1;
+            $attributes['startTokenPos'] = $commentEndTokenPos + 1;
+            $attributes['endTokenPos'] = $commentEndTokenPos;
         }
         return $attributes;
     }

lib/PhpParser/PrettyPrinter/Standard.php

@@ -18,7 +18,8 @@ class Standard extends PrettyPrinterAbstract
     // Special nodes
 
     protected function pParam(Node\Param $node) {
-        return ($node->type ? $this->p($node->type) . ' ' : '')
+        return ($this->pModifiers($node->flags))
+             . ($node->type ? $this->p($node->type) . ' ' : '')
              . ($node->byRef ? '&' : '')
              . ($node->variadic ? '...' : '')
              . $this->p($node->var)
@@ -37,6 +38,10 @@ class Standard extends PrettyPrinterAbstract
         return '?' . $this->p($node->type);
     }
 
+    protected function pUnionType(Node\UnionType $node) {
+        return $this->pImplode($node->types, '|');
+    }
+
     protected function pIdentifier(Node\Identifier $node) {
         return $node->name;
     }
@@ -159,8 +164,13 @@ class Standard extends PrettyPrinterAbstract
             return (string) $node->value;
         }
 
-        $sign = $node->value < 0 ? '-' : '';
+        if ($node->value < 0) {
+            $sign = '-';
+            $str = (string) -$node->value;
+        } else {
+            $sign = '';
             $str = (string) $node->value;
+        }
         switch ($kind) {
             case Scalar\LNumber::KIND_BIN:
                 return $sign . '0b' . base_convert($str, 10, 2);
@@ -584,6 +594,18 @@ class Standard extends PrettyPrinterAbstract
              . ' {' . $this->pStmts($node->stmts) . $this->nl . '}';
     }
 
+    protected function pExpr_Match(Expr\Match_ $node) {
+        return 'match (' . $this->p($node->cond) . ') {'
+            . $this->pCommaSeparatedMultiline($node->arms, true)
+            . $this->nl
+            . '}';
+    }
+
+    protected function pMatchArm(Node\MatchArm $node) {
+        return ($node->conds ? $this->pCommaSeparated($node->conds) : 'default')
+            . ' => ' . $this->p($node->body);
+    }
+
     protected function pExpr_ArrowFunction(Expr\ArrowFunction $node) {
         return ($node->static ? 'static ' : '')
             . 'fn' . ($node->byRef ? '&' : '')
@@ -716,7 +738,7 @@ class Standard extends PrettyPrinterAbstract
     protected function pStmt_ClassMethod(Stmt\ClassMethod $node) {
         return $this->pModifiers($node->flags)
              . 'function ' . ($node->byRef ? '&' : '') . $node->name
-             . '(' . $this->pCommaSeparated($node->params) . ')'
+             . '(' . $this->pMaybeMultiline($node->params) . ')'
              . (null !== $node->returnType ? ' : ' . $this->p($node->returnType) : '')
              . (null !== $node->stmts
                 ? $this->nl . '{' . $this->pStmts($node->stmts) . $this->nl . '}'
@@ -808,8 +830,8 @@ class Standard extends PrettyPrinterAbstract
     }
 
     protected function pStmt_Catch(Stmt\Catch_ $node) {
-        return 'catch (' . $this->pImplode($node->types, '|') . ' '
+        return 'catch (' . $this->pImplode($node->types, '|')
-             . $this->p($node->var)
+             . ($node->var !== null ? ' ' . $this->p($node->var) : '')
              . ') {' . $this->pStmts($node->stmts) . $this->nl . '}';
     }
 

lib/PhpParser/PrettyPrinterAbstract.php

@@ -756,20 +756,13 @@ abstract class PrettyPrinterAbstract
                 $itemEndPos = $origArrItem->getEndTokenPos();
                 \assert($itemStartPos >= 0 && $itemEndPos >= 0);
 
-                if ($itemEndPos < $itemStartPos) {
-                    // End can be before start for Nop nodes, because offsets refer to non-whitespace
-                    // locations, which for an "empty" node might result in an inverted order.
-                    assert($origArrItem instanceof Stmt\Nop);
-                    continue;
-                }
-
                 $origIndentLevel = $this->indentLevel;
                 $lastElemIndentLevel = $this->origTokens->getIndentationBefore($itemStartPos) + $indentAdjustment;
                 $this->setIndentLevel($lastElemIndentLevel);
 
                 $comments = $arrItem->getComments();
                 $origComments = $origArrItem->getComments();
-                $commentStartPos = $origComments ? $origComments[0]->getTokenPos() : $itemStartPos;
+                $commentStartPos = $origComments ? $origComments[0]->getStartTokenPos() : $itemStartPos;
                 \assert($commentStartPos >= 0);
 
                 $commentsChanged = $comments !== $origComments;
@@ -1227,6 +1220,7 @@ abstract class PrettyPrinterAbstract
             'Param->type' => $stripRight,
             'Param->default' => $stripEquals,
             'Stmt_Break->num' => $stripBoth,
+            'Stmt_Catch->var' => $stripLeft,
             'Stmt_ClassMethod->returnType' => $stripColon,
             'Stmt_Class->extends' => ['left' => \T_EXTENDS],
             'Expr_PrintableNewAnonClass->extends' => ['left' => \T_EXTENDS],
@@ -1264,6 +1258,7 @@ abstract class PrettyPrinterAbstract
             'Param->type' => [null, false, null, ' '],
             'Param->default' => [null, false, ' = ', null],
             'Stmt_Break->num' => [\T_BREAK, false, ' ', null],
+            'Stmt_Catch->var' => [null, false, ' ', null],
             'Stmt_ClassMethod->returnType' => [')', false, ' : ', null],
             'Stmt_Class->extends' => [null, false, ' extends ', null],
             'Expr_PrintableNewAnonClass->extends' => [null, ' extends ', null],
@@ -1295,6 +1290,7 @@ abstract class PrettyPrinterAbstract
             //'Expr_ShellExec->parts' => '', // TODO These need to be treated more carefully
             //'Scalar_Encapsed->parts' => '',
             'Stmt_Catch->types' => '|',
+            'UnionType->types' => '|',
             'Stmt_If->elseifs' => ' ',
             'Stmt_TryCatch->catches' => ' ',
 
@@ -1324,12 +1320,14 @@ abstract class PrettyPrinterAbstract
             'Stmt_Global->vars' => ', ',
             'Stmt_GroupUse->uses' => ', ',
             'Stmt_Interface->extends' => ', ',
+            'Stmt_Match->arms' => ', ',
             'Stmt_Property->props' => ', ',
             'Stmt_StaticVar->vars' => ', ',
             'Stmt_TraitUse->traits' => ', ',
             'Stmt_TraitUseAdaptation_Precedence->insteadof' => ', ',
             'Stmt_Unset->vars' => ', ',
             'Stmt_Use->uses' => ', ',
+            'MatchArm->conds' => ', ',
 
             // statement lists
             'Expr_Closure->stmts' => "\n",
@@ -1396,6 +1394,7 @@ abstract class PrettyPrinterAbstract
              * Stmt_TraitUseAdaptation_Precedence->insteadof
              * Stmt_Unset->vars
              * Stmt_Use->uses
+             * UnionType->types
              */
 
             /* TODO
@@ -1418,6 +1417,7 @@ abstract class PrettyPrinterAbstract
             'Stmt_ClassMethod->flags' => \T_FUNCTION,
             'Stmt_Class->flags' => \T_CLASS,
             'Stmt_Property->flags' => \T_VARIABLE,
+            'Param->flags' => \T_VARIABLE,
             //'Stmt_TraitUseAdaptation_Alias->newModifier' => 0, // TODO
         ];
 

test/PhpParser/Builder/InterfaceTest.php

@@ -9,11 +9,8 @@ use PhpParser\Node\Stmt;
 
 class InterfaceTest extends \PHPUnit\Framework\TestCase
 {
-    /** @var Interface_ */
+    protected function createInterfaceBuilder() {
-    protected $builder;
+        return new Interface_('Contract');
-
-    protected function setUp() {
-        $this->builder = new Interface_('Contract');
     }
 
     private function dump($node) {
@@ -22,13 +19,14 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     }
 
     public function testEmpty() {
-        $contract = $this->builder->getNode();
+        $contract = $this->createInterfaceBuilder()->getNode();
         $this->assertInstanceOf(Stmt\Interface_::class, $contract);
         $this->assertEquals(new Node\Identifier('Contract'), $contract->name);
     }
 
     public function testExtending() {
-        $contract = $this->builder->extend('Space\Root1', 'Root2')->getNode();
+        $contract = $this->createInterfaceBuilder()
+            ->extend('Space\Root1', 'Root2')->getNode();
         $this->assertEquals(
             new Stmt\Interface_('Contract', [
                 'extends' => [
@@ -41,7 +39,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
 
     public function testAddMethod() {
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder->addStmt($method)->getNode();
+        $contract = $this->createInterfaceBuilder()->addStmt($method)->getNode();
         $this->assertSame([$method], $contract->stmts);
     }
 
@@ -49,7 +47,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
         $const = new Stmt\ClassConst([
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458.0))
         ]);
-        $contract = $this->builder->addStmt($const)->getNode();
+        $contract = $this->createInterfaceBuilder()->addStmt($const)->getNode();
         $this->assertSame(299792458.0, $contract->stmts[0]->consts[0]->value->value);
     }
 
@@ -58,7 +56,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458))
         ]);
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder
+        $contract = $this->createInterfaceBuilder()
             ->addStmt($method)
             ->addStmt($const)
             ->getNode()
@@ -69,7 +67,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     }
 
     public function testDocComment() {
-        $node = $this->builder
+        $node = $this->createInterfaceBuilder()
             ->setDocComment('/** Test */')
             ->getNode();
 
@@ -81,7 +79,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     public function testInvalidStmtError() {
         $this->expectException(\LogicException::class);
         $this->expectExceptionMessage('Unexpected node of type "Stmt_PropertyProperty"');
-        $this->builder->addStmt(new Stmt\PropertyProperty('invalid'));
+        $this->createInterfaceBuilder()->addStmt(new Stmt\PropertyProperty('invalid'));
     }
 
     public function testFullFunctional() {
@@ -89,7 +87,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458))
         ]);
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder
+        $contract = $this->createInterfaceBuilder()
             ->addStmt($method)
             ->addStmt($const)
             ->getNode()

test/PhpParser/Builder/ParamTest.php

@@ -80,6 +80,8 @@ class ParamTest extends \PHPUnit\Framework\TestCase
 
     /**
      * @dataProvider provideTestTypes
+     * @dataProvider provideTestNullableTypes
+     * @dataProvider provideTestUnionTypes
      */
     public function testTypes($typeHint, $expectedType) {
         $node = $this->createParamBuilder('test')
@@ -111,12 +113,18 @@ class ParamTest extends \PHPUnit\Framework\TestCase
             ['object', new Node\Identifier('object')],
             ['Array', new Node\Identifier('array')],
             ['CALLABLE', new Node\Identifier('callable')],
+            ['mixed', new Node\Identifier('mixed')],
             ['Some\Class', new Node\Name('Some\Class')],
             ['\Foo', new Node\Name\FullyQualified('Foo')],
             ['self', new Node\Name('self')],
+            [new Node\Name('Some\Class'), new Node\Name('Some\Class')],
+        ];
+    }
+
+    public function provideTestNullableTypes() {
+        return [
             ['?array', new Node\NullableType(new Node\Identifier('array'))],
             ['?Some\Class', new Node\NullableType(new Node\Name('Some\Class'))],
-            [new Node\Name('Some\Class'), new Node\Name('Some\Class')],
             [
                 new Node\NullableType(new Node\Identifier('int')),
                 new Node\NullableType(new Node\Identifier('int'))
@@ -128,6 +136,33 @@ class ParamTest extends \PHPUnit\Framework\TestCase
         ];
     }
 
+    public function provideTestUnionTypes() {
+        return [
+            [
+                new Node\UnionType([
+                    new Node\Name('Some\Class'),
+                    new Node\Identifier('array'),
+                ]),
+                new Node\UnionType([
+                    new Node\Name('Some\Class'),
+                    new Node\Identifier('array'),
+                ]),
+            ],
+            [
+                new Node\UnionType([
+                    new Node\Identifier('self'),
+                    new Node\Identifier('array'),
+                    new Node\Name\FullyQualified('Foo')
+                ]),
+                new Node\UnionType([
+                    new Node\Identifier('self'),
+                    new Node\Identifier('array'),
+                    new Node\Name\FullyQualified('Foo')
+                ]),
+            ],
+        ];
+    }
+
     public function testVoidTypeError() {
         $this->expectException(\LogicException::class);
         $this->expectExceptionMessage('Parameter type cannot be void');
@@ -136,7 +171,7 @@ class ParamTest extends \PHPUnit\Framework\TestCase
 
     public function testInvalidTypeError() {
         $this->expectException(\LogicException::class);
-        $this->expectExceptionMessage('Type must be a string, or an instance of Name, Identifier or NullableType');
+        $this->expectExceptionMessage('Type must be a string, or an instance of Name, Identifier, NullableType or UnionType');
         $this->createParamBuilder('test')->setType(new \stdClass);
     }
 

test/PhpParser/Builder/TraitTest.php

@@ -5,6 +5,12 @@ namespace PhpParser\Builder;
 use PhpParser\Comment;
 use PhpParser\Node\Name;
 use PhpParser\Node\Stmt;
+use PhpParser\Node\Stmt\Class_;
+use PhpParser\Node\Stmt\ClassConst;
+use PhpParser\Node\Stmt\ClassMethod;
+use PhpParser\Node\Stmt\Property;
+use PhpParser\Node\Stmt\PropertyProperty;
+use PhpParser\Node\Stmt\TraitUse;
 
 class TraitTest extends \PHPUnit\Framework\TestCase
 {
@@ -43,4 +49,43 @@ class TraitTest extends \PHPUnit\Framework\TestCase
             ->addStmt(new Stmt\Echo_([]))
         ;
     }
+
+    public function testGetMethods() {
+        $methods = [
+            new ClassMethod('foo'),
+            new ClassMethod('bar'),
+            new ClassMethod('fooBar'),
+        ];
+        $trait = new Stmt\Trait_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $methods[0],
+                new ClassConst([]),
+                $methods[1],
+                new Property(0, []),
+                $methods[2],
+            ]
+        ]);
+
+        $this->assertSame($methods, $trait->getMethods());
+    }
+
+    public function testGetProperties()
+    {
+        $properties = [
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo')]),
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('bar')]),
+        ];
+        $trait = new Stmt\Trait_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $properties[0],
+                new ClassConst([]),
+                $properties[1],
+                new ClassMethod('fooBar'),
+            ]
+        ]);
+
+        $this->assertSame($properties, $trait->getProperties());
+    }
 }

test/PhpParser/CommentTest.php

@@ -4,14 +4,21 @@ namespace PhpParser;
 
 class CommentTest extends \PHPUnit\Framework\TestCase
 {
-    public function testGetSet() {
+    public function testGetters() {
-        $comment = new Comment('/* Some comment */', 1, 10, 2);
+        $comment = new Comment('/* Some comment */',
+            1, 10, 2, 1, 27, 2);
 
         $this->assertSame('/* Some comment */', $comment->getText());
         $this->assertSame('/* Some comment */', (string) $comment);
         $this->assertSame(1, $comment->getLine());
         $this->assertSame(10, $comment->getFilePos());
         $this->assertSame(2, $comment->getTokenPos());
+        $this->assertSame(1, $comment->getStartLine());
+        $this->assertSame(10, $comment->getStartFilePos());
+        $this->assertSame(2, $comment->getStartTokenPos());
+        $this->assertSame(1, $comment->getEndLine());
+        $this->assertSame(27, $comment->getEndFilePos());
+        $this->assertSame(2, $comment->getEndTokenPos());
     }
 
     /**

test/PhpParser/Lexer/EmulativeTest.php

@@ -123,10 +123,6 @@ class EmulativeTest extends LexerTest
 
     public function provideTestLexNewFeatures() {
         return [
-            // PHP 7.4
-            ['??=', [
-                [Tokens::T_COALESCE_EQUAL, '??='],
-            ]],
             ['yield from', [
                 [Tokens::T_YIELD_FROM, 'yield from'],
             ]],
@@ -169,7 +165,7 @@ class EmulativeTest extends LexerTest
                 [ord(';'), ';'],
             ]],
 
-            // Flexible heredoc/nowdoc
+            // PHP 7.3: Flexible heredoc/nowdoc
             ["<<<LABEL\nLABEL,", [
                 [Tokens::T_START_HEREDOC, "<<<LABEL\n"],
                 [Tokens::T_END_HEREDOC, "LABEL"],
@@ -205,6 +201,58 @@ class EmulativeTest extends LexerTest
                 [Tokens::T_END_HEREDOC, "    LABEL"],
                 [Tokens::T_STRING, "LABEL"],
             ]],
+
+            // PHP 7.4: Null coalesce equal
+            ['??=', [
+                [Tokens::T_COALESCE_EQUAL, '??='],
+            ]],
+
+            // PHP 7.4: Number literal separator
+            ['1_000', [
+                [Tokens::T_LNUMBER, '1_000'],
+            ]],
+            ['0xCAFE_F00D', [
+                [Tokens::T_LNUMBER, '0xCAFE_F00D'],
+            ]],
+            ['0b0101_1111', [
+                [Tokens::T_LNUMBER, '0b0101_1111'],
+            ]],
+            ['0137_041', [
+                [Tokens::T_LNUMBER, '0137_041'],
+            ]],
+            ['1_000.0', [
+                [Tokens::T_DNUMBER, '1_000.0'],
+            ]],
+            ['1_0.0', [
+                [Tokens::T_DNUMBER, '1_0.0']
+            ]],
+            ['1_000_000_000.0', [
+                [Tokens::T_DNUMBER, '1_000_000_000.0']
+            ]],
+            ['0e1_0', [
+                [Tokens::T_DNUMBER, '0e1_0']
+            ]],
+            ['1_0e+10', [
+                [Tokens::T_DNUMBER, '1_0e+10']
+            ]],
+            ['1_0e-10', [
+                [Tokens::T_DNUMBER, '1_0e-10']
+            ]],
+            ['0b1011010101001010_110101010010_10101101010101_0101101011001_110111100', [
+                [Tokens::T_DNUMBER, '0b1011010101001010_110101010010_10101101010101_0101101011001_110111100'],
+            ]],
+            ['0xFFFF_FFFF_FFFF_FFFF', [
+                [Tokens::T_DNUMBER, '0xFFFF_FFFF_FFFF_FFFF'],
+            ]],
+            ['1_000+1', [
+                [Tokens::T_LNUMBER, '1_000'],
+                [ord('+'), '+'],
+                [Tokens::T_LNUMBER, '1'],
+            ]],
+            ['1_0abc', [
+                [Tokens::T_LNUMBER, '1_0'],
+                [Tokens::T_STRING, 'abc'],
+            ]],
         ];
     }
 }

test/PhpParser/LexerTest.php

@@ -35,6 +35,7 @@ class LexerTest extends \PHPUnit\Framework\TestCase
     public function provideTestError() {
         return [
             ["<?php /*", ["Unterminated comment from 1:7 to 1:9"]],
+            ["<?php /*\n", ["Unterminated comment from 1:7 to 2:1"]],
             ["<?php \1", ["Unexpected character \"\1\" (ASCII 1) from 1:7 to 1:7"]],
             ["<?php \0", ["Unexpected null byte from 1:7 to 1:7"]],
             // Error with potentially emulated token
@@ -103,7 +104,9 @@ class LexerTest extends \PHPUnit\Framework\TestCase
                         [
                             'startLine' => 3,
                             'comments' => [
-                                new Comment\Doc('/** doc' . "\n" . 'comment */', 2, 14, 5),
+                                new Comment\Doc('/** doc' . "\n" . 'comment */',
+                                    2, 14, 5,
+                                    3, 31, 5),
                             ]
                         ],
                         ['endLine' => 3]
@@ -120,10 +123,14 @@ class LexerTest extends \PHPUnit\Framework\TestCase
                         [
                             'startLine' => 2,
                             'comments' => [
-                                new Comment('/* comment */', 1, 6, 1),
+                                new Comment('/* comment */',
-                                new Comment('// comment' . "\n", 1, 20, 3),
+                                    1, 6, 1, 1, 18, 1),
-                                new Comment\Doc('/** docComment 1 */', 2, 31, 4),
+                                new Comment('// comment',
-                                new Comment\Doc('/** docComment 2 */', 2, 50, 5),
+                                    1, 20, 3, 1, 29, 3),
+                                new Comment\Doc('/** docComment 1 */',
+                                    2, 31, 5, 2, 49, 5),
+                                new Comment\Doc('/** docComment 2 */',
+                                    2, 50, 6, 2, 68, 6),
                             ],
                         ],
                         ['endLine' => 2]
@@ -179,11 +186,11 @@ class LexerTest extends \PHPUnit\Framework\TestCase
                     ],
                     [
                         Tokens::T_CONSTANT_ENCAPSED_STRING, '"b"',
-                        ['startTokenPos' => 5], ['endTokenPos' => 5]
+                        ['startTokenPos' => 6], ['endTokenPos' => 6]
                     ],
                     [
                         ord(';'), ';',
-                        ['startTokenPos' => 6], ['endTokenPos' => 6]
+                        ['startTokenPos' => 7], ['endTokenPos' => 7]
                     ],
                 ]
             ],
@@ -208,6 +215,18 @@ class LexerTest extends \PHPUnit\Framework\TestCase
                 [],
                 []
             ],
+            // tests PHP 8 T_NAME_* emulation
+            [
+                '<?php Foo\Bar \Foo\Bar namespace\Foo\Bar Foo\Bar\\',
+                ['usedAttributes' => []],
+                [
+                    [Tokens::T_NAME_QUALIFIED, 'Foo\Bar', [], []],
+                    [Tokens::T_NAME_FULLY_QUALIFIED, '\Foo\Bar', [], []],
+                    [Tokens::T_NAME_RELATIVE, 'namespace\Foo\Bar', [], []],
+                    [Tokens::T_NAME_QUALIFIED, 'Foo\Bar', [], []],
+                    [Tokens::T_NS_SEPARATOR, '\\', [], []],
+                ]
+            ],
         ];
     }
 
@@ -245,14 +264,17 @@ class LexerTest extends \PHPUnit\Framework\TestCase
     }
 
     public function testGetTokens() {
-        $code = '<?php "a";' . "\n" . '// foo' . "\n" . '"b";';
+        $code = '<?php "a";' . "\n" . '// foo' . "\n" . '// bar' . "\n\n" . '"b";';
         $expectedTokens = [
             [T_OPEN_TAG, '<?php ', 1],
             [T_CONSTANT_ENCAPSED_STRING, '"a"', 1],
             ';',
             [T_WHITESPACE, "\n", 1],
-            [T_COMMENT, '// foo' . "\n", 2],
+            [T_COMMENT, '// foo', 2],
-            [T_CONSTANT_ENCAPSED_STRING, '"b"', 3],
+            [T_WHITESPACE, "\n", 2],
+            [T_COMMENT, '// bar', 3],
+            [T_WHITESPACE, "\n\n", 3],
+            [T_CONSTANT_ENCAPSED_STRING, '"b"', 5],
             ';',
         ];
 

test/PhpParser/Node/Stmt/ClassTest.php

@@ -2,6 +2,8 @@
 
 namespace PhpParser\Node\Stmt;
 
+use PhpParser\Node\Scalar\String_;
+
 class ClassTest extends \PHPUnit\Framework\TestCase
 {
     public function testIsAbstract() {
@@ -20,6 +22,22 @@ class ClassTest extends \PHPUnit\Framework\TestCase
         $this->assertFalse($class->isFinal());
     }
 
+    public function testGetTraitUses() {
+        $traitUses = [
+            new TraitUse([new Trait_('foo')]),
+            new TraitUse([new Trait_('bar')]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                $traitUses[0],
+                new ClassMethod('fooBar'),
+                $traitUses[1],
+            ]
+        ]);
+
+        $this->assertSame($traitUses, $class->getTraitUses());
+    }
+
     public function testGetMethods() {
         $methods = [
             new ClassMethod('foo'),
@@ -40,6 +58,67 @@ class ClassTest extends \PHPUnit\Framework\TestCase
         $this->assertSame($methods, $class->getMethods());
     }
 
+    public function testGetConstants() {
+        $constants = [
+            new ClassConst([new \PhpParser\Node\Const_('foo', new String_('foo_value'))]),
+            new ClassConst([new \PhpParser\Node\Const_('bar', new String_('bar_value'))]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $constants[0],
+                new ClassMethod('fooBar'),
+                $constants[1],
+            ]
+        ]);
+
+        $this->assertSame($constants, $class->getConstants());
+    }
+
+    public function testGetProperties()
+    {
+        $properties = [
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo')]),
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('bar')]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $properties[0],
+                new ClassConst([]),
+                $properties[1],
+                new ClassMethod('fooBar'),
+            ]
+        ]);
+
+        $this->assertSame($properties, $class->getProperties());
+    }
+
+    public function testGetProperty() {
+        $properties = [
+            $fooProp = new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo1')]),
+            $barProp = new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('BAR1')]),
+            $fooBarProp = new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo2'), new PropertyProperty('bar2')]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $properties[0],
+                new ClassConst([]),
+                $properties[1],
+                new ClassMethod('fooBar'),
+                $properties[2],
+            ]
+        ]);
+
+        $this->assertSame($fooProp, $class->getProperty('foo1'));
+        $this->assertSame($barProp, $class->getProperty('BAR1'));
+        $this->assertSame($fooBarProp, $class->getProperty('foo2'));
+        $this->assertSame($fooBarProp, $class->getProperty('bar2'));
+        $this->assertNull($class->getProperty('bar1'));
+        $this->assertNull($class->getProperty('nonExisting'));
+    }
+
     public function testGetMethod() {
         $methodConstruct = new ClassMethod('__CONSTRUCT');
         $methodTest = new ClassMethod('test');

test/PhpParser/Node/Stmt/InterfaceTest.php

@@ -3,6 +3,7 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
+use PhpParser\Node\Scalar\String_;
 
 class InterfaceTest extends \PHPUnit\Framework\TestCase
 {
@@ -11,7 +12,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new ClassMethod('foo'),
             new ClassMethod('bar'),
         ];
-        $interface = new Class_('Foo', [
+        $interface = new Interface_('Foo', [
             'stmts' => [
                 new Node\Stmt\ClassConst([new Node\Const_('C1', new Node\Scalar\String_('C1'))]),
                 $methods[0],
@@ -23,4 +24,21 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
 
         $this->assertSame($methods, $interface->getMethods());
     }
+
+    public function testGetConstants() {
+        $constants = [
+            new ClassConst([new \PhpParser\Node\Const_('foo', new String_('foo_value'))]),
+            new ClassConst([new \PhpParser\Node\Const_('bar', new String_('bar_value'))]),
+        ];
+        $class = new Interface_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $constants[0],
+                new ClassMethod('fooBar'),
+                $constants[1],
+            ]
+        ]);
+
+        $this->assertSame($constants, $class->getConstants());
+    }
 }

test/PhpParser/NodeAbstractTest.php

@@ -34,8 +34,9 @@ class NodeAbstractTest extends \PHPUnit\Framework\TestCase
             'startFilePos' => 14,
             'endFilePos' => 15,
             'comments'  => [
-                new Comment('// Comment' . "\n"),
+                new Comment('// Comment 1' . "\n"),
                 new Comment\Doc('/** doc comment */'),
+                new Comment('// Comment 2' . "\n"),
             ],
         ];
 
@@ -79,12 +80,12 @@ class NodeAbstractTest extends \PHPUnit\Framework\TestCase
         $this->assertSame('/** doc comment */', $node->getDocComment()->getText());
         $comments = $node->getComments();
 
-        array_pop($comments); // remove doc comment
+        array_splice($comments, 1, 1, []); // remove doc comment
         $node->setAttribute('comments', $comments);
         $this->assertNull($node->getDocComment());
 
-        array_pop($comments); // remove comment
+        // Remove all comments.
-        $node->setAttribute('comments', $comments);
+        $node->setAttribute('comments', []);
         $this->assertNull($node->getDocComment());
     }
 
@@ -108,6 +109,12 @@ class NodeAbstractTest extends \PHPUnit\Framework\TestCase
         $node->setAttribute('comments', [$c1, $c2]);
         $node->setDocComment($docComment);
         $this->assertSame([$c1, $c2, $docComment], $node->getAttribute('comments'));
+
+        // Replace doc comment that is not at the end.
+        $newDocComment = new Comment\Doc('/** new baz */');
+        $node->setAttribute('comments', [$c1, $docComment, $c2]);
+        $node->setDocComment($newDocComment);
+        $this->assertSame([$c1, $newDocComment, $c2], $node->getAttribute('comments'));
     }
 
     /**
@@ -240,6 +247,7 @@ PHP;
                         "kind": 10
                     }
                 },
+                "flags": 0,
                 "attributes": {
                     "startLine": 4,
                     "endLine": 4
@@ -266,6 +274,7 @@ PHP;
                         "endLine": 4
                     }
                 },
+                "flags": 0,
                 "attributes": {
                     "startLine": 4,
                     "endLine": 4
@@ -298,17 +307,23 @@ PHP;
             "comments": [
                 {
                     "nodeType": "Comment",
-                    "text": "\/\/ comment\n",
+                    "text": "\/\/ comment",
                     "line": 2,
                     "filePos": 6,
-                    "tokenPos": 1
+                    "tokenPos": 1,
+                    "endLine": 2,
+                    "endFilePos": 15,
+                    "endTokenPos": 1
                 },
                 {
                     "nodeType": "Comment_Doc",
                     "text": "\/** doc comment *\/",
                     "line": 3,
                     "filePos": 17,
-                    "tokenPos": 2
+                    "tokenPos": 3,
+                    "endLine": 3,
+                    "endFilePos": 34,
+                    "endTokenPos": 3
                 }
             ],
             "endLine": 6

test/PhpParser/NodeTraverserTest.php

@@ -266,13 +266,17 @@ class NodeTraverserTest extends \PHPUnit\Framework\TestCase
         $traverser->addVisitor($visitor2);
         $traverser->addVisitor($visitor3);
 
+        $getVisitors = (function () {
+            return $this->visitors;
+        })->bindTo($traverser, NodeTraverser::class);
+
         $preExpected = [$visitor1, $visitor2, $visitor3];
-        $this->assertAttributeSame($preExpected, 'visitors', $traverser, 'The appropriate visitors have not been added');
+        $this->assertSame($preExpected, $getVisitors());
 
         $traverser->removeVisitor($visitor2);
 
         $postExpected = [0 => $visitor1, 2 => $visitor3];
-        $this->assertAttributeSame($postExpected, 'visitors', $traverser, 'The appropriate visitors are not present after removal');
+        $this->assertSame($postExpected, $getVisitors());
     }
 
     public function testNoCloneNodes() {

test/PhpParser/NodeVisitor/NameResolverTest.php

@@ -94,13 +94,6 @@ namespace Baz {
     C;
     E;
     K;
-
-    class ClassWithTypeProperties
-    {
-        public float $php = 7.4;
-        public ?Foo $person;
-        protected static ?bool $probability;
-    }
 }
 EOC;
         $expectedCode = <<<'EOC'
@@ -169,12 +162,6 @@ namespace Baz {
     \Y\T\B\C;
     \Y\T\D\E;
     \Z\T\K;
-    class ClassWithTypeProperties
-    {
-        public float $php = 7.4;
-        public ?\Baz\Foo $person;
-        protected static ?bool $probability;
-    }
 }
 EOC;
 
@@ -210,6 +197,14 @@ class A extends B implements C, D {
 
 interface A extends C, D {
     public function a(A $a) : A;
+    public function b(A|B|int $a): A|B|int;
+}
+
+class ClassWithTypeProperties {
+    public float $php = 7.4;
+    public ?Foo $person;
+    protected static ?bool $probability;
+    public A|B|int $prop;
 }
 
 function f(A $a) : A {}
@@ -219,6 +214,10 @@ function(A $a) : A {};
 function fn3(?A $a) : ?A {}
 function fn4(?array $a) : ?array {}
 
+fn(array $a): array => $a;
+fn(A $a): A => $a;
+fn(?A $a): ?A => $a;
+
 A::b();
 A::$b;
 A::B;
@@ -248,6 +247,14 @@ class A extends \NS\B implements \NS\C, \NS\D
 interface A extends \NS\C, \NS\D
 {
     public function a(\NS\A $a) : \NS\A;
+    public function b(\NS\A|\NS\B|int $a) : \NS\A|\NS\B|int;
+}
+class ClassWithTypeProperties
+{
+    public float $php = 7.4;
+    public ?\NS\Foo $person;
+    protected static ?bool $probability;
+    public \NS\A|\NS\B|int $prop;
 }
 function f(\NS\A $a) : \NS\A
 {
@@ -263,6 +270,9 @@ function fn3(?\NS\A $a) : ?\NS\A
 function fn4(?array $a) : ?array
 {
 }
+fn(array $a): array => $a;
+fn(\NS\A $a): \NS\A => $a;
+fn(?\NS\A $a): ?\NS\A => $a;
 \NS\A::b();
 \NS\A::$b;
 \NS\A::B;

test/PhpParser/NodeVisitor/NodeConnectingVisitorTest.php

@@ -0,0 +1,35 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\NodeVisitor;
+
+use PhpParser\Node\Expr\ConstFetch;
+use PhpParser\Node\Stmt\Else_;
+use PhpParser\Node\Stmt\If_;
+use PhpParser\NodeFinder;
+use PhpParser\NodeTraverser;
+use PhpParser\ParserFactory;
+
+final class NodeConnectingVisitorTest extends \PHPUnit\Framework\TestCase
+{
+    public function testConnectsNodeToItsParentNodeAndItsSiblingNodes()
+    {
+        $ast = (new ParserFactory)->create(ParserFactory::PREFER_PHP7)->parse(
+            '<?php if (true) {} else {}'
+        );
+
+        $traverser = new NodeTraverser;
+
+        $traverser->addVisitor(new NodeConnectingVisitor);
+
+        $ast = $traverser->traverse($ast);
+
+        $node = (new NodeFinder)->findFirstInstanceof($ast, Else_::class);
+
+        $this->assertSame(If_::class, get_class($node->getAttribute('parent')));
+        $this->assertSame(ConstFetch::class, get_class($node->getAttribute('previous')));
+
+        $node = (new NodeFinder)->findFirstInstanceof($ast, ConstFetch::class);
+
+        $this->assertSame(Else_::class, get_class($node->getAttribute('next')));
+    }
+}

test/PhpParser/NodeVisitor/ParentConnectingVisitorTest.php

@@ -0,0 +1,28 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\NodeVisitor;
+
+use PhpParser\Node\Stmt\ClassMethod;
+use PhpParser\NodeFinder;
+use PhpParser\NodeTraverser;
+use PhpParser\ParserFactory;
+
+final class ParentConnectingVisitorTest extends \PHPUnit\Framework\TestCase
+{
+    public function testConnectsChildNodeToParentNode()
+    {
+        $ast = (new ParserFactory)->create(ParserFactory::PREFER_PHP7)->parse(
+            '<?php class C { public function m() {} }'
+        );
+
+        $traverser = new NodeTraverser;
+
+        $traverser->addVisitor(new ParentConnectingVisitor);
+
+        $ast = $traverser->traverse($ast);
+
+        $node = (new NodeFinder)->findFirstInstanceof($ast, ClassMethod::class);
+
+        $this->assertSame('C', $node->getAttribute('parent')->name->toString());
+    }
+}

test/PhpParser/ParserTest.php

@@ -60,7 +60,8 @@ EOC;
         $this->assertInstanceOf(Stmt\Function_::class, $fn);
         $this->assertEquals([
             'comments' => [
-                new Comment\Doc('/** Doc comment */', 2, 6, 1),
+                new Comment\Doc('/** Doc comment */',
+                    2, 6, 1, 2, 23, 1),
             ],
             'startLine' => 3,
             'endLine' => 7,
@@ -82,8 +83,10 @@ EOC;
         $this->assertInstanceOf(Stmt\Echo_::class, $echo);
         $this->assertEquals([
             'comments' => [
-                new Comment("// Line\n", 4, 49, 12),
+                new Comment("// Line",
-                new Comment("// Comments\n", 5, 61, 14),
+                    4, 49, 12, 4, 55, 12),
+                new Comment("// Comments",
+                    5, 61, 14, 5, 71, 14),
             ],
             'startLine' => 6,
             'endLine' => 6,

test/code/formatPreservation/classMethodNop.test

@@ -0,0 +1,77 @@
+Adding statement to Class Method containing Nop
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        // I'm just a comment
+    }
+}
+-----
+$stmts[0]->stmts[0]->stmts[] = new Stmt\Expression(new Node\Expr\Variable('foo'));
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        // I'm just a comment
+        $foo;
+    }
+}
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        /* I'm just a comment */
+    }
+}
+-----
+$stmts[0]->stmts[0]->stmts[] = new Stmt\Expression(new Node\Expr\Variable('foo'));
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        /* I'm just a comment */
+        $foo;
+    }
+}
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        /* I'm just a comment */
+    }
+}
+-----
+$stmts[0]->stmts[0]->stmts[0]->setAttribute('comments', [new Comment("/* I'm a new comment */")]);
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        /* I'm a new comment */
+
+    }
+}
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        // I'm just a comment
+    }
+}
+-----
+$stmts[0]->stmts[0]->stmts[0]->setAttribute('comments', [new Comment("// I'm a new comment\n")]);
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        // I'm a new comment
+
+    }
+}

test/code/formatPreservation/insertionOfNullable.test

@@ -174,3 +174,21 @@ $stmts[0]->name = new Node\Name('Foo');
 
 namespace Foo
 { echo 42; }
+-----
+<?php
+try
+{
+}
+catch (Exception)
+{
+}
+-----
+$stmts[0]->catches[0]->var = new Expr\Variable('e');
+-----
+<?php
+try
+{
+}
+catch (Exception $e)
+{
+}

test/code/formatPreservation/listInsertion.test

@@ -307,3 +307,13 @@ $array = [
     1, 2,
     3, 24,
 ];
+-----
+<?php
+function test(): A
+|B {}
+-----
+$stmts[0]->returnType->types[] = new Node\Name('C');
+-----
+<?php
+function test(): A
+|B|C {}

test/code/formatPreservation/listRemoval.test

@@ -39,3 +39,14 @@ function foo(
     $x,
     $y
 ) {}
+-----
+<?php
+function test(): A
+|B
+|C {}
+-----
+array_pop($stmts[0]->returnType->types);
+-----
+<?php
+function test(): A
+|B {}

test/code/formatPreservation/match.test

@@ -0,0 +1,89 @@
+Matches
+-----
+<?php
+$value = match (1) {
+    1
+    =>
+    'one'
+};
+-----
+$stmts[0]->expr->expr->arms[] = new Node\MatchArm(null, new Scalar\String_('two'));
+-----
+<?php
+$value = match (1) {
+    1
+    =>
+    'one',
+    default => 'two',
+};
+-----
+<?php
+$value = match (1) {
+    1, 2 =>
+    'test',
+};
+-----
+$stmts[0]->expr->expr->arms[0]->conds[] = new Scalar\LNumber(3);
+-----
+<?php
+$value = match (1) {
+    1, 2, 3 =>
+    'test',
+};
+-----
+<?php
+$value = match (1) {
+    1
+    =>
+    'one',
+    2
+    =>
+    'two',
+    3
+    =>
+    'three',
+};
+-----
+array_splice($stmts[0]->expr->expr->arms, 1, 1, []);
+-----
+<?php
+$value = match (1) {
+    1
+    =>
+    'one',
+    3
+    =>
+    'three',
+};
+-----
+<?php
+// TODO: Preserve formatting?
+$value = match (1) {
+    default
+    =>
+    'test',
+};
+-----
+$stmts[0]->expr->expr->arms[0]->conds = [new Scalar\LNumber(1)];
+-----
+<?php
+// TODO: Preserve formatting?
+$value = match (1) {
+    1 => 'test',
+};
+-----
+<?php
+// TODO: Preserve formatting?
+$value = match (1) {
+    1
+    =>
+    'test',
+};
+-----
+$stmts[0]->expr->expr->arms[0]->conds = null;
+-----
+<?php
+// TODO: Preserve formatting?
+$value = match (1) {
+    default => 'test',
+};

test/code/formatPreservation/modifierChange.test

@@ -31,3 +31,27 @@ class Bar {
     public final function
     foo() {}
 }
+-----
+<?php
+function test(
+    public T1 $x
+        = 'y',
+    private T2 $y
+        = 'z',
+    T3 $z
+        = 'x',
+) {}
+-----
+$stmts[0]->params[0]->flags = Stmt\Class_::MODIFIER_PRIVATE;
+$stmts[0]->params[1]->flags = 0;
+$stmts[0]->params[2]->flags = Stmt\Class_::MODIFIER_PUBLIC;
+-----
+<?php
+function test(
+    private T1 $x
+        = 'y',
+    T2 $y
+        = 'z',
+    public T3 $z
+        = 'x',
+) {}

test/code/formatPreservation/removalViaNull.test

@@ -192,3 +192,21 @@ $stmts[0]->name = null;
 namespace
     {
     }
+-----
+<?php
+try
+{
+}
+catch (Exception $e)
+{
+}
+-----
+$stmts[0]->catches[0]->var = null;
+-----
+<?php
+try
+{
+}
+catch (Exception)
+{
+}

test/code/parser/errorHandling/lexerErrors.test

@@ -32,24 +32,25 @@ $a = 42;
 @@{ "\1" }@@
 $b = 24;
 -----
+!!positions
 Unexpected character "
" (ASCII 1) from 4:1 to 4:1
 array(
-    0: Stmt_Expression(
+    0: Stmt_Expression[3:1 - 3:8](
-        expr: Expr_Assign(
+        expr: Expr_Assign[3:1 - 3:7](
-            var: Expr_Variable(
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:7](
                 value: 42
             )
         )
     )
-    1: Stmt_Expression(
+    1: Stmt_Expression[5:1 - 5:8](
-        expr: Expr_Assign(
+        expr: Expr_Assign[5:1 - 5:7](
-            var: Expr_Variable(
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:7](
                 value: 24
             )
         )
@@ -62,24 +63,25 @@ $a = 42;
 @@{ "\0" }@@
 $b = 24;
 -----
+!!positions
 Unexpected null byte from 4:1 to 4:1
 array(
-    0: Stmt_Expression(
+    0: Stmt_Expression[3:1 - 3:8](
-        expr: Expr_Assign(
+        expr: Expr_Assign[3:1 - 3:7](
-            var: Expr_Variable(
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:7](
                 value: 42
             )
         )
     )
-    1: Stmt_Expression(
+    1: Stmt_Expression[5:1 - 5:8](
-        expr: Expr_Assign(
+        expr: Expr_Assign[5:1 - 5:7](
-            var: Expr_Variable(
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:7](
                 value: 24
             )
         )
@@ -94,35 +96,36 @@ $b = 2;
 @@{ "\2" }@@
 $c = 3;
 -----
-Unexpected character "@@{ "\1" }@@" (ASCII 1) from 4:1 to 4:1
+!!positions
-Unexpected character "@@{ "\2" }@@" (ASCII 2) from 6:1 to 6:1
+Unexpected character "
" (ASCII 1) from 4:1 to 4:1
+Unexpected character "" (ASCII 2) from 6:1 to 6:1
 array(
-    0: Stmt_Expression(
+    0: Stmt_Expression[3:1 - 3:7](
-        expr: Expr_Assign(
+        expr: Expr_Assign[3:1 - 3:6](
-            var: Expr_Variable(
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:6](
                 value: 1
             )
         )
     )
-    1: Stmt_Expression(
+    1: Stmt_Expression[5:1 - 5:7](
-        expr: Expr_Assign(
+        expr: Expr_Assign[5:1 - 5:6](
-            var: Expr_Variable(
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:6](
                 value: 2
             )
         )
     )
-    2: Stmt_Expression(
+    2: Stmt_Expression[7:1 - 7:7](
-        expr: Expr_Assign(
+        expr: Expr_Assign[7:1 - 7:6](
-            var: Expr_Variable(
+            var: Expr_Variable[7:1 - 7:2](
                 name: c
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[7:6 - 7:6](
                 value: 3
             )
         )

test/code/parser/errorHandling/recovery.test

@@ -594,14 +594,11 @@ isset($x, );
 
 declare(a=42, );
 
-function foo($a, ) {}
-foo($a, );
 global $a, ;
 static $a, ;
 echo $a, ;
 
 for ($a, ; $b, ; $c, );
-function ($a, ) use ($b, ) {};
 -----
 !!php7
 A trailing comma is not allowed here from 5:6 to 5:6
@@ -614,15 +611,12 @@ A trailing comma is not allowed here from 13:17 to 13:17
 A trailing comma is not allowed here from 14:14 to 14:14
 A trailing comma is not allowed here from 16:22 to 16:22
 A trailing comma is not allowed here from 21:13 to 21:13
-A trailing comma is not allowed here from 23:16 to 23:16
+A trailing comma is not allowed here from 23:10 to 23:10
-A trailing comma is not allowed here from 25:10 to 25:10
+A trailing comma is not allowed here from 24:10 to 24:10
-A trailing comma is not allowed here from 26:10 to 26:10
+A trailing comma is not allowed here from 25:8 to 25:8
 A trailing comma is not allowed here from 27:8 to 27:8
-A trailing comma is not allowed here from 29:8 to 29:8
+A trailing comma is not allowed here from 27:14 to 27:14
-A trailing comma is not allowed here from 29:14 to 29:14
+A trailing comma is not allowed here from 27:20 to 27:20
-A trailing comma is not allowed here from 29:20 to 29:20
-A trailing comma is not allowed here from 30:13 to 30:13
-A trailing comma is not allowed here from 30:24 to 30:24
 array(
     0: Stmt_GroupUse(
         type: TYPE_UNKNOWN (0)
@@ -811,52 +805,14 @@ array(
         )
         stmts: null
     )
-    9: Stmt_Function(
+    9: Stmt_Global(
-        byRef: false
-        name: Identifier(
-            name: foo
-        )
-        params: array(
-            0: Param(
-                type: null
-                byRef: false
-                variadic: false
-                var: Expr_Variable(
-                    name: a
-                )
-                default: null
-            )
-        )
-        returnType: null
-        stmts: array(
-        )
-    )
-    10: Stmt_Expression(
-        expr: Expr_FuncCall(
-            name: Name(
-                parts: array(
-                    0: foo
-                )
-            )
-            args: array(
-                0: Arg(
-                    value: Expr_Variable(
-                        name: a
-                    )
-                    byRef: false
-                    unpack: false
-                )
-            )
-        )
-    )
-    11: Stmt_Global(
         vars: array(
             0: Expr_Variable(
                 name: a
             )
         )
     )
-    12: Stmt_Static(
+    10: Stmt_Static(
         vars: array(
             0: Stmt_StaticVar(
                 var: Expr_Variable(
@@ -866,14 +822,14 @@ array(
             )
         )
     )
-    13: Stmt_Echo(
+    11: Stmt_Echo(
         exprs: array(
             0: Expr_Variable(
                 name: a
             )
         )
     )
-    14: Stmt_For(
+    12: Stmt_For(
         init: array(
             0: Expr_Variable(
                 name: a
@@ -892,34 +848,6 @@ array(
         stmts: array(
         )
     )
-    15: Stmt_Expression(
-        expr: Expr_Closure(
-            static: false
-            byRef: false
-            params: array(
-                0: Param(
-                    type: null
-                    byRef: false
-                    variadic: false
-                    var: Expr_Variable(
-                        name: a
-                    )
-                    default: null
-                )
-            )
-            uses: array(
-                0: Expr_ClosureUse(
-                    var: Expr_Variable(
-                        name: b
-                    )
-                    byRef: false
-                )
-            )
-            returnType: null
-            stmts: array(
-            )
-        )
-    )
 )
 -----
 <?php
@@ -1075,6 +1003,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type
@@ -1103,6 +1032,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type1
@@ -1116,6 +1046,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type2
@@ -1144,6 +1075,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: true
@@ -1168,6 +1100,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: true
                 variadic: false
@@ -1192,6 +1125,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Bar
@@ -1225,6 +1159,7 @@ array(
                 )
                 params: array(
                     0: Param(
+                        flags: 0
                         type: Name(
                             parts: array(
                                 0: Baz
@@ -1249,6 +1184,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: Name(
                         parts: array(
                             0: Foo

test/code/parser/expr/arrow_function.test

@@ -16,6 +16,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: Identifier(
                         name: bool
                     )
@@ -39,6 +40,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -62,6 +64,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: true
                     variadic: false
@@ -83,6 +86,7 @@ array(
             byRef: true
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -104,6 +108,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -113,6 +118,7 @@ array(
                     default: null
                 )
                 1: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: true

test/code/parser/expr/closure.test

@@ -16,6 +16,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -43,6 +44,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -96,6 +98,7 @@ array(
             byRef: true
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -131,6 +134,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false

test/code/parser/expr/closure_use_trailing_comma.test

@@ -0,0 +1,27 @@
+Trailing comma in use list
+-----
+<?php
+function() use($a,) {};
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Closure(
+            static: false
+            byRef: false
+            params: array(
+            )
+            uses: array(
+                0: Expr_ClosureUse(
+                    var: Expr_Variable(
+                        name: a
+                    )
+                    byRef: false
+                )
+            )
+            returnType: null
+            stmts: array(
+            )
+        )
+    )
+)

test/code/parser/expr/exprInIsset.test

@@ -0,0 +1,49 @@
+Expressions in isset()
+-----
+<?php
+// This is legal.
+isset(($a), (($b)));
+// This is illegal, but not a syntax error.
+isset(1 + 1);
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Isset(
+            vars: array(
+                0: Expr_Variable(
+                    name: a
+                )
+                1: Expr_Variable(
+                    name: b
+                )
+            )
+            comments: array(
+                0: // This is legal.
+            )
+        )
+        comments: array(
+            0: // This is legal.
+        )
+    )
+    1: Stmt_Expression(
+        expr: Expr_Isset(
+            vars: array(
+                0: Expr_BinaryOp_Plus(
+                    left: Scalar_LNumber(
+                        value: 1
+                    )
+                    right: Scalar_LNumber(
+                        value: 1
+                    )
+                )
+            )
+            comments: array(
+                0: // This is illegal, but not a syntax error.
+            )
+        )
+        comments: array(
+            0: // This is illegal, but not a syntax error.
+        )
+    )
+)

test/code/parser/expr/exprInList.test

@@ -0,0 +1,81 @@
+Expressions in list()
+-----
+<?php
+
+// This is legal.
+list(($a), ((($b)))) = $x;
+// This is illegal, but not a syntax error.
+list(1 + 1) = $x;
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Assign(
+            var: Expr_List(
+                items: array(
+                    0: Expr_ArrayItem(
+                        key: null
+                        value: Expr_Variable(
+                            name: a
+                        )
+                        byRef: false
+                        unpack: false
+                    )
+                    1: Expr_ArrayItem(
+                        key: null
+                        value: Expr_Variable(
+                            name: b
+                        )
+                        byRef: false
+                        unpack: false
+                    )
+                )
+                comments: array(
+                    0: // This is legal.
+                )
+            )
+            expr: Expr_Variable(
+                name: x
+            )
+            comments: array(
+                0: // This is legal.
+            )
+        )
+        comments: array(
+            0: // This is legal.
+        )
+    )
+    1: Stmt_Expression(
+        expr: Expr_Assign(
+            var: Expr_List(
+                items: array(
+                    0: Expr_ArrayItem(
+                        key: null
+                        value: Expr_BinaryOp_Plus(
+                            left: Scalar_LNumber(
+                                value: 1
+                            )
+                            right: Scalar_LNumber(
+                                value: 1
+                            )
+                        )
+                        byRef: false
+                        unpack: false
+                    )
+                )
+                comments: array(
+                    0: // This is illegal, but not a syntax error.
+                )
+            )
+            expr: Expr_Variable(
+                name: x
+            )
+            comments: array(
+                0: // This is illegal, but not a syntax error.
+            )
+        )
+        comments: array(
+            0: // This is illegal, but not a syntax error.
+        )
+    )
+)

test/code/parser/expr/fetchAndCall/constFetch.test

@@ -5,6 +5,8 @@ Constant fetches
 A;
 A::B;
 A::class;
+$a::B;
+$a::class;
 -----
 array(
     0: Stmt_Expression(
@@ -40,4 +42,24 @@ array(
             )
         )
     )
+    3: Stmt_Expression(
+        expr: Expr_ClassConstFetch(
+            class: Expr_Variable(
+                name: a
+            )
+            name: Identifier(
+                name: B
+            )
+        )
+    )
+    4: Stmt_Expression(
+        expr: Expr_ClassConstFetch(
+            class: Expr_Variable(
+                name: a
+            )
+            name: Identifier(
+                name: class
+            )
+        )
+    )
 )

test/code/parser/expr/match.test

@@ -0,0 +1,218 @@
+Match
+-----
+<?php
+
+echo match (1) {
+    0 => 'Foo',
+    1 => 'Bar',
+};
+-----
+!!php7
+array(
+    0: Stmt_Echo(
+        exprs: array(
+            0: Expr_Match(
+                cond: Scalar_LNumber(
+                    value: 1
+                )
+                arms: array(
+                    0: MatchArm(
+                        conds: array(
+                            0: Scalar_LNumber(
+                                value: 0
+                            )
+                        )
+                        body: Scalar_String(
+                            value: Foo
+                        )
+                    )
+                    1: MatchArm(
+                        conds: array(
+                            0: Scalar_LNumber(
+                                value: 1
+                            )
+                        )
+                        body: Scalar_String(
+                            value: Bar
+                        )
+                    )
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+
+$value = match (1) {
+    // list of conditions
+    0, 1 => 'Foo',
+};
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Assign(
+            var: Expr_Variable(
+                name: value
+            )
+            expr: Expr_Match(
+                cond: Scalar_LNumber(
+                    value: 1
+                )
+                arms: array(
+                    0: MatchArm(
+                        conds: array(
+                            0: Scalar_LNumber(
+                                value: 0
+                                comments: array(
+                                    0: // list of conditions
+                                )
+                            )
+                            1: Scalar_LNumber(
+                                value: 1
+                            )
+                        )
+                        body: Scalar_String(
+                            value: Foo
+                        )
+                        comments: array(
+                            0: // list of conditions
+                        )
+                    )
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+
+$result = match ($operator) {
+    BinaryOperator::ADD => $lhs + $rhs,
+};
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Assign(
+            var: Expr_Variable(
+                name: result
+            )
+            expr: Expr_Match(
+                cond: Expr_Variable(
+                    name: operator
+                )
+                arms: array(
+                    0: MatchArm(
+                        conds: array(
+                            0: Expr_ClassConstFetch(
+                                class: Name(
+                                    parts: array(
+                                        0: BinaryOperator
+                                    )
+                                )
+                                name: Identifier(
+                                    name: ADD
+                                )
+                            )
+                        )
+                        body: Expr_BinaryOp_Plus(
+                            left: Expr_Variable(
+                                name: lhs
+                            )
+                            right: Expr_Variable(
+                                name: rhs
+                            )
+                        )
+                    )
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+
+$value = match ($char) {
+    1 => '1',
+    default => 'default'
+};
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Assign(
+            var: Expr_Variable(
+                name: value
+            )
+            expr: Expr_Match(
+                cond: Expr_Variable(
+                    name: char
+                )
+                arms: array(
+                    0: MatchArm(
+                        conds: array(
+                            0: Scalar_LNumber(
+                                value: 1
+                            )
+                        )
+                        body: Scalar_String(
+                            value: 1
+                        )
+                    )
+                    1: MatchArm(
+                        conds: null
+                        body: Scalar_String(
+                            value: default
+                        )
+                    )
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+
+$value = match (1) {
+    0, 1, => 'Foo',
+    default, => 'Bar',
+};
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Assign(
+            var: Expr_Variable(
+                name: value
+            )
+            expr: Expr_Match(
+                cond: Scalar_LNumber(
+                    value: 1
+                )
+                arms: array(
+                    0: MatchArm(
+                        conds: array(
+                            0: Scalar_LNumber(
+                                value: 0
+                            )
+                            1: Scalar_LNumber(
+                                value: 1
+                            )
+                        )
+                        body: Scalar_String(
+                            value: Foo
+                        )
+                    )
+                    1: MatchArm(
+                        conds: null
+                        body: Scalar_String(
+                            value: Bar
+                        )
+                    )
+                )
+            )
+        )
+    )
+)

test/code/parser/expr/uvs/constDeref.test

@@ -0,0 +1,292 @@
+Dereferencing of constants
+-----
+<?php
+
+A->length;
+A->length();
+A[0];
+A[0][1][2];
+A{0};
+
+A::B[0];
+A::B[0][1][2];
+A::B{0};
+A::B->length;
+A::B->length();
+A::B::C;
+A::B::$c;
+A::B::c();
+
+__FUNCTION__[0];
+__FUNCTION__->length;
+__FUNCIONT__->length();
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_PropertyFetch(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+        )
+    )
+    1: Stmt_Expression(
+        expr: Expr_MethodCall(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+            args: array(
+            )
+        )
+    )
+    2: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    3: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ArrayDimFetch(
+                var: Expr_ArrayDimFetch(
+                    var: Expr_ConstFetch(
+                        name: Name(
+                            parts: array(
+                                0: A
+                            )
+                        )
+                    )
+                    dim: Scalar_LNumber(
+                        value: 0
+                    )
+                )
+                dim: Scalar_LNumber(
+                    value: 1
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 2
+            )
+        )
+    )
+    4: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    5: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    6: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ArrayDimFetch(
+                var: Expr_ArrayDimFetch(
+                    var: Expr_ClassConstFetch(
+                        class: Name(
+                            parts: array(
+                                0: A
+                            )
+                        )
+                        name: Identifier(
+                            name: B
+                        )
+                    )
+                    dim: Scalar_LNumber(
+                        value: 0
+                    )
+                )
+                dim: Scalar_LNumber(
+                    value: 1
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 2
+            )
+        )
+    )
+    7: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    8: Stmt_Expression(
+        expr: Expr_PropertyFetch(
+            var: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+        )
+    )
+    9: Stmt_Expression(
+        expr: Expr_MethodCall(
+            var: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+            args: array(
+            )
+        )
+    )
+    10: Stmt_Expression(
+        expr: Expr_ClassConstFetch(
+            class: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: Identifier(
+                name: C
+            )
+        )
+    )
+    11: Stmt_Expression(
+        expr: Expr_StaticPropertyFetch(
+            class: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: VarLikeIdentifier(
+                name: c
+            )
+        )
+    )
+    12: Stmt_Expression(
+        expr: Expr_StaticCall(
+            class: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: Identifier(
+                name: c
+            )
+            args: array(
+            )
+        )
+    )
+    13: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Scalar_MagicConst_Function(
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    14: Stmt_Expression(
+        expr: Expr_PropertyFetch(
+            var: Scalar_MagicConst_Function(
+            )
+            name: Identifier(
+                name: length
+            )
+        )
+    )
+    15: Stmt_Expression(
+        expr: Expr_MethodCall(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: __FUNCIONT__
+                    )
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+            args: array(
+            )
+        )
+    )
+)

test/code/parser/expr/uvs/indirectCall.test

@@ -198,6 +198,7 @@ array(
                         byRef: false
                         params: array(
                             0: Param(
+                                flags: 0
                                 type: null
                                 byRef: false
                                 variadic: false
@@ -264,6 +265,7 @@ array(
                                     byRef: false
                                     params: array(
                                         0: Param(
+                                            flags: 0
                                             type: null
                                             byRef: false
                                             variadic: false

test/code/parser/expr/uvs/misc.test

@@ -2,62 +2,54 @@ Uniform variable syntax in PHP 7 (misc)
 -----
 <?php
 
-A::A[0];
-A::A[0][1][2];
 "string"->length();
+"foo$bar"[0];
+"foo$bar"->length();
 (clone $obj)->b[0](1);
 [0, 1][0] = 1;
 -----
 !!php7
 array(
     0: Stmt_Expression(
-        expr: Expr_ArrayDimFetch(
+        expr: Expr_MethodCall(
-            var: Expr_ClassConstFetch(
+            var: Scalar_String(
-                class: Name(
+                value: string
-                    parts: array(
-                        0: A
-                    )
             )
             name: Identifier(
-                    name: A
+                name: length
             )
-            )
+            args: array(
-            dim: Scalar_LNumber(
-                value: 0
             )
         )
     )
     1: Stmt_Expression(
         expr: Expr_ArrayDimFetch(
-            var: Expr_ArrayDimFetch(
+            var: Scalar_Encapsed(
-                var: Expr_ArrayDimFetch(
-                    var: Expr_ClassConstFetch(
-                        class: Name(
                 parts: array(
-                                0: A
+                    0: Scalar_EncapsedStringPart(
+                        value: foo
                     )
+                    1: Expr_Variable(
+                        name: bar
                     )
-                        name: Identifier(
-                            name: A
                 )
             )
             dim: Scalar_LNumber(
                 value: 0
             )
         )
-                dim: Scalar_LNumber(
-                    value: 1
-                )
-            )
-            dim: Scalar_LNumber(
-                value: 2
-            )
-        )
     )
     2: Stmt_Expression(
         expr: Expr_MethodCall(
-            var: Scalar_String(
+            var: Scalar_Encapsed(
-                value: string
+                parts: array(
+                    0: Scalar_EncapsedStringPart(
+                        value: foo
+                    )
+                    1: Expr_Variable(
+                        name: bar
+                    )
+                )
             )
             name: Identifier(
                 name: length

test/code/parser/expr/uvs/newInstanceofExpr.test

@@ -0,0 +1,61 @@
+Arbitrary expressions in new and instanceof
+-----
+<?php
+
+new ('Foo' . $bar);
+new ('Foo' . $bar)($arg);
+$obj instanceof ('Foo' . $bar);
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_New(
+            class: Expr_BinaryOp_Concat(
+                left: Scalar_String(
+                    value: Foo
+                )
+                right: Expr_Variable(
+                    name: bar
+                )
+            )
+            args: array(
+            )
+        )
+    )
+    1: Stmt_Expression(
+        expr: Expr_New(
+            class: Expr_BinaryOp_Concat(
+                left: Scalar_String(
+                    value: Foo
+                )
+                right: Expr_Variable(
+                    name: bar
+                )
+            )
+            args: array(
+                0: Arg(
+                    value: Expr_Variable(
+                        name: arg
+                    )
+                    byRef: false
+                    unpack: false
+                )
+            )
+        )
+    )
+    2: Stmt_Expression(
+        expr: Expr_Instanceof(
+            expr: Expr_Variable(
+                name: obj
+            )
+            class: Expr_BinaryOp_Concat(
+                left: Scalar_String(
+                    value: Foo
+                )
+                right: Expr_Variable(
+                    name: bar
+                )
+            )
+        )
+    )
+)

test/code/parser/nopPositions.test

@@ -11,3 +11,17 @@ array(
         )
     )
 )
+-----
+<?php
+{
+    /* comment */
+}
+-----
+!!positions
+array(
+    0: Stmt_Nop[3:0 - 3:17](
+        comments: array(
+            0: /* comment */
+        )
+    )
+)

test/code/parser/scalar/numberSeparators.test

@@ -0,0 +1,199 @@
+Different integer syntaxes
+-----
+<?php
+
+6.674_083e-11;
+299_792_458;
+0xCAFE_F00D;
+0b0101_1111;
+0137_041;
+
+// already a valid constant name
+_100;
+
+// syntax errors
+100_;
+1__1;
+1_.0;
+1._0;
+0x_123;
+0b_101;
+1_e2;
+1e_2;
+-----
+Syntax error, unexpected T_STRING from 13:4 to 13:4
+Syntax error, unexpected T_STRING from 14:2 to 14:4
+Syntax error, unexpected T_STRING from 15:2 to 15:2
+Syntax error, unexpected T_STRING from 16:3 to 16:4
+Syntax error, unexpected T_STRING from 17:2 to 17:6
+Syntax error, unexpected T_STRING from 18:2 to 18:6
+Syntax error, unexpected T_STRING from 19:2 to 19:4
+Syntax error, unexpected T_STRING from 20:2 to 20:4
+array(
+    0: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 6.674083E-11
+        )
+    )
+    1: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 299792458
+        )
+    )
+    2: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 3405705229
+        )
+    )
+    3: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 95
+        )
+    )
+    4: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 48673
+        )
+    )
+    5: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _100
+                )
+                comments: array(
+                    0: // already a valid constant name
+                )
+            )
+            comments: array(
+                0: // already a valid constant name
+            )
+        )
+        comments: array(
+            0: // already a valid constant name
+        )
+    )
+    6: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 100
+            comments: array(
+                0: // syntax errors
+            )
+        )
+        comments: array(
+            0: // syntax errors
+        )
+    )
+    7: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _
+                )
+            )
+        )
+    )
+    8: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    9: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: __1
+                )
+            )
+        )
+    )
+    10: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    11: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _
+                )
+            )
+        )
+    )
+    12: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 0
+        )
+    )
+    13: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 1
+        )
+    )
+    14: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _0
+                )
+            )
+        )
+    )
+    15: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 0
+        )
+    )
+    16: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: x_123
+                )
+            )
+        )
+    )
+    17: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 0
+        )
+    )
+    18: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: b_101
+                )
+            )
+        )
+    )
+    19: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    20: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _e2
+                )
+            )
+        )
+    )
+    21: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    22: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: e_2
+                )
+            )
+        )
+    )
+)

test/code/parser/stmt/class/property_promotion.test

@@ -0,0 +1,81 @@
+Property promotion
+-----
+<?php
+
+class Point {
+    public function __construct(
+        public float $x = 0.0,
+        protected array $y = [],
+        private string $z = 'hello'
+    ) {}
+}
+-----
+!!php7
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: Identifier(
+            name: Point
+        )
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassMethod(
+                flags: MODIFIER_PUBLIC (1)
+                byRef: false
+                name: Identifier(
+                    name: __construct
+                )
+                params: array(
+                    0: Param(
+                        flags: MODIFIER_PUBLIC (1)
+                        type: Identifier(
+                            name: float
+                        )
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: x
+                        )
+                        default: Scalar_DNumber(
+                            value: 0
+                        )
+                    )
+                    1: Param(
+                        flags: MODIFIER_PROTECTED (2)
+                        type: Identifier(
+                            name: array
+                        )
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: y
+                        )
+                        default: Expr_Array(
+                            items: array(
+                            )
+                        )
+                    )
+                    2: Param(
+                        flags: MODIFIER_PRIVATE (4)
+                        type: Identifier(
+                            name: string
+                        )
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: z
+                        )
+                        default: Scalar_String(
+                            value: hello
+                        )
+                    )
+                )
+                returnType: null
+                stmts: array(
+                )
+            )
+        )
+    )
+)

test/code/parser/stmt/class/simple.test

@@ -127,6 +127,7 @@ array(
                 )
                 params: array(
                     0: Param(
+                        flags: 0
                         type: null
                         byRef: false
                         variadic: false

test/code/parser/stmt/function/builtinTypeDeclarations.test

@@ -1,7 +1,7 @@
 Scalar type declarations
 -----
 <?php
-function test(bool $a, Int $b, FLOAT $c, StRiNg $d, iterable $e, object $f) : void {}
+function test(bool $a, Int $b, FLOAT $c, StRiNg $d, iterable $e, object $f, mixed $g) : void {}
 -----
 !!php7
 array(
@@ -12,6 +12,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: Identifier(
                     name: bool
                 )
@@ -23,6 +24,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Identifier(
                     name: int
                 )
@@ -34,6 +36,7 @@ array(
                 default: null
             )
             2: Param(
+                flags: 0
                 type: Identifier(
                     name: float
                 )
@@ -45,6 +48,7 @@ array(
                 default: null
             )
             3: Param(
+                flags: 0
                 type: Identifier(
                     name: string
                 )
@@ -56,6 +60,7 @@ array(
                 default: null
             )
             4: Param(
+                flags: 0
                 type: Identifier(
                     name: iterable
                 )
@@ -67,6 +72,7 @@ array(
                 default: null
             )
             5: Param(
+                flags: 0
                 type: Identifier(
                     name: object
                 )
@@ -77,6 +83,18 @@ array(
                 )
                 default: null
             )
+            6: Param(
+                flags: 0
+                type: Identifier(
+                    name: mixed
+                )
+                byRef: false
+                variadic: false
+                var: Expr_Variable(
+                    name: g
+                )
+                default: null
+            )
         )
         returnType: Identifier(
             name: void

test/code/parser/stmt/function/byRef.test

@@ -13,6 +13,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: true
                 variadic: false
@@ -33,6 +34,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false

test/code/parser/stmt/function/defaultValues.test

@@ -22,6 +22,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -37,6 +38,7 @@ array(
                 )
             )
             1: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -48,6 +50,7 @@ array(
                 )
             )
             2: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -66,6 +69,7 @@ array(
                 )
             )
             3: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -79,6 +83,7 @@ array(
                 )
             )
             4: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -92,6 +97,7 @@ array(
                 )
             )
             5: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -104,6 +110,7 @@ array(
                 )
             )
             6: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -116,6 +123,7 @@ array(
                 )
             )
             7: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -136,6 +144,7 @@ array(
                 )
             )
             8: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false

test/code/parser/stmt/function/nullableTypes.test

@@ -14,6 +14,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: NullableType(
                     type: Name(
                         parts: array(
@@ -29,6 +30,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: NullableType(
                     type: Identifier(
                         name: string

test/code/parser/stmt/function/parameters_trailing_comma.test

@@ -0,0 +1,116 @@
+Trailing comma in parameter list
+-----
+<?php
+
+function foo($bar, ) {
+}
+-----
+!!php7
+array(
+    0: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: foo
+        )
+        params: array(
+            0: Param(
+                flags: 0
+                type: null
+                byRef: false
+                variadic: false
+                var: Expr_Variable(
+                    name: bar
+                )
+                default: null
+            )
+        )
+        returnType: null
+        stmts: array(
+        )
+    )
+)
+-----
+<?php
+
+class Foo
+{
+    function __construct($name, $value, )
+    {
+    }
+}
+-----
+!!php7
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: Identifier(
+            name: Foo
+        )
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassMethod(
+                flags: 0
+                byRef: false
+                name: Identifier(
+                    name: __construct
+                )
+                params: array(
+                    0: Param(
+                        flags: 0
+                        type: null
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: name
+                        )
+                        default: null
+                    )
+                    1: Param(
+                        flags: 0
+                        type: null
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: value
+                        )
+                        default: null
+                    )
+                )
+                returnType: null
+                stmts: array(
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+fn($foo, ) => $bar;
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_ArrowFunction(
+            static: false
+            byRef: false
+            params: array(
+                0: Param(
+                    flags: 0
+                    type: null
+                    byRef: false
+                    variadic: false
+                    var: Expr_Variable(
+                        name: foo
+                    )
+                    default: null
+                )
+            )
+            returnType: null
+            expr: Expr_Variable(
+                name: bar
+            )
+        )
+    )
+)

test/code/parser/stmt/function/typeDeclarations.test

@@ -12,6 +12,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -21,6 +22,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Identifier(
                     name: array
                 )
@@ -32,6 +34,7 @@ array(
                 default: null
             )
             2: Param(
+                flags: 0
                 type: Identifier(
                     name: callable
                 )
@@ -43,6 +46,7 @@ array(
                 default: null
             )
             3: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: E

test/code/parser/stmt/function/unionTypes.test

@@ -0,0 +1,93 @@
+Union types
+-----
+<?php
+
+class Test {
+    public A|iterable|null $prop;
+}
+
+function test(A|B $a): int|false {}
+-----
+!!php7
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: Identifier(
+            name: Test
+        )
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_Property(
+                flags: MODIFIER_PUBLIC (1)
+                type: UnionType(
+                    types: array(
+                        0: Name(
+                            parts: array(
+                                0: A
+                            )
+                        )
+                        1: Identifier(
+                            name: iterable
+                        )
+                        2: Identifier(
+                            name: null
+                        )
+                    )
+                )
+                props: array(
+                    0: Stmt_PropertyProperty(
+                        name: VarLikeIdentifier(
+                            name: prop
+                        )
+                        default: null
+                    )
+                )
+            )
+        )
+    )
+    1: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: test
+        )
+        params: array(
+            0: Param(
+                flags: 0
+                type: UnionType(
+                    types: array(
+                        0: Name(
+                            parts: array(
+                                0: A
+                            )
+                        )
+                        1: Name(
+                            parts: array(
+                                0: B
+                            )
+                        )
+                    )
+                )
+                byRef: false
+                variadic: false
+                var: Expr_Variable(
+                    name: a
+                )
+                default: null
+            )
+        )
+        returnType: UnionType(
+            types: array(
+                0: Identifier(
+                    name: int
+                )
+                1: Identifier(
+                    name: false
+                )
+            )
+        )
+        stmts: array(
+        )
+    )
+)

test/code/parser/stmt/function/variadic.test

@@ -14,6 +14,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -23,6 +24,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: true
@@ -43,6 +45,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -52,6 +55,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: null
                 byRef: true
                 variadic: true
@@ -72,6 +76,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -81,6 +86,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type
@@ -105,6 +111,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -114,6 +121,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type