Release PHP-Parser 4.6.0

The trailing newline is no longer part of the comment token.

.gitattributes

@@ -0,0 +1,9 @@
+/doc export-ignore
+/test export-ignore
+/test_old export-ignore
+.gitattributes export-ignore
+.gitignore export-ignore
+.travis.yml export-ignore
+CHANGELOG.md export-ignore
+phpunit.xml.dist export-ignore
+UPGRADE-*.md export-ignore

.gitignore

@@ -2,3 +2,4 @@ vendor/
 composer.lock
 grammar/kmyacc.exe
 grammar/y.output
+.phpunit.result.cache

.travis.yml

@@ -1,6 +1,5 @@
 language: php
 dist: xenial
-sudo: false
 
 cache:
   directories:
@@ -11,12 +10,17 @@ php:
   - 7.1
   - 7.2
   - 7.3
-  - 7.4snapshot
+  - 7.4
   - nightly
 
 install:
   - if [ $TRAVIS_PHP_VERSION = '7.0' ]; then composer require satooshi/php-coveralls '~1.0'; fi
-  - composer install --prefer-dist --ignore-platform-reqs
+  - |
+    if [ $TRAVIS_PHP_VERSION = 'nightly' ]; then
+      composer install --prefer-dist --ignore-platform-reqs;
+    else
+      composer install --prefer-dist;
+    fi
 
 matrix:
   allow_failures:

CHANGELOG.md

@@ -1,8 +1,94 @@
-Version 4.2.3-dev
+Version 4.6.1-dev
 -----------------
 
 Nothing yet.
 
+Version 4.6.0 (2020-07-02)
+--------------------------
+
+### Added
+
+* [PHP 8.0] Added support for trailing commas in parameter lists.
+* [PHP 8.0] Added support for constructor promotion. The parameter visibility is stored in
+  `Node\Param::$flags`.
+
+### Fixed
+
+* Comment tokens now always follow the PHP 8 interpretation, and do not include trailing
+  whitespace.
+* As a result of the previous change, some whitespace issues when inserting a statement into a
+  method containing only a comment, and using the formatting-preserving pretty printer, have been
+  resolved.
+
+Version 4.5.0 (2020-06-03)
+--------------------------
+
+### Added
+
+* [PHP 8.0] Added support for the mixed type. This means `mixed` types are now parsed as an
+  `Identifier` rather than a `Name`.
+* [PHP 8.0] Added support for catching without capturing the exception. This means that
+  `Catch_::$var` may now be null.
+
+Version 4.4.0 (2020-04-10)
+--------------------------
+
+### Added
+
+* Added support for passing union types in builders.
+* Added end line, token position and file position information for comments.
+* Added `getProperty()` method to `ClassLike` nodes.
+
+### Fixed
+
+* Fixed generation of invalid code when using the formatting preserving pretty printer, and
+  inserting code next to certain nop statements. The formatting is still ugly though.
+* `getDocComment()` no longer requires that the very last comment before a node be a doc comment.
+  There may not be non-doc comments between the doc comment and the declaration.
+* Allowed arbitrary expressions in `isset()` and `list()`, rather than just variables.
+  In particular, this allows `isset(($x))`, which is legal PHP code.
+* [PHP 8.0] Add support for [variable syntax tweaks RFC](https://wiki.php.net/rfc/variable_syntax_tweaks).
+
+Version 4.3.0 (2019-11-08)
+--------------------------
+
+### Added
+
+* [PHP 8.0] Added support for union types using a new `UnionType` node.
+
+Version 4.2.5 (2019-10-25)
+--------------------------
+
+### Changed
+
+* Tests and documentation are no longer included in source archives. They can still be accessed
+  by cloning the repository.
+* php-yacc is now used to generate the parser. This has no impact on users of the library.
+
+Version 4.2.4 (2019-09-01)
+--------------------------
+
+### Added
+
+* Added getProperties(), getConstants() and getTraitUses() to ClassLike. (#629, #630)
+
+### Fixed
+
+* Fixed flexible heredoc emulation to check for digits after the end label. This synchronizes
+  behavior with the upcoming PHP 7.3.10 release.
+
+Version 4.2.3 (2019-08-12)
+--------------------------
+
+### Added
+
+* [PHP 7.4] Add support for numeric literal separators. (#615)
+
+### Fixed
+
+* Fixed resolution of return types for arrow functions. (#613)
+* Fixed compatibility with PHP 7.4.
+
 Version 4.2.2 (2019-05-25)
 --------------------------
 

README.md

@@ -3,10 +3,10 @@ PHP Parser
 
 [![Build Status](https://travis-ci.org/nikic/PHP-Parser.svg?branch=master)](https://travis-ci.org/nikic/PHP-Parser) [![Coverage Status](https://coveralls.io/repos/github/nikic/PHP-Parser/badge.svg?branch=master)](https://coveralls.io/github/nikic/PHP-Parser?branch=master)
 
-This is a PHP 5.2 to PHP 7.3 parser written in PHP. Its purpose is to simplify static code analysis and
+This is a PHP 5.2 to PHP 7.4 parser written in PHP. Its purpose is to simplify static code analysis and
 manipulation.
 
-[**Documentation for version 4.x**][doc_master] (stable; for running on PHP >= 7.0; for parsing PHP 5.2 to PHP 7.3).
+[**Documentation for version 4.x**][doc_master] (stable; for running on PHP >= 7.0; for parsing PHP 5.2 to PHP 7.4).
 
 [Documentation for version 3.x][doc_3_x] (unsupported; for running on PHP >= 5.5; for parsing PHP 5.2 to PHP 7.2).
 

composer.json

@@ -17,11 +17,12 @@
         "ext-tokenizer": "*"
     },
     "require-dev": {
-        "phpunit/phpunit": "^6.5 || ^7.0"
+        "phpunit/phpunit": "^6.5 || ^7.0 || ^8.0",
+        "ircmaxell/php-yacc": "0.0.5"
     },
     "extra": {
         "branch-alias": {
-            "dev-master": "4.2-dev"
+            "dev-master": "4.3-dev"
         }
     },
     "autoload": {

doc/0_Introduction.markdown

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-This project is a PHP 5.2 to PHP 7.3 parser **written in PHP itself**.
+This project is a PHP 5.2 to PHP 7.4 parser **written in PHP itself**.
 
 What is this for?
 -----------------
@@ -26,11 +26,11 @@ programmatic PHP code analysis are incidentally PHP developers, not C developers
 What can it parse?
 ------------------
 
-The parser supports parsing PHP 5.2-7.3.
+The parser supports parsing PHP 5.2-7.4.
 
 As the parser is based on the tokens returned by `token_get_all` (which is only able to lex the PHP
 version it runs on), additionally a wrapper for emulating tokens from newer versions is provided.
-This allows to parse PHP 7.3 source code running on PHP 7.0, for example. This emulation is somewhat
+This allows to parse PHP 7.4 source code running on PHP 7.0, for example. This emulation is somewhat
 hacky and not perfect, but it should work well on any sane code.
 
 What output does it produce?
@@ -56,7 +56,7 @@ array(
 ```
 
 This matches the structure of the code: An echo statement, which takes two strings as expressions,
-with the values `Hi` and `World!`.
+with the values `Hi` and `World`.
 
 You can also see that the AST does not contain any whitespace information (but most comments are saved).
 So using it for formatting analysis is not possible.

doc/2_Usage_of_basic_components.markdown

@@ -77,7 +77,7 @@ A parser instance can be reused to parse multiple files.
 Node dumping
 ------------
 
-To dump the abstact syntax tree in human readable form, a `NodeDumper` can be used:
+To dump the abstract syntax tree in human readable form, a `NodeDumper` can be used:
 
 ```php
 <?php

doc/component/Walking_the_AST.markdown

@@ -295,7 +295,7 @@ Simple node finding
 
 While the node visitor mechanism is very flexible, creating a node visitor can be overly cumbersome
 for minor tasks. For this reason a `NodeFinder` is provided, which can find AST nodes that either
-satisfy a certain callback, or which are instanced of a certain node type. A couple of examples are
+satisfy a certain callback, or which are instances of a certain node type. A couple of examples are
 shown in the following:
 
 ```php
@@ -312,7 +312,7 @@ $extendingClasses = $nodeFinder->find($stmts, function(Node $node) {
         && $node->extends !== null;
 });
 
-// Find first class occuring in the AST. Returns null if no class exists.
+// Find first class occurring in the AST. Returns null if no class exists.
 $class = $nodeFinder->findFirstInstanceOf($stmts, Node\Stmt\Class_::class);
 
 // Find first class that has name $name

grammar/README.md

@@ -21,8 +21,10 @@ applied to it:
 Building the parser
 ===================
 
-In order to rebuild the parser, you need [moriyoshi's fork of kmyacc](https://github.com/moriyoshi/kmyacc-forked).
+Run `php grammar/rebuildParsers.php` to rebuild the parsers. Additional options:
-After you compiled/installed it, run the `rebuildParsers.php` script.
 
-By default only the `Parser.php` is built. If you want to additionally emit debug symbols and create `y.output`, run the
+ * The `KMYACC` environment variable can be used to specify an alternative `kmyacc` binary.
-script with `--debug`. If you want to retain the preprocessed grammar pass `--keep-tmp-grammar`.
+   By default the `phpyacc` dev dependency will be used. To use the original `kmyacc`, you
+   need to compile [moriyoshi's fork](https://github.com/moriyoshi/kmyacc-forked).
+ * The `--debug` option enables emission of debug symbols and creates the `y.output` file.
+ * The `--keep-tmp-grammar` option preserves the preprocessed grammar file.

grammar/php7.y

@@ -232,7 +232,7 @@ non_empty_statement:
     | T_RETURN optional_expr semi                           { $$ = Stmt\Return_[$2]; }
     | T_GLOBAL global_var_list semi                         { $$ = Stmt\Global_[$2]; }
     | T_STATIC static_var_list semi                         { $$ = Stmt\Static_[$2]; }
-    | T_ECHO expr_list semi                                 { $$ = Stmt\Echo_[$2]; }
+    | T_ECHO expr_list_forbid_comma semi                    { $$ = Stmt\Echo_[$2]; }
     | T_INLINE_HTML                                         { $$ = Stmt\InlineHTML[$1]; }
     | expr semi                                             { $$ = Stmt\Expression[$1]; }
     | T_UNSET '(' variables_list ')' semi                   { $$ = Stmt\Unset_[$3]; }
@@ -269,7 +269,7 @@ name_union:
 ;
 
 catch:
-    T_CATCH '(' name_union plain_variable ')' '{' inner_statement_list '}'
+    T_CATCH '(' name_union optional_plain_variable ')' '{' inner_statement_list '}'
         { $$ = Stmt\Catch_[$3, $4, $7]; }
 ;
 
@@ -440,7 +440,7 @@ foreach_variable:
 ;
 
 parameter_list:
-      non_empty_parameter_list no_comma                     { $$ = $1; }
+      non_empty_parameter_list optional_comma               { $$ = $1; }
     | /* empty */                                           { $$ = array(); }
 ;
 
@@ -449,18 +449,28 @@ non_empty_parameter_list:
     | non_empty_parameter_list ',' parameter                { push($1, $3); }
 ;
 
+optional_visibility_modifier:
+      /* empty */               { $$ = 0; }
+    | T_PUBLIC                  { $$ = Stmt\Class_::MODIFIER_PUBLIC; }
+    | T_PROTECTED               { $$ = Stmt\Class_::MODIFIER_PROTECTED; }
+    | T_PRIVATE                 { $$ = Stmt\Class_::MODIFIER_PRIVATE; }
+;
+
 parameter:
-      optional_type optional_ref optional_ellipsis plain_variable
+      optional_visibility_modifier optional_type optional_ref optional_ellipsis plain_variable
-          { $$ = Node\Param[$4, null, $1, $2, $3]; $this->checkParam($$); }
+          { $$ = new Node\Param($5, null, $2, $3, $4, attributes(), $1);
-    | optional_type optional_ref optional_ellipsis plain_variable '=' expr
+            $this->checkParam($$); }
-          { $$ = Node\Param[$4, $6, $1, $2, $3]; $this->checkParam($$); }
+    | optional_visibility_modifier optional_type optional_ref optional_ellipsis plain_variable '=' expr
-    | optional_type optional_ref optional_ellipsis error
+          { $$ = new Node\Param($5, $7, $2, $3, $4, attributes(), $1);
-          { $$ = Node\Param[Expr\Error[], null, $1, $2, $3]; }
+            $this->checkParam($$); }
+    | optional_visibility_modifier optional_type optional_ref optional_ellipsis error
+          { $$ = new Node\Param(Expr\Error[], null, $2, $3, $4, attributes(), $1); }
 ;
 
 type_expr:
       type                                                  { $$ = $1; }
     | '?' type                                              { $$ = Node\NullableType[$2]; }
+    | union_type                                            { $$ = Node\UnionType[$1]; }
 ;
 
 type:
@@ -469,6 +479,11 @@ type:
     | T_CALLABLE                                            { $$ = Node\Identifier['callable']; }
 ;
 
+union_type:
+      type '|' type                                         { init($1, $3); }
+    | union_type '|' type                                   { push($1, $3); }
+;
+
 optional_type:
       /* empty */                                           { $$ = null; }
     | type_expr                                             { $$ = $1; }
@@ -626,10 +641,14 @@ property_declaration:
     | property_decl_name '=' expr                           { $$ = Stmt\PropertyProperty[$1, $3]; }
 ;
 
-expr_list:
+expr_list_forbid_comma:
       non_empty_expr_list no_comma                          { $$ = $1; }
 ;
 
+expr_list_allow_comma:
+      non_empty_expr_list optional_comma                    { $$ = $1; }
+;
+
 non_empty_expr_list:
       non_empty_expr_list ',' expr                          { push($1, $3); }
     | expr                                                  { init($1); }
@@ -637,7 +656,7 @@ non_empty_expr_list:
 
 for_expr:
       /* empty */                                           { $$ = array(); }
-    | expr_list                                             { $$ = $1; }
+    | expr_list_forbid_comma                                { $$ = $1; }
 ;
 
 expr:
@@ -700,7 +719,7 @@ expr:
     | expr '?' expr ':' expr                                { $$ = Expr\Ternary[$1, $3,   $5]; }
     | expr '?' ':' expr                                     { $$ = Expr\Ternary[$1, null, $4]; }
     | expr T_COALESCE expr                                  { $$ = Expr\BinaryOp\Coalesce[$1, $3]; }
-    | T_ISSET '(' variables_list ')'                        { $$ = Expr\Isset_[$3]; }
+    | T_ISSET '(' expr_list_allow_comma ')'                 { $$ = Expr\Isset_[$3]; }
     | T_EMPTY '(' expr ')'                                  { $$ = Expr\Empty_[$3]; }
     | T_INCLUDE expr                                        { $$ = Expr\Include_[$2, Expr\Include_::TYPE_INCLUDE]; }
     | T_INCLUDE_ONCE expr                                   { $$ = Expr\Include_[$2, Expr\Include_::TYPE_INCLUDE_ONCE]; }
@@ -794,12 +813,13 @@ name:
 class_name_reference:
       class_name                                            { $$ = $1; }
     | new_variable                                          { $$ = $1; }
+    | '(' expr ')'                                          { $$ = $2; }
     | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 class_name_or_var:
       class_name                                            { $$ = $1; }
-    | dereferencable                                        { $$ = $1; }
+    | fully_dereferencable                                  { $$ = $1; }
 ;
 
 exit_expr:
@@ -821,9 +841,20 @@ ctor_arguments:
 
 constant:
       name                                                  { $$ = Expr\ConstFetch[$1]; }
-    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM identifier_ex
+    | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
+    | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
+    | T_DIR                                                 { $$ = Scalar\MagicConst\Dir[]; }
+    | T_CLASS_C                                             { $$ = Scalar\MagicConst\Class_[]; }
+    | T_TRAIT_C                                             { $$ = Scalar\MagicConst\Trait_[]; }
+    | T_METHOD_C                                            { $$ = Scalar\MagicConst\Method[]; }
+    | T_FUNC_C                                              { $$ = Scalar\MagicConst\Function_[]; }
+    | T_NS_C                                                { $$ = Scalar\MagicConst\Namespace_[]; }
+;
+
+class_constant:
+      class_name_or_var T_PAAMAYIM_NEKUDOTAYIM identifier_ex
           { $$ = Expr\ClassConstFetch[$1, $3]; }
-    /* We interpret and isolated FOO:: as an unfinished class constant fetch. It could also be
+    /* We interpret an isolated FOO:: as an unfinished class constant fetch. It could also be
        an unfinished static property fetch or unfinished scoped call. */
     | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM error
           { $$ = Expr\ClassConstFetch[$1, new Expr\Error(stackAttributes(#3))]; $this->errorState = 2; }
@@ -843,28 +874,21 @@ dereferencable_scalar:
     | T_CONSTANT_ENCAPSED_STRING
           { $attrs = attributes(); $attrs['kind'] = strKind($1);
             $$ = new Scalar\String_(Scalar\String_::parse($1), $attrs); }
+    | '"' encaps_list '"'
+          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
+            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
 ;
 
 scalar:
       T_LNUMBER                                             { $$ = $this->parseLNumber($1, attributes()); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
-    | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
-    | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
-    | T_DIR                                                 { $$ = Scalar\MagicConst\Dir[]; }
-    | T_CLASS_C                                             { $$ = Scalar\MagicConst\Class_[]; }
-    | T_TRAIT_C                                             { $$ = Scalar\MagicConst\Trait_[]; }
-    | T_METHOD_C                                            { $$ = Scalar\MagicConst\Method[]; }
-    | T_FUNC_C                                              { $$ = Scalar\MagicConst\Function_[]; }
-    | T_NS_C                                                { $$ = Scalar\MagicConst\Namespace_[]; }
     | dereferencable_scalar                                 { $$ = $1; }
     | constant                                              { $$ = $1; }
+    | class_constant                                        { $$ = $1; }
     | T_START_HEREDOC T_ENCAPSED_AND_WHITESPACE T_END_HEREDOC
           { $$ = $this->parseDocString($1, $2, $3, attributes(), stackAttributes(#3), true); }
     | T_START_HEREDOC T_END_HEREDOC
           { $$ = $this->parseDocString($1, '', $2, attributes(), stackAttributes(#2), true); }
-    | '"' encaps_list '"'
-          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
-            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
     | T_START_HEREDOC encaps_list T_END_HEREDOC
           { $$ = $this->parseDocString($1, $2, $3, attributes(), stackAttributes(#3), true); }
 ;
@@ -874,10 +898,16 @@ optional_expr:
     | expr                                                  { $$ = $1; }
 ;
 
-dereferencable:
+fully_dereferencable:
       variable                                              { $$ = $1; }
     | '(' expr ')'                                          { $$ = $2; }
     | dereferencable_scalar                                 { $$ = $1; }
+    | class_constant                                        { $$ = $1; }
+;
+
+array_object_dereferencable:
+      fully_dereferencable                                  { $$ = $1; }
+    | constant                                              { $$ = $1; }
 ;
 
 callable_expr:
@@ -888,18 +918,23 @@ callable_expr:
 
 callable_variable:
       simple_variable                                       { $$ = Expr\Variable[$1]; }
-    | dereferencable '[' optional_expr ']'                  { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | array_object_dereferencable '[' optional_expr ']'     { $$ = Expr\ArrayDimFetch[$1, $3]; }
-    | constant '[' optional_expr ']'                        { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | array_object_dereferencable '{' expr '}'              { $$ = Expr\ArrayDimFetch[$1, $3]; }
-    | dereferencable '{' expr '}'                           { $$ = Expr\ArrayDimFetch[$1, $3]; }
     | function_call                                         { $$ = $1; }
-    | dereferencable T_OBJECT_OPERATOR property_name argument_list
+    | array_object_dereferencable T_OBJECT_OPERATOR property_name argument_list
           { $$ = Expr\MethodCall[$1, $3, $4]; }
 ;
 
+optional_plain_variable:
+      /* empty */                                           { $$ = null; }
+    | plain_variable                                        { $$ = $1; }
+;
+
 variable:
       callable_variable                                     { $$ = $1; }
     | static_member                                         { $$ = $1; }
-    | dereferencable T_OBJECT_OPERATOR property_name        { $$ = Expr\PropertyFetch[$1, $3]; }
+    | array_object_dereferencable T_OBJECT_OPERATOR property_name
+          { $$ = Expr\PropertyFetch[$1, $3]; }
 ;
 
 simple_variable:
@@ -944,22 +979,7 @@ property_name:
 ;
 
 list_expr:
-      T_LIST '(' list_expr_elements ')'                     { $$ = Expr\List_[$3]; }
+      T_LIST '(' inner_array_pair_list ')'                  { $$ = Expr\List_[$3]; }
-;
-
-list_expr_elements:
-      list_expr_elements ',' list_expr_element              { push($1, $3); }
-    | list_expr_element                                     { init($1); }
-;
-
-list_expr_element:
-      variable                                              { $$ = Expr\ArrayItem[$1, null, false]; }
-    | '&' variable                                          { $$ = Expr\ArrayItem[$2, null, true]; }
-    | list_expr                                             { $$ = Expr\ArrayItem[$1, null, false]; }
-    | expr T_DOUBLE_ARROW variable                          { $$ = Expr\ArrayItem[$3, $1, false]; }
-    | expr T_DOUBLE_ARROW '&' variable                      { $$ = Expr\ArrayItem[$4, $1, true]; }
-    | expr T_DOUBLE_ARROW list_expr                         { $$ = Expr\ArrayItem[$3, $1, false]; }
-    | /* empty */                                           { $$ = null; }
 ;
 
 array_pair_list:
@@ -979,10 +999,12 @@ inner_array_pair_list:
 ;
 
 array_pair:
-      expr T_DOUBLE_ARROW expr                              { $$ = Expr\ArrayItem[$3, $1,   false]; }
+      expr                                                  { $$ = Expr\ArrayItem[$1, null, false]; }
-    | expr                                                  { $$ = Expr\ArrayItem[$1, null, false]; }
-    | expr T_DOUBLE_ARROW '&' variable                      { $$ = Expr\ArrayItem[$4, $1,   true]; }
     | '&' variable                                          { $$ = Expr\ArrayItem[$2, null, true]; }
+    | list_expr                                             { $$ = Expr\ArrayItem[$1, null, false]; }
+    | expr T_DOUBLE_ARROW expr                              { $$ = Expr\ArrayItem[$3, $1,   false]; }
+    | expr T_DOUBLE_ARROW '&' variable                      { $$ = Expr\ArrayItem[$4, $1,   true]; }
+    | expr T_DOUBLE_ARROW list_expr                         { $$ = Expr\ArrayItem[$3, $1,   false]; }
     | T_ELLIPSIS expr                                       { $$ = Expr\ArrayItem[$2, null, false, attributes(), true]; }
     | /* empty */                                           { $$ = null; }
 ;

grammar/rebuildParsers.php

@@ -13,13 +13,10 @@ $tmpResultFile  = __DIR__ . '/tmp_parser.php';
 $resultDir = __DIR__ . '/../lib/PhpParser/Parser';
 $tokensResultsFile = $resultDir . '/Tokens.php';
 
-// check for kmyacc binary in this directory, otherwise fall back to global name
+$kmyacc = getenv('KMYACC');
-if (file_exists(__DIR__ . '/kmyacc.exe')) {
+if (!$kmyacc) {
-    $kmyacc = __DIR__ . '/kmyacc.exe';
+    // Use phpyacc from dev dependencies by default.
-} else if (file_exists(__DIR__ . '/kmyacc')) {
+    $kmyacc = PHP_BINARY . ' ' . __DIR__ . '/../vendor/bin/phpyacc';
-    $kmyacc = __DIR__ . '/kmyacc';
-} else {
-    $kmyacc = 'kmyacc';
 }
 
 $options = array_flip($argv);
@@ -62,8 +59,7 @@ foreach ($grammarFileToName as $grammarFile => $name) {
     $additionalArgs = $optionDebug ? '-t -v' : '';
 
     echo "Building $name parser.\n";
-    $output = trim(shell_exec("$kmyacc $additionalArgs -l -m $skeletonFile -p $name $tmpGrammarFile 2>&1"));
+    $output = execCmd("$kmyacc $additionalArgs -m $skeletonFile -p $name $tmpGrammarFile");
-    echo "Output: \"$output\"\n";
 
     $resultCode = file_get_contents($tmpResultFile);
     $resultCode = removeTrailingWhitespace($resultCode);
@@ -73,8 +69,7 @@ foreach ($grammarFileToName as $grammarFile => $name) {
     unlink($tmpResultFile);
 
     echo "Building token definition.\n";
-    $output = trim(shell_exec("$kmyacc -l -m $tokensTemplate $tmpGrammarFile 2>&1"));
+    $output = execCmd("$kmyacc -m $tokensTemplate $tmpGrammarFile");
-    assert($output === '');
     rename($tmpResultFile, $tokensResultsFile);
 
     if (!$optionKeepTmpGrammar) {
@@ -183,7 +178,7 @@ function resolveMacros($code) {
 
                 return '$startAttributes = ' . $args[1] . ';'
                     . ' if (isset($startAttributes[\'comments\']))'
-                    . ' { ' . $args[0] . ' = new Stmt\Nop($this->createZeroLengthAttributes($startAttributes)); }'
+                    . ' { ' . $args[0] . ' = new Stmt\Nop($this->createCommentNopAttributes($startAttributes[\'comments\'])); }'
                     . ' else { ' . $args[0] . ' = null; }';
             }
 
@@ -234,6 +229,15 @@ function ensureDirExists($dir) {
     }
 }
 
+function execCmd($cmd) {
+    $output = trim(shell_exec("$cmd 2>&1"));
+    if ($output !== "") {
+        echo "> " . $cmd . "\n";
+        echo $output;
+    }
+    return $output;
+}
+
 //////////////////////////////
 /// Regex helper functions ///
 //////////////////////////////

lib/PhpParser/Builder/Param.php

@@ -12,7 +12,7 @@ class Param implements PhpParser\Builder
 
     protected $default = null;
 
-    /** @var string|Node\Name|Node\NullableType|null */
+    /** @var Node\Identifier|Node\Name|Node\NullableType|null */
     protected $type = null;
 
     protected $byRef = false;
@@ -44,7 +44,7 @@ class Param implements PhpParser\Builder
     /**
      * Sets type for the parameter.
      *
-     * @param string|Node\Name|Node\NullableType $type Parameter type
+     * @param string|Node\Name|Node\NullableType|Node\UnionType $type Parameter type
      *
      * @return $this The builder instance (for fluid interface)
      */
@@ -60,7 +60,7 @@ class Param implements PhpParser\Builder
     /**
      * Sets type for the parameter.
      *
-     * @param string|Node\Name|Node\NullableType $type Parameter type
+     * @param string|Node\Name|Node\NullableType|Node\UnionType $type Parameter type
      *
      * @return $this The builder instance (for fluid interface)
      *

lib/PhpParser/BuilderHelpers.php

@@ -8,6 +8,7 @@ use PhpParser\Node\Name;
 use PhpParser\Node\NullableType;
 use PhpParser\Node\Scalar;
 use PhpParser\Node\Stmt;
+use PhpParser\Node\UnionType;
 
 /**
  * This class defines helpers used in the implementation of builders. Don't use it directly.
@@ -158,16 +159,19 @@ final class BuilderHelpers
      * In particular, builtin types become Identifiers, custom types become Names and nullables
      * are wrapped in NullableType nodes.
      *
-     * @param string|Name|Identifier|NullableType $type The type to normalize
+     * @param string|Name|Identifier|NullableType|UnionType $type The type to normalize
      *
-     * @return Name|Identifier|NullableType The normalized type
+     * @return Name|Identifier|NullableType|UnionType The normalized type
      */
     public static function normalizeType($type) {
         if (!is_string($type)) {
-            if (!$type instanceof Name && !$type instanceof Identifier
+            if (
-                    && !$type instanceof NullableType) {
+                !$type instanceof Name && !$type instanceof Identifier &&
+                !$type instanceof NullableType && !$type instanceof UnionType
+            ) {
                 throw new \LogicException(
-                    'Type must be a string, or an instance of Name, Identifier or NullableType');
+                    'Type must be a string, or an instance of Name, Identifier, NullableType or UnionType'
+                );
             }
             return $type;
         }
@@ -179,7 +183,7 @@ final class BuilderHelpers
         }
 
         $builtinTypes = [
-            'array', 'callable', 'string', 'int', 'float', 'bool', 'iterable', 'void', 'object'
+            'array', 'callable', 'string', 'int', 'float', 'bool', 'iterable', 'void', 'object', 'mixed'
         ];
 
         $lowerType = strtolower($type);
@@ -193,7 +197,11 @@ final class BuilderHelpers
             throw new \LogicException('void type cannot be nullable');
         }
 
-        return $nullable ? new Node\NullableType($type) : $type;
+        if ($nullable && (string) $type === 'mixed') {
+            throw new \LogicException('mixed type cannot be nullable');
+        }
+
+        return $nullable ? new NullableType($type) : $type;
     }
 
     /**

lib/PhpParser/Comment.php

@@ -5,9 +5,12 @@ namespace PhpParser;
 class Comment implements \JsonSerializable
 {
     protected $text;
-    protected $line;
+    protected $startLine;
-    protected $filePos;
+    protected $startFilePos;
-    protected $tokenPos;
+    protected $startTokenPos;
+    protected $endLine;
+    protected $endFilePos;
+    protected $endTokenPos;
 
     /**
      * Constructs a comment node.
@@ -18,12 +21,17 @@ class Comment implements \JsonSerializable
      * @param int    $startTokenPos Token offset the comment started on
      */
     public function __construct(
-        string $text, int $startLine = -1, int $startFilePos = -1, int $startTokenPos = -1
+        string $text,
+        int $startLine = -1, int $startFilePos = -1, int $startTokenPos = -1,
+        int $endLine = -1, int $endFilePos = -1, int $endTokenPos = -1
     ) {
         $this->text = $text;
-        $this->line = $startLine;
+        $this->startLine = $startLine;
-        $this->filePos = $startFilePos;
+        $this->startFilePos = $startFilePos;
-        $this->tokenPos = $startTokenPos;
+        $this->startTokenPos = $startTokenPos;
+        $this->endLine = $endLine;
+        $this->endFilePos = $endFilePos;
+        $this->endTokenPos = $endTokenPos;
     }
 
     /**
@@ -38,28 +46,88 @@ class Comment implements \JsonSerializable
     /**
      * Gets the line number the comment started on.
      *
-     * @return int Line number
+     * @return int Line number (or -1 if not available)
      */
-    public function getLine() : int {
+    public function getStartLine() : int {
-        return $this->line;
+        return $this->startLine;
     }
 
     /**
      * Gets the file offset the comment started on.
      *
-     * @return int File offset
+     * @return int File offset (or -1 if not available)
      */
-    public function getFilePos() : int {
+    public function getStartFilePos() : int {
-        return $this->filePos;
+        return $this->startFilePos;
     }
 
     /**
      * Gets the token offset the comment started on.
      *
+     * @return int Token offset (or -1 if not available)
+     */
+    public function getStartTokenPos() : int {
+        return $this->startTokenPos;
+    }
+
+    /**
+     * Gets the line number the comment ends on.
+     *
+     * @return int Line number (or -1 if not available)
+     */
+    public function getEndLine() : int {
+        return $this->endLine;
+    }
+
+    /**
+     * Gets the file offset the comment ends on.
+     *
+     * @return int File offset (or -1 if not available)
+     */
+    public function getEndFilePos() : int {
+        return $this->endFilePos;
+    }
+
+    /**
+     * Gets the token offset the comment ends on.
+     *
+     * @return int Token offset (or -1 if not available)
+     */
+    public function getEndTokenPos() : int {
+        return $this->endTokenPos;
+    }
+
+    /**
+     * Gets the line number the comment started on.
+     *
+     * @deprecated Use getStartLine() instead
+     *
+     * @return int Line number
+     */
+    public function getLine() : int {
+        return $this->startLine;
+    }
+
+    /**
+     * Gets the file offset the comment started on.
+     *
+     * @deprecated Use getStartFilePos() instead
+     *
+     * @return int File offset
+     */
+    public function getFilePos() : int {
+        return $this->startFilePos;
+    }
+
+    /**
+     * Gets the token offset the comment started on.
+     *
+     * @deprecated Use getStartTokenPos() instead
+     *
      * @return int Token offset
      */
     public function getTokenPos() : int {
-        return $this->tokenPos;
+        return $this->startTokenPos;
     }
 
     /**
@@ -159,9 +227,13 @@ class Comment implements \JsonSerializable
         return [
             'nodeType' => $type,
             'text' => $this->text,
-            'line' => $this->line,
+            // TODO: Rename these to include "start".
-            'filePos' => $this->filePos,
+            'line' => $this->startLine,
-            'tokenPos' => $this->tokenPos,
+            'filePos' => $this->startFilePos,
+            'tokenPos' => $this->startTokenPos,
+            'endLine' => $this->endLine,
+            'endFilePos' => $this->endFilePos,
+            'endTokenPos' => $this->endTokenPos,
         ];
     }
 }

lib/PhpParser/JsonDecoder.php

@@ -73,7 +73,9 @@ class JsonDecoder
         }
 
         return new $className(
-            $value['text'], $value['line'] ?? -1, $value['filePos'] ?? -1, $value['tokenPos'] ?? -1
+            $value['text'],
+            $value['line'] ?? -1, $value['filePos'] ?? -1, $value['tokenPos'] ?? -1,
+            $value['endLine'] ?? -1, $value['endFilePos'] ?? -1, $value['endTokenPos'] ?? -1
         );
     }
 

lib/PhpParser/Lexer.php

@@ -37,10 +37,15 @@ class Lexer
         // map from internal tokens to PhpParser tokens
         $this->tokenMap = $this->createTokenMap();
 
+        // Compatibility define for PHP < 7.4
+        if (!defined('T_BAD_CHARACTER')) {
+            \define('T_BAD_CHARACTER', -1);
+        }
+
         // map of tokens to drop while lexing (the map is only used for isset lookup,
         // that's why the value is simply set to 1; the value is never actually used.)
         $this->dropTokens = array_fill_keys(
-            [\T_WHITESPACE, \T_OPEN_TAG, \T_COMMENT, \T_DOC_COMMENT], 1
+            [\T_WHITESPACE, \T_OPEN_TAG, \T_COMMENT, \T_DOC_COMMENT, \T_BAD_CHARACTER], 1
         );
 
         $defaultAttributes = ['comments', 'startLine', 'endLine'];
@@ -84,7 +89,7 @@ class Lexer
 
         error_clear_last();
         $this->tokens = @token_get_all($code);
-        $this->handleErrors($errorHandler);
+        $this->postprocessTokens($errorHandler);
 
         if (false !== $scream) {
             ini_set('xdebug.scream', $scream);
@@ -92,13 +97,9 @@ class Lexer
     }
 
     private function handleInvalidCharacterRange($start, $end, $line, ErrorHandler $errorHandler) {
+        $tokens = [];
         for ($i = $start; $i < $end; $i++) {
             $chr = $this->code[$i];
-            if ($chr === 'b' || $chr === 'B') {
-                // HHVM does not treat b" tokens correctly, so ignore these
-                continue;
-            }
-
             if ($chr === "\0") {
                 // PHP cuts error message after null byte, so need special case
                 $errorMsg = 'Unexpected null byte';
@@ -108,6 +109,7 @@ class Lexer
                 );
             }
 
+            $tokens[] = [\T_BAD_CHARACTER, $chr, $line];
             $errorHandler->handleError(new Error($errorMsg, [
                 'startLine' => $line,
                 'endLine' => $line,
@@ -115,6 +117,7 @@ class Lexer
                 'endFilePos' => $i,
             ]));
         }
+        return $tokens;
     }
 
     /**
@@ -128,43 +131,57 @@ class Lexer
             && substr($token[1], -2) !== '*/';
     }
 
-    /**
+    protected function postprocessTokens(ErrorHandler $errorHandler) {
-     * Check whether an error *may* have occurred during tokenization.
-     *
-     * @return bool
-     */
-    private function errorMayHaveOccurred() : bool {
-        if (defined('HHVM_VERSION')) {
-            // In HHVM token_get_all() does not throw warnings, so we need to conservatively
-            // assume that an error occurred
-            return true;
-        }
-
-        return null !== error_get_last();
-    }
-
-    protected function handleErrors(ErrorHandler $errorHandler) {
-        if (!$this->errorMayHaveOccurred()) {
-            return;
-        }
-
         // PHP's error handling for token_get_all() is rather bad, so if we want detailed
         // error information we need to compute it ourselves. Invalid character errors are
         // detected by finding "gaps" in the token array. Unterminated comments are detected
         // by checking if a trailing comment has a "*/" at the end.
+        //
+        // Additionally, we canonicalize to the PHP 8 comment format here, which does not include
+        // the trailing whitespace anymore
 
         $filePos = 0;
         $line = 1;
-        foreach ($this->tokens as $token) {
+        $numTokens = \count($this->tokens);
+        for ($i = 0; $i < $numTokens; $i++) {
+            $token = $this->tokens[$i];
+
+            // Since PHP 7.4 invalid characters are represented by a T_BAD_CHARACTER token.
+            // In this case we only need to emit an error.
+            if ($token[0] === \T_BAD_CHARACTER) {
+                $this->handleInvalidCharacterRange($filePos, $filePos + 1, $line, $errorHandler);
+            }
+
+            if ($token[0] === \T_COMMENT && preg_match('/(\r\n|\n|\r)$/D', $token[1], $matches)) {
+                $trailingNewline = $matches[0];
+                $token[1] = substr($token[1], 0, -strlen($trailingNewline));
+                $this->tokens[$i] = $token;
+                if (isset($this->tokens[$i + 1]) && $this->tokens[$i + 1][0] === \T_WHITESPACE) {
+                    // Move trailing newline into following T_WHITESPACE token, if it already exists.
+                    $this->tokens[$i + 1][1] = $trailingNewline . $this->tokens[$i + 1][1];
+                    $this->tokens[$i + 1][2]--;
+                } else {
+                    // Otherwise, we need to create a new T_WHITESPACE token.
+                    array_splice($this->tokens, $i + 1, 0, [
+                        [\T_WHITESPACE, $trailingNewline, $line],
+                    ]);
+                    $numTokens++;
+                }
+            }
+
             $tokenValue = \is_string($token) ? $token : $token[1];
             $tokenLen = \strlen($tokenValue);
 
             if (substr($this->code, $filePos, $tokenLen) !== $tokenValue) {
                 // Something is missing, must be an invalid character
                 $nextFilePos = strpos($this->code, $tokenValue, $filePos);
-                $this->handleInvalidCharacterRange(
+                $badCharTokens = $this->handleInvalidCharacterRange(
                     $filePos, $nextFilePos, $line, $errorHandler);
                 $filePos = (int) $nextFilePos;
+
+                array_splice($this->tokens, $i, 0, $badCharTokens);
+                $numTokens += \count($badCharTokens);
+                $i += \count($badCharTokens);
             }
 
             $filePos += $tokenLen;
@@ -187,8 +204,9 @@ class Lexer
                 $this->tokens[] = [$isDocComment ? \T_DOC_COMMENT : \T_COMMENT, $comment, $line];
             } else {
                 // Invalid characters at the end of the input
-                $this->handleInvalidCharacterRange(
+                $badCharTokens = $this->handleInvalidCharacterRange(
                     $filePos, \strlen($this->code), $line, $errorHandler);
+                $this->tokens = array_merge($this->tokens, $badCharTokens);
             }
             return;
         }
@@ -273,17 +291,23 @@ class Lexer
                 $this->line += substr_count($value, "\n");
                 $this->filePos += \strlen($value);
             } else {
+                $origLine = $this->line;
+                $origFilePos = $this->filePos;
+                $this->line += substr_count($token[1], "\n");
+                $this->filePos += \strlen($token[1]);
+
                 if (\T_COMMENT === $token[0] || \T_DOC_COMMENT === $token[0]) {
                     if ($this->attributeCommentsUsed) {
                         $comment = \T_DOC_COMMENT === $token[0]
-                            ? new Comment\Doc($token[1], $this->line, $this->filePos, $this->pos)
+                            ? new Comment\Doc($token[1],
-                            : new Comment($token[1], $this->line, $this->filePos, $this->pos);
+                                $origLine, $origFilePos, $this->pos,
+                                $this->line, $this->filePos - 1, $this->pos)
+                            : new Comment($token[1],
+                                $origLine, $origFilePos, $this->pos,
+                                $this->line, $this->filePos - 1, $this->pos);
                         $startAttributes['comments'][] = $comment;
                     }
                 }
-
-                $this->line += substr_count($token[1], "\n");
-                $this->filePos += \strlen($token[1]);
                 continue;
             }
 

lib/PhpParser/Lexer/Emulative.php

@@ -7,17 +7,22 @@ use PhpParser\ErrorHandler;
 use PhpParser\Lexer;
 use PhpParser\Lexer\TokenEmulator\CoaleseEqualTokenEmulator;
 use PhpParser\Lexer\TokenEmulator\FnTokenEmulator;
+use PhpParser\Lexer\TokenEmulator\NumericLiteralSeparatorEmulator;
 use PhpParser\Lexer\TokenEmulator\TokenEmulatorInterface;
+use PhpParser\Parser\Tokens;
 
 class Emulative extends Lexer
 {
     const PHP_7_3 = '7.3.0dev';
     const PHP_7_4 = '7.4.0dev';
 
+    const T_COALESCE_EQUAL = 1007;
+    const T_FN = 1008;
+
     const FLEXIBLE_DOC_STRING_REGEX = <<<'REGEX'
 /<<<[ \t]*(['"]?)([a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*)\1\r?\n
 (?:.*\r?\n)*?
-(?<indentation>\h*)\2(?![a-zA-Z_\x80-\xff])(?<separator>(?:;?[\r\n])?)/x
+(?<indentation>\h*)\2(?![a-zA-Z0-9_\x80-\xff])(?<separator>(?:;?[\r\n])?)/x
 REGEX;
 
     /** @var mixed[] Patches used to reverse changes introduced in the code */
@@ -33,14 +38,12 @@ REGEX;
     {
         parent::__construct($options);
 
-        // prepare token emulators
         $this->tokenEmulators[] = new FnTokenEmulator();
         $this->tokenEmulators[] = new CoaleseEqualTokenEmulator();
+        $this->tokenEmulators[] = new NumericLiteralSeparatorEmulator();
 
-        // add emulated tokens here
+        $this->tokenMap[self::T_COALESCE_EQUAL] = Tokens::T_COALESCE_EQUAL;
-        foreach ($this->tokenEmulators as $emulativeToken) {
+        $this->tokenMap[self::T_FN] = Tokens::T_FN;
-            $this->tokenMap[$emulativeToken->getTokenId()] = $emulativeToken->getParserTokenId();
-        }
     }
 
     public function startLexing(string $code, ErrorHandler $errorHandler = null) {
@@ -57,14 +60,6 @@ REGEX;
         // 1. emulation of heredoc and nowdoc new syntax
         $preparedCode = $this->processHeredocNowdoc($code);
         parent::startLexing($preparedCode, $collector);
-
-        // add token emulation
-        foreach ($this->tokenEmulators as $emulativeToken) {
-            if ($emulativeToken->isEmulationNeeded($code)) {
-                $this->tokens = $emulativeToken->emulate($code, $this->tokens);
-            }
-        }
-
         $this->fixupTokens();
 
         $errors = $collector->getErrors();
@@ -74,6 +69,13 @@ REGEX;
                 $errorHandler->handleError($error);
             }
         }
+
+        // add token emulation
+        foreach ($this->tokenEmulators as $emulativeToken) {
+            if ($emulativeToken->isEmulationNeeded($code)) {
+                $this->tokens = $emulativeToken->emulate($code, $this->tokens);
+            }
+        }
     }
 
     private function isHeredocNowdocEmulationNeeded(string $code): bool

lib/PhpParser/Lexer/TokenEmulator/CoaleseEqualTokenEmulator.php

@@ -3,22 +3,9 @@
 namespace PhpParser\Lexer\TokenEmulator;
 
 use PhpParser\Lexer\Emulative;
-use PhpParser\Parser\Tokens;
 
 final class CoaleseEqualTokenEmulator implements TokenEmulatorInterface
 {
-    const T_COALESCE_EQUAL = 1007;
-
-    public function getTokenId(): int
-    {
-        return self::T_COALESCE_EQUAL;
-    }
-
-    public function getParserTokenId(): int
-    {
-        return Tokens::T_COALESCE_EQUAL;
-    }
-
     public function isEmulationNeeded(string $code) : bool
     {
         // skip version where this is supported
@@ -38,7 +25,7 @@ final class CoaleseEqualTokenEmulator implements TokenEmulatorInterface
             if (isset($tokens[$i + 1])) {
                 if ($tokens[$i][0] === T_COALESCE && $tokens[$i + 1] === '=') {
                     array_splice($tokens, $i, 2, [
-                        [self::T_COALESCE_EQUAL, '??=', $line]
+                        [Emulative::T_COALESCE_EQUAL, '??=', $line]
                     ]);
                     $c--;
                     continue;

lib/PhpParser/Lexer/TokenEmulator/FnTokenEmulator.php

@@ -3,22 +3,9 @@
 namespace PhpParser\Lexer\TokenEmulator;
 
 use PhpParser\Lexer\Emulative;
-use PhpParser\Parser\Tokens;
 
 final class FnTokenEmulator implements TokenEmulatorInterface
 {
-    const T_FN = 1008;
-
-    public function getTokenId(): int
-    {
-        return self::T_FN;
-    }
-
-    public function getParserTokenId(): int
-    {
-        return Tokens::T_FN;
-    }
-
     public function isEmulationNeeded(string $code) : bool
     {
         // skip version where this is supported
@@ -40,7 +27,7 @@ final class FnTokenEmulator implements TokenEmulatorInterface
                     continue;
                 }
 
-                $tokens[$i][0] = self::T_FN;
+                $tokens[$i][0] = Emulative::T_FN;
             }
         }
 

lib/PhpParser/Lexer/TokenEmulator/NumericLiteralSeparatorEmulator.php

@@ -0,0 +1,99 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\Lexer\TokenEmulator;
+
+use PhpParser\Lexer\Emulative;
+
+final class NumericLiteralSeparatorEmulator implements TokenEmulatorInterface
+{
+    const BIN = '(?:0b[01]+(?:_[01]+)*)';
+    const HEX = '(?:0x[0-9a-f]+(?:_[0-9a-f]+)*)';
+    const DEC = '(?:[0-9]+(?:_[0-9]+)*)';
+    const SIMPLE_FLOAT = '(?:' . self::DEC . '\.' . self::DEC . '?|\.' . self::DEC . ')';
+    const EXP = '(?:e[+-]?' . self::DEC . ')';
+    const FLOAT = '(?:' . self::SIMPLE_FLOAT . self::EXP . '?|' . self::DEC . self::EXP . ')';
+    const NUMBER = '~' . self::FLOAT . '|' . self::BIN . '|' . self::HEX . '|' . self::DEC . '~iA';
+
+    public function isEmulationNeeded(string $code) : bool
+    {
+        // skip version where this is supported
+        if (version_compare(\PHP_VERSION, Emulative::PHP_7_4, '>=')) {
+            return false;
+        }
+
+        return preg_match('~[0-9]_[0-9]~', $code)
+            || preg_match('~0x[0-9a-f]+_[0-9a-f]~i', $code);
+    }
+
+    public function emulate(string $code, array $tokens): array
+    {
+        // We need to manually iterate and manage a count because we'll change
+        // the tokens array on the way
+        $codeOffset = 0;
+        for ($i = 0, $c = count($tokens); $i < $c; ++$i) {
+            $token = $tokens[$i];
+            $tokenLen = \strlen(\is_array($token) ? $token[1] : $token);
+
+            if ($token[0] !== T_LNUMBER && $token[0] !== T_DNUMBER) {
+                $codeOffset += $tokenLen;
+                continue;
+            }
+
+            $res = preg_match(self::NUMBER, $code, $matches, 0, $codeOffset);
+            assert($res, "No number at number token position");
+
+            $match = $matches[0];
+            $matchLen = \strlen($match);
+            if ($matchLen === $tokenLen) {
+                // Original token already holds the full number.
+                $codeOffset += $tokenLen;
+                continue;
+            }
+
+            $tokenKind = $this->resolveIntegerOrFloatToken($match);
+            $newTokens = [[$tokenKind, $match, $token[2]]];
+
+            $numTokens = 1;
+            $len = $tokenLen;
+            while ($matchLen > $len) {
+                $nextToken = $tokens[$i + $numTokens];
+                $nextTokenText = \is_array($nextToken) ? $nextToken[1] : $nextToken;
+                $nextTokenLen = \strlen($nextTokenText);
+
+                $numTokens++;
+                if ($matchLen < $len + $nextTokenLen) {
+                    // Split trailing characters into a partial token.
+                    assert(is_array($nextToken), "Partial token should be an array token");
+                    $partialText = substr($nextTokenText, $matchLen - $len);
+                    $newTokens[] = [$nextToken[0], $partialText, $nextToken[2]];
+                    break;
+                }
+
+                $len += $nextTokenLen;
+            }
+
+            array_splice($tokens, $i, $numTokens, $newTokens);
+            $c -= $numTokens - \count($newTokens);
+            $codeOffset += $matchLen;
+        }
+
+        return $tokens;
+    }
+
+    private function resolveIntegerOrFloatToken(string $str): int
+    {
+        $str = str_replace('_', '', $str);
+
+        if (stripos($str, '0b') === 0) {
+            $num = bindec($str);
+        } elseif (stripos($str, '0x') === 0) {
+            $num = hexdec($str);
+        } elseif (stripos($str, '0') === 0 && ctype_digit($str)) {
+            $num = octdec($str);
+        } else {
+            $num = +$str;
+        }
+
+        return is_float($num) ? T_DNUMBER : T_LNUMBER;
+    }
+}

lib/PhpParser/Lexer/TokenEmulator/TokenEmulatorInterface.php

@@ -2,12 +2,9 @@
 
 namespace PhpParser\Lexer\TokenEmulator;
 
+/** @internal */
 interface TokenEmulatorInterface
 {
-    public function getTokenId(): int;
-
-    public function getParserTokenId(): int;
-
     public function isEmulationNeeded(string $code): bool;
 
     /**

lib/PhpParser/Node.php

@@ -95,8 +95,6 @@ interface Node
     /**
      * Gets the doc comment of the node.
      *
-     * The doc comment has to be the last comment associated with the node.
-     *
      * @return null|Comment\Doc Doc comment object or null
      */
     public function getDocComment();

lib/PhpParser/Node/Expr/ArrowFunction.php

@@ -17,7 +17,7 @@ class ArrowFunction extends Expr implements FunctionLike
     /** @var Node\Param[] */
     public $params = [];
 
-    /** @var null|Node\Identifier|Node\Name|Node\NullableType */
+    /** @var null|Node\Identifier|Node\Name|Node\NullableType|Node\UnionType */
     public $returnType;
 
     /** @var Expr */

lib/PhpParser/Node/Expr/Closure.php

@@ -16,7 +16,7 @@ class Closure extends Expr implements FunctionLike
     public $params;
     /** @var ClosureUse[] use()s */
     public $uses;
-    /** @var null|Node\Identifier|Node\Name|Node\NullableType Return type */
+    /** @var null|Node\Identifier|Node\Name|Node\NullableType|Node\UnionType Return type */
     public $returnType;
     /** @var Node\Stmt[] Statements */
     public $stmts;

lib/PhpParser/Node/FunctionLike.php

@@ -23,7 +23,7 @@ interface FunctionLike extends Node
     /**
      * Get the declared return type or null
      *
-     * @return null|Identifier|Node\Name|Node\NullableType
+     * @return null|Identifier|Node\Name|Node\NullableType|Node\UnionType
      */
     public function getReturnType();
 

lib/PhpParser/Node/Param.php

@@ -6,7 +6,7 @@ use PhpParser\NodeAbstract;
 
 class Param extends NodeAbstract
 {
-    /** @var null|Identifier|Name|NullableType Type declaration */
+    /** @var null|Identifier|Name|NullableType|UnionType Type declaration */
     public $type;
     /** @var bool Whether parameter is passed by reference */
     public $byRef;
@@ -16,20 +16,25 @@ class Param extends NodeAbstract
     public $var;
     /** @var null|Expr Default value */
     public $default;
+    /** @var int */
+    public $flags;
 
     /**
      * Constructs a parameter node.
      *
-     * @param Expr\Variable|Expr\Error                 $var        Parameter variable
+     * @param Expr\Variable|Expr\Error                           $var        Parameter variable
-     * @param null|Expr                                $default    Default value
+     * @param null|Expr                                          $default    Default value
-     * @param null|string|Identifier|Name|NullableType $type       Type declaration
+     * @param null|string|Identifier|Name|NullableType|UnionType $type       Type declaration
-     * @param bool                                     $byRef      Whether is passed by reference
+     * @param bool                                               $byRef      Whether is passed by reference
-     * @param bool                                     $variadic   Whether this is a variadic argument
+     * @param bool                                               $variadic   Whether this is a variadic argument
-     * @param array                                    $attributes Additional attributes
+     * @param array                                              $flags      Optional visibility flags
+     * @param array                                              $attributes Additional attributes
      */
     public function __construct(
         $var, Expr $default = null, $type = null,
-        bool $byRef = false, bool $variadic = false, array $attributes = []
+        bool $byRef = false, bool $variadic = false,
+        array $attributes = [],
+        int $flags = 0
     ) {
         $this->attributes = $attributes;
         $this->type = \is_string($type) ? new Identifier($type) : $type;
@@ -37,10 +42,11 @@ class Param extends NodeAbstract
         $this->variadic = $variadic;
         $this->var = $var;
         $this->default = $default;
+        $this->flags = $flags;
     }
 
     public function getSubNodeNames() : array {
-        return ['type', 'byRef', 'variadic', 'var', 'default'];
+        return ['flags', 'type', 'byRef', 'variadic', 'var', 'default'];
     }
 
     public function getType() : string {

lib/PhpParser/Node/Scalar/DNumber.php

@@ -34,6 +34,8 @@ class DNumber extends Scalar
      * @return float The parsed number
      */
     public static function parse(string $str) : float {
+        $str = str_replace('_', '', $str);
+
         // if string contains any of .eE just cast it to float
         if (false !== strpbrk($str, '.eE')) {
             return (float) $str;

lib/PhpParser/Node/Scalar/LNumber.php

@@ -41,6 +41,8 @@ class LNumber extends Scalar
      * @return LNumber The constructed LNumber, including kind attribute
      */
     public static function fromString(string $str, array $attributes = [], bool $allowInvalidOctal = false) : LNumber {
+        $str = str_replace('_', '', $str);
+
         if ('0' !== $str[0] || '0' === $str) {
             $attributes['kind'] = LNumber::KIND_DEC;
             return new LNumber((int) $str, $attributes);

lib/PhpParser/Node/Scalar/String_.php

@@ -100,7 +100,7 @@ class String_ extends Scalar
                 if (isset(self::$replacements[$str])) {
                     return self::$replacements[$str];
                 } elseif ('x' === $str[0] || 'X' === $str[0]) {
-                    return chr(hexdec($str));
+                    return chr(hexdec(substr($str, 1)));
                 } elseif ('u' === $str[0]) {
                     return self::codePointToUtf8(hexdec($matches[2]));
                 } else {
@@ -134,7 +134,7 @@ class String_ extends Scalar
         }
         throw new Error('Invalid UTF-8 codepoint escape sequence: Codepoint too large');
     }
-    
+
     public function getType() : string {
         return 'Scalar_String';
     }

lib/PhpParser/Node/Stmt/Catch_.php

@@ -9,7 +9,7 @@ class Catch_ extends Node\Stmt
 {
     /** @var Node\Name[] Types of exceptions to catch */
     public $types;
-    /** @var Expr\Variable Variable for exception */
+    /** @var Expr\Variable|null Variable for exception */
     public $var;
     /** @var Node\Stmt[] Statements */
     public $stmts;
@@ -17,13 +17,13 @@ class Catch_ extends Node\Stmt
     /**
      * Constructs a catch node.
      *
-     * @param Node\Name[]   $types      Types of exceptions to catch
+     * @param Node\Name[]           $types      Types of exceptions to catch
-     * @param Expr\Variable $var        Variable for exception
+     * @param Expr\Variable|null    $var        Variable for exception
-     * @param Node\Stmt[]   $stmts      Statements
+     * @param Node\Stmt[]           $stmts      Statements
-     * @param array         $attributes Additional attributes
+     * @param array                 $attributes Additional attributes
      */
     public function __construct(
-        array $types, Expr\Variable $var, array $stmts = [], array $attributes = []
+        array $types, Expr\Variable $var = null, array $stmts = [], array $attributes = []
     ) {
         $this->attributes = $attributes;
         $this->types = $types;
@@ -34,7 +34,7 @@ class Catch_ extends Node\Stmt
     public function getSubNodeNames() : array {
         return ['types', 'var', 'stmts'];
     }
-    
+
     public function getType() : string {
         return 'Stmt_Catch';
     }

lib/PhpParser/Node/Stmt/ClassLike.php

@@ -14,6 +14,65 @@ abstract class ClassLike extends Node\Stmt
     /** @var Node\Stmt[] Statements */
     public $stmts;
 
+    /**
+     * @return TraitUse[]
+     */
+    public function getTraitUses() : array {
+        $traitUses = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof TraitUse) {
+                $traitUses[] = $stmt;
+            }
+        }
+        return $traitUses;
+    }
+
+    /**
+     * @return ClassConst[]
+     */
+    public function getConstants() : array {
+        $constants = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof ClassConst) {
+                $constants[] = $stmt;
+            }
+        }
+        return $constants;
+    }
+
+    /**
+     * @return Property[]
+     */
+    public function getProperties() : array {
+        $properties = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof Property) {
+                $properties[] = $stmt;
+            }
+        }
+        return $properties;
+    }
+
+    /**
+     * Gets property with the given name defined directly in this class/interface/trait.
+     *
+     * @param string $name Name of the property
+     *
+     * @return Property|null Property node or null if the property does not exist
+     */
+    public function getProperty(string $name) {
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof Property) {
+                foreach ($stmt->props as $prop) {
+                    if ($prop instanceof PropertyProperty && $name === $prop->name->toString()) {
+                        return $stmt;
+                    }
+                }
+            }
+        }
+        return null;
+    }
+
     /**
      * Gets all methods defined directly in this class/interface/trait
      *

lib/PhpParser/Node/Stmt/ClassMethod.php

@@ -15,7 +15,7 @@ class ClassMethod extends Node\Stmt implements FunctionLike
     public $name;
     /** @var Node\Param[] Parameters */
     public $params;
-    /** @var null|Node\Identifier|Node\Name|Node\NullableType Return type */
+    /** @var null|Node\Identifier|Node\Name|Node\NullableType|Node\UnionType Return type */
     public $returnType;
     /** @var Node\Stmt[]|null Statements */
     public $stmts;

lib/PhpParser/Node/Stmt/Class_.php

@@ -98,7 +98,7 @@ class Class_ extends ClassLike
             throw new Error('Cannot use the final modifier on an abstract class member');
         }
     }
-    
+
     public function getType() : string {
         return 'Stmt_Class';
     }

lib/PhpParser/Node/Stmt/Function_.php

@@ -16,7 +16,7 @@ class Function_ extends Node\Stmt implements FunctionLike
     public $name;
     /** @var Node\Param[] Parameters */
     public $params;
-    /** @var null|Node\Identifier|Node\Name|Node\NullableType Return type */
+    /** @var null|Node\Identifier|Node\Name|Node\NullableType|Node\UnionType Return type */
     public $returnType;
     /** @var Node\Stmt[] Statements */
     public $stmts;

lib/PhpParser/Node/Stmt/Interface_.php

@@ -28,7 +28,7 @@ class Interface_ extends ClassLike
     public function getSubNodeNames() : array {
         return ['name', 'extends', 'stmts'];
     }
-    
+
     public function getType() : string {
         return 'Stmt_Interface';
     }

lib/PhpParser/Node/Stmt/Property.php

@@ -6,6 +6,7 @@ use PhpParser\Node;
 use PhpParser\Node\Identifier;
 use PhpParser\Node\Name;
 use PhpParser\Node\NullableType;
+use PhpParser\Node\UnionType;
 
 class Property extends Node\Stmt
 {
@@ -13,16 +14,16 @@ class Property extends Node\Stmt
     public $flags;
     /** @var PropertyProperty[] Properties */
     public $props;
-    /** @var null|Identifier|Name|NullableType Type declaration */
+    /** @var null|Identifier|Name|NullableType|UnionType Type declaration */
     public $type;
 
     /**
      * Constructs a class property list node.
      *
-     * @param int                                      $flags      Modifiers
+     * @param int                                                $flags      Modifiers
-     * @param PropertyProperty[]                       $props      Properties
+     * @param PropertyProperty[]                                 $props      Properties
-     * @param array                                    $attributes Additional attributes
+     * @param array                                              $attributes Additional attributes
-     * @param null|string|Identifier|Name|NullableType $type       Type declaration
+     * @param null|string|Identifier|Name|NullableType|UnionType $type       Type declaration
      */
     public function __construct(int $flags, array $props, array $attributes = [], $type = null) {
         $this->attributes = $attributes;

lib/PhpParser/Node/Stmt/Trait_.php

@@ -23,7 +23,7 @@ class Trait_ extends ClassLike
     public function getSubNodeNames() : array {
         return ['name', 'stmts'];
     }
-    
+
     public function getType() : string {
         return 'Stmt_Trait';
     }

lib/PhpParser/Node/Stmt/TryCatch.php

@@ -18,7 +18,7 @@ class TryCatch extends Node\Stmt
      *
      * @param Node\Stmt[]   $stmts      Statements
      * @param Catch_[]      $catches    Catches
-     * @param null|Finally_ $finally    Optionaly finally node
+     * @param null|Finally_ $finally    Optional finally node
      * @param array         $attributes Additional attributes
      */
     public function __construct(array $stmts, array $catches, Finally_ $finally = null, array $attributes = []) {
@@ -31,7 +31,7 @@ class TryCatch extends Node\Stmt
     public function getSubNodeNames() : array {
         return ['stmts', 'catches', 'finally'];
     }
-    
+
     public function getType() : string {
         return 'Stmt_TryCatch';
     }

lib/PhpParser/Node/UnionType.php

@@ -0,0 +1,30 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\Node;
+
+use PhpParser\NodeAbstract;
+
+class UnionType extends NodeAbstract
+{
+    /** @var (Identifier|Name)[] Types */
+    public $types;
+
+    /**
+     * Constructs a union type.
+     *
+     * @param (Identifier|Name)[] $types      Types
+     * @param array               $attributes Additional attributes
+     */
+    public function __construct(array $types, array $attributes = []) {
+        $this->attributes = $attributes;
+        $this->types = $types;
+    }
+
+    public function getSubNodeNames() : array {
+        return ['types'];
+    }
+    
+    public function getType() : string {
+        return 'UnionType';
+    }
+}

lib/PhpParser/NodeAbstract.php

@@ -108,22 +108,18 @@ abstract class NodeAbstract implements Node, \JsonSerializable
     /**
      * Gets the doc comment of the node.
      *
-     * The doc comment has to be the last comment associated with the node.
-     *
      * @return null|Comment\Doc Doc comment object or null
      */
     public function getDocComment() {
         $comments = $this->getComments();
-        if (!$comments) {
+        for ($i = count($comments) - 1; $i >= 0; $i--) {
-            return null;
+            $comment = $comments[$i];
+            if ($comment instanceof Comment\Doc) {
+                return $comment;
+            }
         }
 
-        $lastComment = $comments[count($comments) - 1];
+        return null;
-        if (!$lastComment instanceof Comment\Doc) {
-            return null;
-        }
-
-        return $lastComment;
     }
 
     /**
@@ -135,16 +131,17 @@ abstract class NodeAbstract implements Node, \JsonSerializable
      */
     public function setDocComment(Comment\Doc $docComment) {
         $comments = $this->getComments();
-
+        for ($i = count($comments) - 1; $i >= 0; $i--) {
-        $numComments = count($comments);
+            if ($comments[$i] instanceof Comment\Doc) {
-        if ($numComments > 0 && $comments[$numComments - 1] instanceof Comment\Doc) {
+                // Replace existing doc comment.
-            // Replace existing doc comment
+                $comments[$i] = $docComment;
-            $comments[$numComments - 1] = $docComment;
+                $this->setAttribute('comments', $comments);
-        } else {
+                return;
-            // Append new comment
+            }
-            $comments[] = $docComment;
         }
 
+        // Append new doc comment.
+        $comments[] = $docComment;
         $this->setAttribute('comments', $comments);
     }
 

lib/PhpParser/NodeVisitor/NameResolver.php

@@ -91,6 +91,7 @@ class NameResolver extends NodeVisitorAbstract
             $this->resolveSignature($node);
         } elseif ($node instanceof Stmt\ClassMethod
                   || $node instanceof Expr\Closure
+                  || $node instanceof Expr\ArrowFunction
         ) {
             $this->resolveSignature($node);
         } elseif ($node instanceof Stmt\Property) {
@@ -161,12 +162,18 @@ class NameResolver extends NodeVisitorAbstract
     }
 
     private function resolveType($node) {
+        if ($node instanceof Name) {
+            return $this->resolveClassName($node);
+        }
         if ($node instanceof Node\NullableType) {
             $node->type = $this->resolveType($node->type);
             return $node;
         }
-        if ($node instanceof Name) {
+        if ($node instanceof Node\UnionType) {
-            return $this->resolveClassName($node);
+            foreach ($node->types as &$type) {
+                $type = $this->resolveType($type);
+            }
+            return $node;
         }
         return $node;
     }

lib/PhpParser/Parser/Php5.php

@@ -944,7 +944,7 @@ class Php5 extends \PhpParser\ParserAbstract
                  $this->semValue = array();
             },
             4 => function ($stackPos) {
-                 $startAttributes = $this->lookaheadStartAttributes; if (isset($startAttributes['comments'])) { $nop = new Stmt\Nop($this->createZeroLengthAttributes($startAttributes)); } else { $nop = null; };
+                 $startAttributes = $this->lookaheadStartAttributes; if (isset($startAttributes['comments'])) { $nop = new Stmt\Nop($this->createCommentNopAttributes($startAttributes['comments'])); } else { $nop = null; };
             if ($nop !== null) { $this->semStack[$stackPos-(1-1)][] = $nop; } $this->semValue = $this->semStack[$stackPos-(1-1)];
             },
             5 => function ($stackPos) {
@@ -1317,7 +1317,7 @@ class Php5 extends \PhpParser\ParserAbstract
                  $this->semValue = array();
             },
             126 => function ($stackPos) {
-                 $startAttributes = $this->lookaheadStartAttributes; if (isset($startAttributes['comments'])) { $nop = new Stmt\Nop($this->createZeroLengthAttributes($startAttributes)); } else { $nop = null; };
+                 $startAttributes = $this->lookaheadStartAttributes; if (isset($startAttributes['comments'])) { $nop = new Stmt\Nop($this->createCommentNopAttributes($startAttributes['comments'])); } else { $nop = null; };
             if ($nop !== null) { $this->semStack[$stackPos-(1-1)][] = $nop; } $this->semValue = $this->semStack[$stackPos-(1-1)];
             },
             127 => function ($stackPos) {
@@ -1715,7 +1715,7 @@ class Php5 extends \PhpParser\ParserAbstract
                  $this->semValue = array();
             },
             255 => function ($stackPos) {
-                 $startAttributes = $this->lookaheadStartAttributes; if (isset($startAttributes['comments'])) { $nop = new Stmt\Nop($this->createZeroLengthAttributes($startAttributes)); } else { $nop = null; };
+                 $startAttributes = $this->lookaheadStartAttributes; if (isset($startAttributes['comments'])) { $nop = new Stmt\Nop($this->createCommentNopAttributes($startAttributes['comments'])); } else { $nop = null; };
             if ($nop !== null) { $this->semStack[$stackPos-(1-1)][] = $nop; } $this->semValue = $this->semStack[$stackPos-(1-1)];
             },
             256 => function ($stackPos) {

lib/PhpParser/ParserAbstract.php

@@ -648,7 +648,7 @@ abstract class ParserAbstract implements Parser
     }
 
     protected function handleBuiltinTypes(Name $name) {
-        $scalarTypes = [
+        $builtinTypes = [
             'bool'     => true,
             'int'      => true,
             'float'    => true,
@@ -656,6 +656,9 @@ abstract class ParserAbstract implements Parser
             'iterable' => true,
             'void'     => true,
             'object'   => true,
+            'null'     => true,
+            'false'    => true,
+            'mixed'    => true,
         ];
 
         if (!$name->isUnqualified()) {
@@ -663,7 +666,7 @@ abstract class ParserAbstract implements Parser
         }
 
         $lowerName = $name->toLowerString();
-        if (!isset($scalarTypes[$lowerName])) {
+        if (!isset($builtinTypes[$lowerName])) {
             return $name;
         }
 
@@ -840,21 +843,29 @@ abstract class ParserAbstract implements Parser
     }
 
     /**
-     * Create attributes for a zero-length node with the given start attributes.
+     * Create attributes for a zero-length common-capturing nop.
      *
-     * @param array $startAttributes
+     * @param Comment[] $comments
      * @return array
      */
-    protected function createZeroLengthAttributes(array $startAttributes) {
+    protected function createCommentNopAttributes(array $comments) {
-        $attributes = $startAttributes;
+        $comment = $comments[count($comments) - 1];
-        if (isset($startAttributes['startLine'])) {
+        $commentEndLine = $comment->getEndLine();
-            $attributes['endLine'] = $startAttributes['startLine'];
+        $commentEndFilePos = $comment->getEndFilePos();
+        $commentEndTokenPos = $comment->getEndTokenPos();
+
+        $attributes = ['comments' => $comments];
+        if (-1 !== $commentEndLine) {
+            $attributes['startLine'] = $commentEndLine;
+            $attributes['endLine'] = $commentEndLine;
         }
-        if (isset($startAttributes['startTokenPos'])) {
+        if (-1 !== $commentEndFilePos) {
-            $attributes['endTokenPos'] = $startAttributes['startTokenPos'] - 1;
+            $attributes['startFilePos'] = $commentEndFilePos + 1;
+            $attributes['endFilePos'] = $commentEndFilePos;
         }
-        if (isset($startAttributes['startFilePos'])) {
+        if (-1 !== $commentEndTokenPos) {
-            $attributes['endFilePos'] = $startAttributes['startFilePos'] - 1;
+            $attributes['startTokenPos'] = $commentEndTokenPos + 1;
+            $attributes['endTokenPos'] = $commentEndTokenPos;
         }
         return $attributes;
     }

lib/PhpParser/PrettyPrinter/Standard.php

@@ -18,7 +18,8 @@ class Standard extends PrettyPrinterAbstract
     // Special nodes
 
     protected function pParam(Node\Param $node) {
-        return ($node->type ? $this->p($node->type) . ' ' : '')
+        return ($this->pModifiers($node->flags))
+             . ($node->type ? $this->p($node->type) . ' ' : '')
              . ($node->byRef ? '&' : '')
              . ($node->variadic ? '...' : '')
              . $this->p($node->var)
@@ -37,6 +38,10 @@ class Standard extends PrettyPrinterAbstract
         return '?' . $this->p($node->type);
     }
 
+    protected function pUnionType(Node\UnionType $node) {
+        return $this->pImplode($node->types, '|');
+    }
+
     protected function pIdentifier(Node\Identifier $node) {
         return $node->name;
     }
@@ -159,8 +164,13 @@ class Standard extends PrettyPrinterAbstract
             return (string) $node->value;
         }
 
-        $sign = $node->value < 0 ? '-' : '';
+        if ($node->value < 0) {
-        $str = (string) $node->value;
+            $sign = '-';
+            $str = (string) -$node->value;
+        } else {
+            $sign = '';
+            $str = (string) $node->value;
+        }
         switch ($kind) {
             case Scalar\LNumber::KIND_BIN:
                 return $sign . '0b' . base_convert($str, 10, 2);
@@ -716,7 +726,7 @@ class Standard extends PrettyPrinterAbstract
     protected function pStmt_ClassMethod(Stmt\ClassMethod $node) {
         return $this->pModifiers($node->flags)
              . 'function ' . ($node->byRef ? '&' : '') . $node->name
-             . '(' . $this->pCommaSeparated($node->params) . ')'
+             . '(' . $this->pMaybeMultiline($node->params) . ')'
              . (null !== $node->returnType ? ' : ' . $this->p($node->returnType) : '')
              . (null !== $node->stmts
                 ? $this->nl . '{' . $this->pStmts($node->stmts) . $this->nl . '}'
@@ -808,8 +818,8 @@ class Standard extends PrettyPrinterAbstract
     }
 
     protected function pStmt_Catch(Stmt\Catch_ $node) {
-        return 'catch (' . $this->pImplode($node->types, '|') . ' '
+        return 'catch (' . $this->pImplode($node->types, '|')
-             . $this->p($node->var)
+             . ($node->var !== null ? ' ' . $this->p($node->var) : '')
              . ') {' . $this->pStmts($node->stmts) . $this->nl . '}';
     }
 

lib/PhpParser/PrettyPrinterAbstract.php

@@ -756,20 +756,13 @@ abstract class PrettyPrinterAbstract
                 $itemEndPos = $origArrItem->getEndTokenPos();
                 \assert($itemStartPos >= 0 && $itemEndPos >= 0);
 
-                if ($itemEndPos < $itemStartPos) {
-                    // End can be before start for Nop nodes, because offsets refer to non-whitespace
-                    // locations, which for an "empty" node might result in an inverted order.
-                    assert($origArrItem instanceof Stmt\Nop);
-                    continue;
-                }
-
                 $origIndentLevel = $this->indentLevel;
                 $lastElemIndentLevel = $this->origTokens->getIndentationBefore($itemStartPos) + $indentAdjustment;
                 $this->setIndentLevel($lastElemIndentLevel);
 
                 $comments = $arrItem->getComments();
                 $origComments = $origArrItem->getComments();
-                $commentStartPos = $origComments ? $origComments[0]->getTokenPos() : $itemStartPos;
+                $commentStartPos = $origComments ? $origComments[0]->getStartTokenPos() : $itemStartPos;
                 \assert($commentStartPos >= 0);
 
                 $commentsChanged = $comments !== $origComments;
@@ -1227,6 +1220,7 @@ abstract class PrettyPrinterAbstract
             'Param->type' => $stripRight,
             'Param->default' => $stripEquals,
             'Stmt_Break->num' => $stripBoth,
+            'Stmt_Catch->var' => $stripLeft,
             'Stmt_ClassMethod->returnType' => $stripColon,
             'Stmt_Class->extends' => ['left' => \T_EXTENDS],
             'Expr_PrintableNewAnonClass->extends' => ['left' => \T_EXTENDS],
@@ -1264,6 +1258,7 @@ abstract class PrettyPrinterAbstract
             'Param->type' => [null, false, null, ' '],
             'Param->default' => [null, false, ' = ', null],
             'Stmt_Break->num' => [\T_BREAK, false, ' ', null],
+            'Stmt_Catch->var' => [null, false, ' ', null],
             'Stmt_ClassMethod->returnType' => [')', false, ' : ', null],
             'Stmt_Class->extends' => [null, false, ' extends ', null],
             'Expr_PrintableNewAnonClass->extends' => [null, ' extends ', null],
@@ -1295,6 +1290,7 @@ abstract class PrettyPrinterAbstract
             //'Expr_ShellExec->parts' => '', // TODO These need to be treated more carefully
             //'Scalar_Encapsed->parts' => '',
             'Stmt_Catch->types' => '|',
+            'UnionType->types' => '|',
             'Stmt_If->elseifs' => ' ',
             'Stmt_TryCatch->catches' => ' ',
 
@@ -1396,6 +1392,7 @@ abstract class PrettyPrinterAbstract
              * Stmt_TraitUseAdaptation_Precedence->insteadof
              * Stmt_Unset->vars
              * Stmt_Use->uses
+             * UnionType->types
              */
 
             /* TODO
@@ -1418,6 +1415,7 @@ abstract class PrettyPrinterAbstract
             'Stmt_ClassMethod->flags' => \T_FUNCTION,
             'Stmt_Class->flags' => \T_CLASS,
             'Stmt_Property->flags' => \T_VARIABLE,
+            'Param->flags' => \T_VARIABLE,
             //'Stmt_TraitUseAdaptation_Alias->newModifier' => 0, // TODO
         ];
 

test/PhpParser/Builder/InterfaceTest.php

@@ -9,11 +9,8 @@ use PhpParser\Node\Stmt;
 
 class InterfaceTest extends \PHPUnit\Framework\TestCase
 {
-    /** @var Interface_ */
+    protected function createInterfaceBuilder() {
-    protected $builder;
+        return new Interface_('Contract');
-
-    protected function setUp() {
-        $this->builder = new Interface_('Contract');
     }
 
     private function dump($node) {
@@ -22,13 +19,14 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     }
 
     public function testEmpty() {
-        $contract = $this->builder->getNode();
+        $contract = $this->createInterfaceBuilder()->getNode();
         $this->assertInstanceOf(Stmt\Interface_::class, $contract);
         $this->assertEquals(new Node\Identifier('Contract'), $contract->name);
     }
 
     public function testExtending() {
-        $contract = $this->builder->extend('Space\Root1', 'Root2')->getNode();
+        $contract = $this->createInterfaceBuilder()
+            ->extend('Space\Root1', 'Root2')->getNode();
         $this->assertEquals(
             new Stmt\Interface_('Contract', [
                 'extends' => [
@@ -41,7 +39,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
 
     public function testAddMethod() {
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder->addStmt($method)->getNode();
+        $contract = $this->createInterfaceBuilder()->addStmt($method)->getNode();
         $this->assertSame([$method], $contract->stmts);
     }
 
@@ -49,7 +47,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
         $const = new Stmt\ClassConst([
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458.0))
         ]);
-        $contract = $this->builder->addStmt($const)->getNode();
+        $contract = $this->createInterfaceBuilder()->addStmt($const)->getNode();
         $this->assertSame(299792458.0, $contract->stmts[0]->consts[0]->value->value);
     }
 
@@ -58,7 +56,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458))
         ]);
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder
+        $contract = $this->createInterfaceBuilder()
             ->addStmt($method)
             ->addStmt($const)
             ->getNode()
@@ -69,7 +67,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     }
 
     public function testDocComment() {
-        $node = $this->builder
+        $node = $this->createInterfaceBuilder()
             ->setDocComment('/** Test */')
             ->getNode();
 
@@ -81,7 +79,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     public function testInvalidStmtError() {
         $this->expectException(\LogicException::class);
         $this->expectExceptionMessage('Unexpected node of type "Stmt_PropertyProperty"');
-        $this->builder->addStmt(new Stmt\PropertyProperty('invalid'));
+        $this->createInterfaceBuilder()->addStmt(new Stmt\PropertyProperty('invalid'));
     }
 
     public function testFullFunctional() {
@@ -89,7 +87,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458))
         ]);
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder
+        $contract = $this->createInterfaceBuilder()
             ->addStmt($method)
             ->addStmt($const)
             ->getNode()

test/PhpParser/Builder/ParamTest.php

@@ -80,6 +80,8 @@ class ParamTest extends \PHPUnit\Framework\TestCase
 
     /**
      * @dataProvider provideTestTypes
+     * @dataProvider provideTestNullableTypes
+     * @dataProvider provideTestUnionTypes
      */
     public function testTypes($typeHint, $expectedType) {
         $node = $this->createParamBuilder('test')
@@ -111,12 +113,18 @@ class ParamTest extends \PHPUnit\Framework\TestCase
             ['object', new Node\Identifier('object')],
             ['Array', new Node\Identifier('array')],
             ['CALLABLE', new Node\Identifier('callable')],
+            ['mixed', new Node\Identifier('mixed')],
             ['Some\Class', new Node\Name('Some\Class')],
             ['\Foo', new Node\Name\FullyQualified('Foo')],
             ['self', new Node\Name('self')],
+            [new Node\Name('Some\Class'), new Node\Name('Some\Class')],
+        ];
+    }
+
+    public function provideTestNullableTypes() {
+        return [
             ['?array', new Node\NullableType(new Node\Identifier('array'))],
             ['?Some\Class', new Node\NullableType(new Node\Name('Some\Class'))],
-            [new Node\Name('Some\Class'), new Node\Name('Some\Class')],
             [
                 new Node\NullableType(new Node\Identifier('int')),
                 new Node\NullableType(new Node\Identifier('int'))
@@ -128,6 +136,33 @@ class ParamTest extends \PHPUnit\Framework\TestCase
         ];
     }
 
+    public function provideTestUnionTypes() {
+        return [
+            [
+                new Node\UnionType([
+                    new Node\Name('Some\Class'),
+                    new Node\Identifier('array'),
+                ]),
+                new Node\UnionType([
+                    new Node\Name('Some\Class'),
+                    new Node\Identifier('array'),
+                ]),
+            ],
+            [
+                new Node\UnionType([
+                    new Node\Identifier('self'),
+                    new Node\Identifier('array'),
+                    new Node\Name\FullyQualified('Foo')
+                ]),
+                new Node\UnionType([
+                    new Node\Identifier('self'),
+                    new Node\Identifier('array'),
+                    new Node\Name\FullyQualified('Foo')
+                ]),
+            ],
+        ];
+    }
+
     public function testVoidTypeError() {
         $this->expectException(\LogicException::class);
         $this->expectExceptionMessage('Parameter type cannot be void');
@@ -136,7 +171,7 @@ class ParamTest extends \PHPUnit\Framework\TestCase
 
     public function testInvalidTypeError() {
         $this->expectException(\LogicException::class);
-        $this->expectExceptionMessage('Type must be a string, or an instance of Name, Identifier or NullableType');
+        $this->expectExceptionMessage('Type must be a string, or an instance of Name, Identifier, NullableType or UnionType');
         $this->createParamBuilder('test')->setType(new \stdClass);
     }
 

test/PhpParser/Builder/TraitTest.php

@@ -5,6 +5,12 @@ namespace PhpParser\Builder;
 use PhpParser\Comment;
 use PhpParser\Node\Name;
 use PhpParser\Node\Stmt;
+use PhpParser\Node\Stmt\Class_;
+use PhpParser\Node\Stmt\ClassConst;
+use PhpParser\Node\Stmt\ClassMethod;
+use PhpParser\Node\Stmt\Property;
+use PhpParser\Node\Stmt\PropertyProperty;
+use PhpParser\Node\Stmt\TraitUse;
 
 class TraitTest extends \PHPUnit\Framework\TestCase
 {
@@ -43,4 +49,43 @@ class TraitTest extends \PHPUnit\Framework\TestCase
             ->addStmt(new Stmt\Echo_([]))
         ;
     }
+
+    public function testGetMethods() {
+        $methods = [
+            new ClassMethod('foo'),
+            new ClassMethod('bar'),
+            new ClassMethod('fooBar'),
+        ];
+        $trait = new Stmt\Trait_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $methods[0],
+                new ClassConst([]),
+                $methods[1],
+                new Property(0, []),
+                $methods[2],
+            ]
+        ]);
+
+        $this->assertSame($methods, $trait->getMethods());
+    }
+
+    public function testGetProperties()
+    {
+        $properties = [
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo')]),
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('bar')]),
+        ];
+        $trait = new Stmt\Trait_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $properties[0],
+                new ClassConst([]),
+                $properties[1],
+                new ClassMethod('fooBar'),
+            ]
+        ]);
+
+        $this->assertSame($properties, $trait->getProperties());
+    }
 }

test/PhpParser/CommentTest.php

@@ -4,14 +4,21 @@ namespace PhpParser;
 
 class CommentTest extends \PHPUnit\Framework\TestCase
 {
-    public function testGetSet() {
+    public function testGetters() {
-        $comment = new Comment('/* Some comment */', 1, 10, 2);
+        $comment = new Comment('/* Some comment */',
+            1, 10, 2, 1, 27, 2);
 
         $this->assertSame('/* Some comment */', $comment->getText());
         $this->assertSame('/* Some comment */', (string) $comment);
         $this->assertSame(1, $comment->getLine());
         $this->assertSame(10, $comment->getFilePos());
         $this->assertSame(2, $comment->getTokenPos());
+        $this->assertSame(1, $comment->getStartLine());
+        $this->assertSame(10, $comment->getStartFilePos());
+        $this->assertSame(2, $comment->getStartTokenPos());
+        $this->assertSame(1, $comment->getEndLine());
+        $this->assertSame(27, $comment->getEndFilePos());
+        $this->assertSame(2, $comment->getEndTokenPos());
     }
 
     /**

test/PhpParser/Lexer/EmulativeTest.php

@@ -123,10 +123,6 @@ class EmulativeTest extends LexerTest
 
     public function provideTestLexNewFeatures() {
         return [
-            // PHP 7.4
-            ['??=', [
-                [Tokens::T_COALESCE_EQUAL, '??='],
-            ]],
             ['yield from', [
                 [Tokens::T_YIELD_FROM, 'yield from'],
             ]],
@@ -169,7 +165,7 @@ class EmulativeTest extends LexerTest
                 [ord(';'), ';'],
             ]],
 
-            // Flexible heredoc/nowdoc
+            // PHP 7.3: Flexible heredoc/nowdoc
             ["<<<LABEL\nLABEL,", [
                 [Tokens::T_START_HEREDOC, "<<<LABEL\n"],
                 [Tokens::T_END_HEREDOC, "LABEL"],
@@ -205,6 +201,58 @@ class EmulativeTest extends LexerTest
                 [Tokens::T_END_HEREDOC, "    LABEL"],
                 [Tokens::T_STRING, "LABEL"],
             ]],
+
+            // PHP 7.4: Null coalesce equal
+            ['??=', [
+                [Tokens::T_COALESCE_EQUAL, '??='],
+            ]],
+
+            // PHP 7.4: Number literal separator
+            ['1_000', [
+                [Tokens::T_LNUMBER, '1_000'],
+            ]],
+            ['0xCAFE_F00D', [
+                [Tokens::T_LNUMBER, '0xCAFE_F00D'],
+            ]],
+            ['0b0101_1111', [
+                [Tokens::T_LNUMBER, '0b0101_1111'],
+            ]],
+            ['0137_041', [
+                [Tokens::T_LNUMBER, '0137_041'],
+            ]],
+            ['1_000.0', [
+                [Tokens::T_DNUMBER, '1_000.0'],
+            ]],
+            ['1_0.0', [
+                [Tokens::T_DNUMBER, '1_0.0']
+            ]],
+            ['1_000_000_000.0', [
+                [Tokens::T_DNUMBER, '1_000_000_000.0']
+            ]],
+            ['0e1_0', [
+                [Tokens::T_DNUMBER, '0e1_0']
+            ]],
+            ['1_0e+10', [
+                [Tokens::T_DNUMBER, '1_0e+10']
+            ]],
+            ['1_0e-10', [
+                [Tokens::T_DNUMBER, '1_0e-10']
+            ]],
+            ['0b1011010101001010_110101010010_10101101010101_0101101011001_110111100', [
+                [Tokens::T_DNUMBER, '0b1011010101001010_110101010010_10101101010101_0101101011001_110111100'],
+            ]],
+            ['0xFFFF_FFFF_FFFF_FFFF', [
+                [Tokens::T_DNUMBER, '0xFFFF_FFFF_FFFF_FFFF'],
+            ]],
+            ['1_000+1', [
+                [Tokens::T_LNUMBER, '1_000'],
+                [ord('+'), '+'],
+                [Tokens::T_LNUMBER, '1'],
+            ]],
+            ['1_0abc', [
+                [Tokens::T_LNUMBER, '1_0'],
+                [Tokens::T_STRING, 'abc'],
+            ]],
         ];
     }
 }

test/PhpParser/LexerTest.php

@@ -103,7 +103,9 @@ class LexerTest extends \PHPUnit\Framework\TestCase
                         [
                             'startLine' => 3,
                             'comments' => [
-                                new Comment\Doc('/** doc' . "\n" . 'comment */', 2, 14, 5),
+                                new Comment\Doc('/** doc' . "\n" . 'comment */',
+                                    2, 14, 5,
+                                    3, 31, 5),
                             ]
                         ],
                         ['endLine' => 3]
@@ -120,10 +122,14 @@ class LexerTest extends \PHPUnit\Framework\TestCase
                         [
                             'startLine' => 2,
                             'comments' => [
-                                new Comment('/* comment */', 1, 6, 1),
+                                new Comment('/* comment */',
-                                new Comment('// comment' . "\n", 1, 20, 3),
+                                    1, 6, 1, 1, 18, 1),
-                                new Comment\Doc('/** docComment 1 */', 2, 31, 4),
+                                new Comment('// comment',
-                                new Comment\Doc('/** docComment 2 */', 2, 50, 5),
+                                    1, 20, 3, 1, 29, 3),
+                                new Comment\Doc('/** docComment 1 */',
+                                    2, 31, 5, 2, 49, 5),
+                                new Comment\Doc('/** docComment 2 */',
+                                    2, 50, 6, 2, 68, 6),
                             ],
                         ],
                         ['endLine' => 2]
@@ -179,11 +185,11 @@ class LexerTest extends \PHPUnit\Framework\TestCase
                     ],
                     [
                         Tokens::T_CONSTANT_ENCAPSED_STRING, '"b"',
-                        ['startTokenPos' => 5], ['endTokenPos' => 5]
+                        ['startTokenPos' => 6], ['endTokenPos' => 6]
                     ],
                     [
                         ord(';'), ';',
-                        ['startTokenPos' => 6], ['endTokenPos' => 6]
+                        ['startTokenPos' => 7], ['endTokenPos' => 7]
                     ],
                 ]
             ],
@@ -245,14 +251,17 @@ class LexerTest extends \PHPUnit\Framework\TestCase
     }
 
     public function testGetTokens() {
-        $code = '<?php "a";' . "\n" . '// foo' . "\n" . '"b";';
+        $code = '<?php "a";' . "\n" . '// foo' . "\n" . '// bar' . "\n\n" . '"b";';
         $expectedTokens = [
             [T_OPEN_TAG, '<?php ', 1],
             [T_CONSTANT_ENCAPSED_STRING, '"a"', 1],
             ';',
             [T_WHITESPACE, "\n", 1],
-            [T_COMMENT, '// foo' . "\n", 2],
+            [T_COMMENT, '// foo', 2],
-            [T_CONSTANT_ENCAPSED_STRING, '"b"', 3],
+            [T_WHITESPACE, "\n", 2],
+            [T_COMMENT, '// bar', 3],
+            [T_WHITESPACE, "\n\n", 3],
+            [T_CONSTANT_ENCAPSED_STRING, '"b"', 5],
             ';',
         ];
 

test/PhpParser/Node/Stmt/ClassTest.php

@@ -2,6 +2,8 @@
 
 namespace PhpParser\Node\Stmt;
 
+use PhpParser\Node\Scalar\String_;
+
 class ClassTest extends \PHPUnit\Framework\TestCase
 {
     public function testIsAbstract() {
@@ -20,6 +22,22 @@ class ClassTest extends \PHPUnit\Framework\TestCase
         $this->assertFalse($class->isFinal());
     }
 
+    public function testGetTraitUses() {
+        $traitUses = [
+            new TraitUse([new Trait_('foo')]),
+            new TraitUse([new Trait_('bar')]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                $traitUses[0],
+                new ClassMethod('fooBar'),
+                $traitUses[1],
+            ]
+        ]);
+
+        $this->assertSame($traitUses, $class->getTraitUses());
+    }
+
     public function testGetMethods() {
         $methods = [
             new ClassMethod('foo'),
@@ -40,6 +58,67 @@ class ClassTest extends \PHPUnit\Framework\TestCase
         $this->assertSame($methods, $class->getMethods());
     }
 
+    public function testGetConstants() {
+        $constants = [
+            new ClassConst([new \PhpParser\Node\Const_('foo', new String_('foo_value'))]),
+            new ClassConst([new \PhpParser\Node\Const_('bar', new String_('bar_value'))]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $constants[0],
+                new ClassMethod('fooBar'),
+                $constants[1],
+            ]
+        ]);
+
+        $this->assertSame($constants, $class->getConstants());
+    }
+
+    public function testGetProperties()
+    {
+        $properties = [
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo')]),
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('bar')]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $properties[0],
+                new ClassConst([]),
+                $properties[1],
+                new ClassMethod('fooBar'),
+            ]
+        ]);
+
+        $this->assertSame($properties, $class->getProperties());
+    }
+
+    public function testGetProperty() {
+        $properties = [
+            $fooProp = new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo1')]),
+            $barProp = new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('BAR1')]),
+            $fooBarProp = new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo2'), new PropertyProperty('bar2')]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $properties[0],
+                new ClassConst([]),
+                $properties[1],
+                new ClassMethod('fooBar'),
+                $properties[2],
+            ]
+        ]);
+
+        $this->assertSame($fooProp, $class->getProperty('foo1'));
+        $this->assertSame($barProp, $class->getProperty('BAR1'));
+        $this->assertSame($fooBarProp, $class->getProperty('foo2'));
+        $this->assertSame($fooBarProp, $class->getProperty('bar2'));
+        $this->assertNull($class->getProperty('bar1'));
+        $this->assertNull($class->getProperty('nonExisting'));
+    }
+
     public function testGetMethod() {
         $methodConstruct = new ClassMethod('__CONSTRUCT');
         $methodTest = new ClassMethod('test');

test/PhpParser/Node/Stmt/InterfaceTest.php

@@ -3,6 +3,7 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
+use PhpParser\Node\Scalar\String_;
 
 class InterfaceTest extends \PHPUnit\Framework\TestCase
 {
@@ -11,7 +12,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new ClassMethod('foo'),
             new ClassMethod('bar'),
         ];
-        $interface = new Class_('Foo', [
+        $interface = new Interface_('Foo', [
             'stmts' => [
                 new Node\Stmt\ClassConst([new Node\Const_('C1', new Node\Scalar\String_('C1'))]),
                 $methods[0],
@@ -23,4 +24,21 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
 
         $this->assertSame($methods, $interface->getMethods());
     }
+
+    public function testGetConstants() {
+        $constants = [
+            new ClassConst([new \PhpParser\Node\Const_('foo', new String_('foo_value'))]),
+            new ClassConst([new \PhpParser\Node\Const_('bar', new String_('bar_value'))]),
+        ];
+        $class = new Interface_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $constants[0],
+                new ClassMethod('fooBar'),
+                $constants[1],
+            ]
+        ]);
+
+        $this->assertSame($constants, $class->getConstants());
+    }
 }

test/PhpParser/NodeAbstractTest.php

@@ -34,8 +34,9 @@ class NodeAbstractTest extends \PHPUnit\Framework\TestCase
             'startFilePos' => 14,
             'endFilePos' => 15,
             'comments'  => [
-                new Comment('// Comment' . "\n"),
+                new Comment('// Comment 1' . "\n"),
                 new Comment\Doc('/** doc comment */'),
+                new Comment('// Comment 2' . "\n"),
             ],
         ];
 
@@ -79,12 +80,12 @@ class NodeAbstractTest extends \PHPUnit\Framework\TestCase
         $this->assertSame('/** doc comment */', $node->getDocComment()->getText());
         $comments = $node->getComments();
 
-        array_pop($comments); // remove doc comment
+        array_splice($comments, 1, 1, []); // remove doc comment
         $node->setAttribute('comments', $comments);
         $this->assertNull($node->getDocComment());
 
-        array_pop($comments); // remove comment
+        // Remove all comments.
-        $node->setAttribute('comments', $comments);
+        $node->setAttribute('comments', []);
         $this->assertNull($node->getDocComment());
     }
 
@@ -108,6 +109,12 @@ class NodeAbstractTest extends \PHPUnit\Framework\TestCase
         $node->setAttribute('comments', [$c1, $c2]);
         $node->setDocComment($docComment);
         $this->assertSame([$c1, $c2, $docComment], $node->getAttribute('comments'));
+
+        // Replace doc comment that is not at the end.
+        $newDocComment = new Comment\Doc('/** new baz */');
+        $node->setAttribute('comments', [$c1, $docComment, $c2]);
+        $node->setDocComment($newDocComment);
+        $this->assertSame([$c1, $newDocComment, $c2], $node->getAttribute('comments'));
     }
 
     /**
@@ -240,6 +247,7 @@ PHP;
                         "kind": 10
                     }
                 },
+                "flags": 0,
                 "attributes": {
                     "startLine": 4,
                     "endLine": 4
@@ -266,6 +274,7 @@ PHP;
                         "endLine": 4
                     }
                 },
+                "flags": 0,
                 "attributes": {
                     "startLine": 4,
                     "endLine": 4
@@ -298,17 +307,23 @@ PHP;
             "comments": [
                 {
                     "nodeType": "Comment",
-                    "text": "\/\/ comment\n",
+                    "text": "\/\/ comment",
                     "line": 2,
                     "filePos": 6,
-                    "tokenPos": 1
+                    "tokenPos": 1,
+                    "endLine": 2,
+                    "endFilePos": 15,
+                    "endTokenPos": 1
                 },
                 {
                     "nodeType": "Comment_Doc",
                     "text": "\/** doc comment *\/",
                     "line": 3,
                     "filePos": 17,
-                    "tokenPos": 2
+                    "tokenPos": 3,
+                    "endLine": 3,
+                    "endFilePos": 34,
+                    "endTokenPos": 3
                 }
             ],
             "endLine": 6

test/PhpParser/NodeTraverserTest.php

@@ -266,13 +266,17 @@ class NodeTraverserTest extends \PHPUnit\Framework\TestCase
         $traverser->addVisitor($visitor2);
         $traverser->addVisitor($visitor3);
 
+        $getVisitors = (function () {
+            return $this->visitors;
+        })->bindTo($traverser, NodeTraverser::class);
+
         $preExpected = [$visitor1, $visitor2, $visitor3];
-        $this->assertAttributeSame($preExpected, 'visitors', $traverser, 'The appropriate visitors have not been added');
+        $this->assertSame($preExpected, $getVisitors());
 
         $traverser->removeVisitor($visitor2);
 
         $postExpected = [0 => $visitor1, 2 => $visitor3];
-        $this->assertAttributeSame($postExpected, 'visitors', $traverser, 'The appropriate visitors are not present after removal');
+        $this->assertSame($postExpected, $getVisitors());
     }
 
     public function testNoCloneNodes() {

test/PhpParser/NodeVisitor/NameResolverTest.php

@@ -94,13 +94,6 @@ namespace Baz {
     C;
     E;
     K;
-
-    class ClassWithTypeProperties
-    {
-        public float $php = 7.4;
-        public ?Foo $person;
-        protected static ?bool $probability;
-    }
 }
 EOC;
         $expectedCode = <<<'EOC'
@@ -169,12 +162,6 @@ namespace Baz {
     \Y\T\B\C;
     \Y\T\D\E;
     \Z\T\K;
-    class ClassWithTypeProperties
-    {
-        public float $php = 7.4;
-        public ?\Baz\Foo $person;
-        protected static ?bool $probability;
-    }
 }
 EOC;
 
@@ -210,6 +197,14 @@ class A extends B implements C, D {
 
 interface A extends C, D {
     public function a(A $a) : A;
+    public function b(A|B|int $a): A|B|int;
+}
+
+class ClassWithTypeProperties {
+    public float $php = 7.4;
+    public ?Foo $person;
+    protected static ?bool $probability;
+    public A|B|int $prop;
 }
 
 function f(A $a) : A {}
@@ -219,6 +214,10 @@ function(A $a) : A {};
 function fn3(?A $a) : ?A {}
 function fn4(?array $a) : ?array {}
 
+fn(array $a): array => $a;
+fn(A $a): A => $a;
+fn(?A $a): ?A => $a;
+
 A::b();
 A::$b;
 A::B;
@@ -248,6 +247,14 @@ class A extends \NS\B implements \NS\C, \NS\D
 interface A extends \NS\C, \NS\D
 {
     public function a(\NS\A $a) : \NS\A;
+    public function b(\NS\A|\NS\B|int $a) : \NS\A|\NS\B|int;
+}
+class ClassWithTypeProperties
+{
+    public float $php = 7.4;
+    public ?\NS\Foo $person;
+    protected static ?bool $probability;
+    public \NS\A|\NS\B|int $prop;
 }
 function f(\NS\A $a) : \NS\A
 {
@@ -263,6 +270,9 @@ function fn3(?\NS\A $a) : ?\NS\A
 function fn4(?array $a) : ?array
 {
 }
+fn(array $a): array => $a;
+fn(\NS\A $a): \NS\A => $a;
+fn(?\NS\A $a): ?\NS\A => $a;
 \NS\A::b();
 \NS\A::$b;
 \NS\A::B;

test/PhpParser/ParserTest.php

@@ -60,7 +60,8 @@ EOC;
         $this->assertInstanceOf(Stmt\Function_::class, $fn);
         $this->assertEquals([
             'comments' => [
-                new Comment\Doc('/** Doc comment */', 2, 6, 1),
+                new Comment\Doc('/** Doc comment */',
+                    2, 6, 1, 2, 23, 1),
             ],
             'startLine' => 3,
             'endLine' => 7,
@@ -82,8 +83,10 @@ EOC;
         $this->assertInstanceOf(Stmt\Echo_::class, $echo);
         $this->assertEquals([
             'comments' => [
-                new Comment("// Line\n", 4, 49, 12),
+                new Comment("// Line",
-                new Comment("// Comments\n", 5, 61, 14),
+                    4, 49, 12, 4, 55, 12),
+                new Comment("// Comments",
+                    5, 61, 14, 5, 71, 14),
             ],
             'startLine' => 6,
             'endLine' => 6,

test/code/formatPreservation/classMethodNop.test

@@ -0,0 +1,77 @@
+Adding statement to Class Method containing Nop
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        // I'm just a comment
+    }
+}
+-----
+$stmts[0]->stmts[0]->stmts[] = new Stmt\Expression(new Node\Expr\Variable('foo'));
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        // I'm just a comment
+        $foo;
+    }
+}
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        /* I'm just a comment */
+    }
+}
+-----
+$stmts[0]->stmts[0]->stmts[] = new Stmt\Expression(new Node\Expr\Variable('foo'));
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        /* I'm just a comment */
+        $foo;
+    }
+}
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        /* I'm just a comment */
+    }
+}
+-----
+$stmts[0]->stmts[0]->stmts[0]->setAttribute('comments', [new Comment("/* I'm a new comment */")]);
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        /* I'm a new comment */
+
+    }
+}
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        // I'm just a comment
+    }
+}
+-----
+$stmts[0]->stmts[0]->stmts[0]->setAttribute('comments', [new Comment("// I'm a new comment\n")]);
+-----
+<?php
+class Foo {
+    public function __construct()
+    {
+        // I'm a new comment
+
+    }
+}

test/code/formatPreservation/insertionOfNullable.test

@@ -173,4 +173,22 @@ $stmts[0]->name = new Node\Name('Foo');
 <?php
 
 namespace Foo
-{ echo 42; }
+{ echo 42; }
+-----
+<?php
+try
+{
+}
+catch (Exception)
+{
+}
+-----
+$stmts[0]->catches[0]->var = new Expr\Variable('e');
+-----
+<?php
+try
+{
+}
+catch (Exception $e)
+{
+}

test/code/formatPreservation/listInsertion.test

@@ -306,4 +306,14 @@ $stmts[0]->expr->expr->items[] = new Expr\ArrayItem(new Scalar\LNumber(24));
 $array = [
     1, 2,
     3, 24,
-];
+];
+-----
+<?php
+function test(): A
+|B {}
+-----
+$stmts[0]->returnType->types[] = new Node\Name('C');
+-----
+<?php
+function test(): A
+|B|C {}

test/code/formatPreservation/listRemoval.test

@@ -38,4 +38,15 @@ function foo(
     $b,
     $x,
     $y
-) {}
+) {}
+-----
+<?php
+function test(): A
+|B
+|C {}
+-----
+array_pop($stmts[0]->returnType->types);
+-----
+<?php
+function test(): A
+|B {}

test/code/formatPreservation/modifierChange.test

@@ -30,4 +30,28 @@ class Bar {
 
     public final function
     foo() {}
-}
+}
+-----
+<?php
+function test(
+    public T1 $x
+        = 'y',
+    private T2 $y
+        = 'z',
+    T3 $z
+        = 'x',
+) {}
+-----
+$stmts[0]->params[0]->flags = Stmt\Class_::MODIFIER_PRIVATE;
+$stmts[0]->params[1]->flags = 0;
+$stmts[0]->params[2]->flags = Stmt\Class_::MODIFIER_PUBLIC;
+-----
+<?php
+function test(
+    private T1 $x
+        = 'y',
+    T2 $y
+        = 'z',
+    public T3 $z
+        = 'x',
+) {}

test/code/formatPreservation/removalViaNull.test

@@ -191,4 +191,22 @@ $stmts[0]->name = null;
 
 namespace
     {
-    }
+    }
+-----
+<?php
+try
+{
+}
+catch (Exception $e)
+{
+}
+-----
+$stmts[0]->catches[0]->var = null;
+-----
+<?php
+try
+{
+}
+catch (Exception)
+{
+}

test/code/parser/errorHandling/lexerErrors.test

@@ -32,24 +32,25 @@ $a = 42;
 @@{ "\1" }@@
 $b = 24;
 -----
+!!positions
 Unexpected character "
" (ASCII 1) from 4:1 to 4:1
 array(
-    0: Stmt_Expression(
+    0: Stmt_Expression[3:1 - 3:8](
-        expr: Expr_Assign(
+        expr: Expr_Assign[3:1 - 3:7](
-            var: Expr_Variable(
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:7](
                 value: 42
             )
         )
     )
-    1: Stmt_Expression(
+    1: Stmt_Expression[5:1 - 5:8](
-        expr: Expr_Assign(
+        expr: Expr_Assign[5:1 - 5:7](
-            var: Expr_Variable(
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:7](
                 value: 24
             )
         )
@@ -62,24 +63,25 @@ $a = 42;
 @@{ "\0" }@@
 $b = 24;
 -----
+!!positions
 Unexpected null byte from 4:1 to 4:1
 array(
-    0: Stmt_Expression(
+    0: Stmt_Expression[3:1 - 3:8](
-        expr: Expr_Assign(
+        expr: Expr_Assign[3:1 - 3:7](
-            var: Expr_Variable(
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:7](
                 value: 42
             )
         )
     )
-    1: Stmt_Expression(
+    1: Stmt_Expression[5:1 - 5:8](
-        expr: Expr_Assign(
+        expr: Expr_Assign[5:1 - 5:7](
-            var: Expr_Variable(
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:7](
                 value: 24
             )
         )
@@ -94,35 +96,36 @@ $b = 2;
 @@{ "\2" }@@
 $c = 3;
 -----
-Unexpected character "@@{ "\1" }@@" (ASCII 1) from 4:1 to 4:1
+!!positions
-Unexpected character "@@{ "\2" }@@" (ASCII 2) from 6:1 to 6:1
+Unexpected character "
" (ASCII 1) from 4:1 to 4:1
+Unexpected character "" (ASCII 2) from 6:1 to 6:1
 array(
-    0: Stmt_Expression(
+    0: Stmt_Expression[3:1 - 3:7](
-        expr: Expr_Assign(
+        expr: Expr_Assign[3:1 - 3:6](
-            var: Expr_Variable(
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:6](
                 value: 1
             )
         )
     )
-    1: Stmt_Expression(
+    1: Stmt_Expression[5:1 - 5:7](
-        expr: Expr_Assign(
+        expr: Expr_Assign[5:1 - 5:6](
-            var: Expr_Variable(
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:6](
                 value: 2
             )
         )
     )
-    2: Stmt_Expression(
+    2: Stmt_Expression[7:1 - 7:7](
-        expr: Expr_Assign(
+        expr: Expr_Assign[7:1 - 7:6](
-            var: Expr_Variable(
+            var: Expr_Variable[7:1 - 7:2](
                 name: c
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[7:6 - 7:6](
                 value: 3
             )
         )

test/code/parser/errorHandling/recovery.test

@@ -594,14 +594,13 @@ isset($x, );
 
 declare(a=42, );
 
-function foo($a, ) {}
 foo($a, );
 global $a, ;
 static $a, ;
 echo $a, ;
 
 for ($a, ; $b, ; $c, );
-function ($a, ) use ($b, ) {};
+function ($a) use ($b, ) {};
 -----
 !!php7
 A trailing comma is not allowed here from 5:6 to 5:6
@@ -614,15 +613,13 @@ A trailing comma is not allowed here from 13:17 to 13:17
 A trailing comma is not allowed here from 14:14 to 14:14
 A trailing comma is not allowed here from 16:22 to 16:22
 A trailing comma is not allowed here from 21:13 to 21:13
-A trailing comma is not allowed here from 23:16 to 23:16
+A trailing comma is not allowed here from 24:10 to 24:10
 A trailing comma is not allowed here from 25:10 to 25:10
-A trailing comma is not allowed here from 26:10 to 26:10
+A trailing comma is not allowed here from 26:8 to 26:8
-A trailing comma is not allowed here from 27:8 to 27:8
+A trailing comma is not allowed here from 28:8 to 28:8
-A trailing comma is not allowed here from 29:8 to 29:8
+A trailing comma is not allowed here from 28:14 to 28:14
-A trailing comma is not allowed here from 29:14 to 29:14
+A trailing comma is not allowed here from 28:20 to 28:20
-A trailing comma is not allowed here from 29:20 to 29:20
+A trailing comma is not allowed here from 29:22 to 29:22
-A trailing comma is not allowed here from 30:13 to 30:13
-A trailing comma is not allowed here from 30:24 to 30:24
 array(
     0: Stmt_GroupUse(
         type: TYPE_UNKNOWN (0)
@@ -811,27 +808,7 @@ array(
         )
         stmts: null
     )
-    9: Stmt_Function(
+    9: Stmt_Expression(
-        byRef: false
-        name: Identifier(
-            name: foo
-        )
-        params: array(
-            0: Param(
-                type: null
-                byRef: false
-                variadic: false
-                var: Expr_Variable(
-                    name: a
-                )
-                default: null
-            )
-        )
-        returnType: null
-        stmts: array(
-        )
-    )
-    10: Stmt_Expression(
         expr: Expr_FuncCall(
             name: Name(
                 parts: array(
@@ -849,14 +826,14 @@ array(
             )
         )
     )
-    11: Stmt_Global(
+    10: Stmt_Global(
         vars: array(
             0: Expr_Variable(
                 name: a
             )
         )
     )
-    12: Stmt_Static(
+    11: Stmt_Static(
         vars: array(
             0: Stmt_StaticVar(
                 var: Expr_Variable(
@@ -866,14 +843,14 @@ array(
             )
         )
     )
-    13: Stmt_Echo(
+    12: Stmt_Echo(
         exprs: array(
             0: Expr_Variable(
                 name: a
             )
         )
     )
-    14: Stmt_For(
+    13: Stmt_For(
         init: array(
             0: Expr_Variable(
                 name: a
@@ -892,12 +869,13 @@ array(
         stmts: array(
         )
     )
-    15: Stmt_Expression(
+    14: Stmt_Expression(
         expr: Expr_Closure(
             static: false
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -1075,6 +1053,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type
@@ -1103,6 +1082,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type1
@@ -1116,6 +1096,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type2
@@ -1144,6 +1125,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: true
@@ -1168,6 +1150,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: true
                 variadic: false
@@ -1192,6 +1175,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Bar
@@ -1225,6 +1209,7 @@ array(
                 )
                 params: array(
                     0: Param(
+                        flags: 0
                         type: Name(
                             parts: array(
                                 0: Baz
@@ -1249,6 +1234,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: Name(
                         parts: array(
                             0: Foo

test/code/parser/expr/arrow_function.test

@@ -16,6 +16,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: Identifier(
                         name: bool
                     )
@@ -39,6 +40,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -62,6 +64,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: true
                     variadic: false
@@ -83,6 +86,7 @@ array(
             byRef: true
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -104,6 +108,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -113,6 +118,7 @@ array(
                     default: null
                 )
                 1: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: true
@@ -142,4 +148,4 @@ array(
             )
         )
     )
-)
+)

test/code/parser/expr/closure.test

@@ -16,6 +16,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -43,6 +44,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -96,6 +98,7 @@ array(
             byRef: true
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false
@@ -131,6 +134,7 @@ array(
             byRef: false
             params: array(
                 0: Param(
+                    flags: 0
                     type: null
                     byRef: false
                     variadic: false

test/code/parser/expr/exprInIsset.test

@@ -0,0 +1,49 @@
+Expressions in isset()
+-----
+<?php
+// This is legal.
+isset(($a), (($b)));
+// This is illegal, but not a syntax error.
+isset(1 + 1);
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Isset(
+            vars: array(
+                0: Expr_Variable(
+                    name: a
+                )
+                1: Expr_Variable(
+                    name: b
+                )
+            )
+            comments: array(
+                0: // This is legal.
+            )
+        )
+        comments: array(
+            0: // This is legal.
+        )
+    )
+    1: Stmt_Expression(
+        expr: Expr_Isset(
+            vars: array(
+                0: Expr_BinaryOp_Plus(
+                    left: Scalar_LNumber(
+                        value: 1
+                    )
+                    right: Scalar_LNumber(
+                        value: 1
+                    )
+                )
+            )
+            comments: array(
+                0: // This is illegal, but not a syntax error.
+            )
+        )
+        comments: array(
+            0: // This is illegal, but not a syntax error.
+        )
+    )
+)

test/code/parser/expr/exprInList.test

@@ -0,0 +1,81 @@
+Expressions in list()
+-----
+<?php
+
+// This is legal.
+list(($a), ((($b)))) = $x;
+// This is illegal, but not a syntax error.
+list(1 + 1) = $x;
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_Assign(
+            var: Expr_List(
+                items: array(
+                    0: Expr_ArrayItem(
+                        key: null
+                        value: Expr_Variable(
+                            name: a
+                        )
+                        byRef: false
+                        unpack: false
+                    )
+                    1: Expr_ArrayItem(
+                        key: null
+                        value: Expr_Variable(
+                            name: b
+                        )
+                        byRef: false
+                        unpack: false
+                    )
+                )
+                comments: array(
+                    0: // This is legal.
+                )
+            )
+            expr: Expr_Variable(
+                name: x
+            )
+            comments: array(
+                0: // This is legal.
+            )
+        )
+        comments: array(
+            0: // This is legal.
+        )
+    )
+    1: Stmt_Expression(
+        expr: Expr_Assign(
+            var: Expr_List(
+                items: array(
+                    0: Expr_ArrayItem(
+                        key: null
+                        value: Expr_BinaryOp_Plus(
+                            left: Scalar_LNumber(
+                                value: 1
+                            )
+                            right: Scalar_LNumber(
+                                value: 1
+                            )
+                        )
+                        byRef: false
+                        unpack: false
+                    )
+                )
+                comments: array(
+                    0: // This is illegal, but not a syntax error.
+                )
+            )
+            expr: Expr_Variable(
+                name: x
+            )
+            comments: array(
+                0: // This is illegal, but not a syntax error.
+            )
+        )
+        comments: array(
+            0: // This is illegal, but not a syntax error.
+        )
+    )
+)

test/code/parser/expr/fetchAndCall/constFetch.test

@@ -5,6 +5,8 @@ Constant fetches
 A;
 A::B;
 A::class;
+$a::B;
+$a::class;
 -----
 array(
     0: Stmt_Expression(
@@ -40,4 +42,24 @@ array(
             )
         )
     )
+    3: Stmt_Expression(
+        expr: Expr_ClassConstFetch(
+            class: Expr_Variable(
+                name: a
+            )
+            name: Identifier(
+                name: B
+            )
+        )
+    )
+    4: Stmt_Expression(
+        expr: Expr_ClassConstFetch(
+            class: Expr_Variable(
+                name: a
+            )
+            name: Identifier(
+                name: class
+            )
+        )
+    )
 )

test/code/parser/expr/uvs/constDeref.test

@@ -0,0 +1,292 @@
+Dereferencing of constants
+-----
+<?php
+
+A->length;
+A->length();
+A[0];
+A[0][1][2];
+A{0};
+
+A::B[0];
+A::B[0][1][2];
+A::B{0};
+A::B->length;
+A::B->length();
+A::B::C;
+A::B::$c;
+A::B::c();
+
+__FUNCTION__[0];
+__FUNCTION__->length;
+__FUNCIONT__->length();
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_PropertyFetch(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+        )
+    )
+    1: Stmt_Expression(
+        expr: Expr_MethodCall(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+            args: array(
+            )
+        )
+    )
+    2: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    3: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ArrayDimFetch(
+                var: Expr_ArrayDimFetch(
+                    var: Expr_ConstFetch(
+                        name: Name(
+                            parts: array(
+                                0: A
+                            )
+                        )
+                    )
+                    dim: Scalar_LNumber(
+                        value: 0
+                    )
+                )
+                dim: Scalar_LNumber(
+                    value: 1
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 2
+            )
+        )
+    )
+    4: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    5: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    6: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ArrayDimFetch(
+                var: Expr_ArrayDimFetch(
+                    var: Expr_ClassConstFetch(
+                        class: Name(
+                            parts: array(
+                                0: A
+                            )
+                        )
+                        name: Identifier(
+                            name: B
+                        )
+                    )
+                    dim: Scalar_LNumber(
+                        value: 0
+                    )
+                )
+                dim: Scalar_LNumber(
+                    value: 1
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 2
+            )
+        )
+    )
+    7: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    8: Stmt_Expression(
+        expr: Expr_PropertyFetch(
+            var: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+        )
+    )
+    9: Stmt_Expression(
+        expr: Expr_MethodCall(
+            var: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+            args: array(
+            )
+        )
+    )
+    10: Stmt_Expression(
+        expr: Expr_ClassConstFetch(
+            class: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: Identifier(
+                name: C
+            )
+        )
+    )
+    11: Stmt_Expression(
+        expr: Expr_StaticPropertyFetch(
+            class: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: VarLikeIdentifier(
+                name: c
+            )
+        )
+    )
+    12: Stmt_Expression(
+        expr: Expr_StaticCall(
+            class: Expr_ClassConstFetch(
+                class: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                name: Identifier(
+                    name: B
+                )
+            )
+            name: Identifier(
+                name: c
+            )
+            args: array(
+            )
+        )
+    )
+    13: Stmt_Expression(
+        expr: Expr_ArrayDimFetch(
+            var: Scalar_MagicConst_Function(
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+    )
+    14: Stmt_Expression(
+        expr: Expr_PropertyFetch(
+            var: Scalar_MagicConst_Function(
+            )
+            name: Identifier(
+                name: length
+            )
+        )
+    )
+    15: Stmt_Expression(
+        expr: Expr_MethodCall(
+            var: Expr_ConstFetch(
+                name: Name(
+                    parts: array(
+                        0: __FUNCIONT__
+                    )
+                )
+            )
+            name: Identifier(
+                name: length
+            )
+            args: array(
+            )
+        )
+    )
+)

test/code/parser/expr/uvs/indirectCall.test

@@ -198,6 +198,7 @@ array(
                         byRef: false
                         params: array(
                             0: Param(
+                                flags: 0
                                 type: null
                                 byRef: false
                                 variadic: false
@@ -264,6 +265,7 @@ array(
                                     byRef: false
                                     params: array(
                                         0: Param(
+                                            flags: 0
                                             type: null
                                             byRef: false
                                             variadic: false

test/code/parser/expr/uvs/misc.test

@@ -2,24 +2,36 @@ Uniform variable syntax in PHP 7 (misc)
 -----
 <?php
 
-A::A[0];
-A::A[0][1][2];
 "string"->length();
+"foo$bar"[0];
+"foo$bar"->length();
 (clone $obj)->b[0](1);
 [0, 1][0] = 1;
 -----
 !!php7
 array(
     0: Stmt_Expression(
+        expr: Expr_MethodCall(
+            var: Scalar_String(
+                value: string
+            )
+            name: Identifier(
+                name: length
+            )
+            args: array(
+            )
+        )
+    )
+    1: Stmt_Expression(
         expr: Expr_ArrayDimFetch(
-            var: Expr_ClassConstFetch(
+            var: Scalar_Encapsed(
-                class: Name(
+                parts: array(
-                    parts: array(
+                    0: Scalar_EncapsedStringPart(
-                        0: A
+                        value: foo
+                    )
+                    1: Expr_Variable(
+                        name: bar
                     )
-                )
-                name: Identifier(
-                    name: A
                 )
             )
             dim: Scalar_LNumber(
@@ -27,37 +39,17 @@ array(
             )
         )
     )
-    1: Stmt_Expression(
-        expr: Expr_ArrayDimFetch(
-            var: Expr_ArrayDimFetch(
-                var: Expr_ArrayDimFetch(
-                    var: Expr_ClassConstFetch(
-                        class: Name(
-                            parts: array(
-                                0: A
-                            )
-                        )
-                        name: Identifier(
-                            name: A
-                        )
-                    )
-                    dim: Scalar_LNumber(
-                        value: 0
-                    )
-                )
-                dim: Scalar_LNumber(
-                    value: 1
-                )
-            )
-            dim: Scalar_LNumber(
-                value: 2
-            )
-        )
-    )
     2: Stmt_Expression(
         expr: Expr_MethodCall(
-            var: Scalar_String(
+            var: Scalar_Encapsed(
-                value: string
+                parts: array(
+                    0: Scalar_EncapsedStringPart(
+                        value: foo
+                    )
+                    1: Expr_Variable(
+                        name: bar
+                    )
+                )
             )
             name: Identifier(
                 name: length

test/code/parser/expr/uvs/newInstanceofExpr.test

@@ -0,0 +1,61 @@
+Arbitrary expressions in new and instanceof
+-----
+<?php
+
+new ('Foo' . $bar);
+new ('Foo' . $bar)($arg);
+$obj instanceof ('Foo' . $bar);
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_New(
+            class: Expr_BinaryOp_Concat(
+                left: Scalar_String(
+                    value: Foo
+                )
+                right: Expr_Variable(
+                    name: bar
+                )
+            )
+            args: array(
+            )
+        )
+    )
+    1: Stmt_Expression(
+        expr: Expr_New(
+            class: Expr_BinaryOp_Concat(
+                left: Scalar_String(
+                    value: Foo
+                )
+                right: Expr_Variable(
+                    name: bar
+                )
+            )
+            args: array(
+                0: Arg(
+                    value: Expr_Variable(
+                        name: arg
+                    )
+                    byRef: false
+                    unpack: false
+                )
+            )
+        )
+    )
+    2: Stmt_Expression(
+        expr: Expr_Instanceof(
+            expr: Expr_Variable(
+                name: obj
+            )
+            class: Expr_BinaryOp_Concat(
+                left: Scalar_String(
+                    value: Foo
+                )
+                right: Expr_Variable(
+                    name: bar
+                )
+            )
+        )
+    )
+)

test/code/parser/nopPositions.test

@@ -10,4 +10,18 @@ array(
             0: /* Comment */
         )
     )
+)
+-----
+<?php
+{
+    /* comment */
+}
+-----
+!!positions
+array(
+    0: Stmt_Nop[3:0 - 3:17](
+        comments: array(
+            0: /* comment */
+        )
+    )
 )

test/code/parser/scalar/numberSeparators.test

@@ -0,0 +1,199 @@
+Different integer syntaxes
+-----
+<?php
+
+6.674_083e-11;
+299_792_458;
+0xCAFE_F00D;
+0b0101_1111;
+0137_041;
+
+// already a valid constant name
+_100;
+
+// syntax errors
+100_;
+1__1;
+1_.0;
+1._0;
+0x_123;
+0b_101;
+1_e2;
+1e_2;
+-----
+Syntax error, unexpected T_STRING from 13:4 to 13:4
+Syntax error, unexpected T_STRING from 14:2 to 14:4
+Syntax error, unexpected T_STRING from 15:2 to 15:2
+Syntax error, unexpected T_STRING from 16:3 to 16:4
+Syntax error, unexpected T_STRING from 17:2 to 17:6
+Syntax error, unexpected T_STRING from 18:2 to 18:6
+Syntax error, unexpected T_STRING from 19:2 to 19:4
+Syntax error, unexpected T_STRING from 20:2 to 20:4
+array(
+    0: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 6.674083E-11
+        )
+    )
+    1: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 299792458
+        )
+    )
+    2: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 3405705229
+        )
+    )
+    3: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 95
+        )
+    )
+    4: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 48673
+        )
+    )
+    5: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _100
+                )
+                comments: array(
+                    0: // already a valid constant name
+                )
+            )
+            comments: array(
+                0: // already a valid constant name
+            )
+        )
+        comments: array(
+            0: // already a valid constant name
+        )
+    )
+    6: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 100
+            comments: array(
+                0: // syntax errors
+            )
+        )
+        comments: array(
+            0: // syntax errors
+        )
+    )
+    7: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _
+                )
+            )
+        )
+    )
+    8: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    9: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: __1
+                )
+            )
+        )
+    )
+    10: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    11: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _
+                )
+            )
+        )
+    )
+    12: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 0
+        )
+    )
+    13: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 1
+        )
+    )
+    14: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _0
+                )
+            )
+        )
+    )
+    15: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 0
+        )
+    )
+    16: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: x_123
+                )
+            )
+        )
+    )
+    17: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 0
+        )
+    )
+    18: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: b_101
+                )
+            )
+        )
+    )
+    19: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    20: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _e2
+                )
+            )
+        )
+    )
+    21: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    22: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: e_2
+                )
+            )
+        )
+    )
+)

test/code/parser/stmt/class/propertyTypes.test

@@ -67,4 +67,4 @@ array(
             )
         )
     )
-)
+)

test/code/parser/stmt/class/property_promotion.test

@@ -0,0 +1,81 @@
+Property promotion
+-----
+<?php
+
+class Point {
+    public function __construct(
+        public float $x = 0.0,
+        protected array $y = [],
+        private string $z = 'hello'
+    ) {}
+}
+-----
+!!php7
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: Identifier(
+            name: Point
+        )
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassMethod(
+                flags: MODIFIER_PUBLIC (1)
+                byRef: false
+                name: Identifier(
+                    name: __construct
+                )
+                params: array(
+                    0: Param(
+                        flags: MODIFIER_PUBLIC (1)
+                        type: Identifier(
+                            name: float
+                        )
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: x
+                        )
+                        default: Scalar_DNumber(
+                            value: 0
+                        )
+                    )
+                    1: Param(
+                        flags: MODIFIER_PROTECTED (2)
+                        type: Identifier(
+                            name: array
+                        )
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: y
+                        )
+                        default: Expr_Array(
+                            items: array(
+                            )
+                        )
+                    )
+                    2: Param(
+                        flags: MODIFIER_PRIVATE (4)
+                        type: Identifier(
+                            name: string
+                        )
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: z
+                        )
+                        default: Scalar_String(
+                            value: hello
+                        )
+                    )
+                )
+                returnType: null
+                stmts: array(
+                )
+            )
+        )
+    )
+)

test/code/parser/stmt/class/simple.test

@@ -127,6 +127,7 @@ array(
                 )
                 params: array(
                     0: Param(
+                        flags: 0
                         type: null
                         byRef: false
                         variadic: false

test/code/parser/stmt/function/builtinTypeDeclarations.test

@@ -1,7 +1,7 @@
 Scalar type declarations
 -----
 <?php
-function test(bool $a, Int $b, FLOAT $c, StRiNg $d, iterable $e, object $f) : void {}
+function test(bool $a, Int $b, FLOAT $c, StRiNg $d, iterable $e, object $f, mixed $g) : void {}
 -----
 !!php7
 array(
@@ -12,6 +12,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: Identifier(
                     name: bool
                 )
@@ -23,6 +24,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Identifier(
                     name: int
                 )
@@ -34,6 +36,7 @@ array(
                 default: null
             )
             2: Param(
+                flags: 0
                 type: Identifier(
                     name: float
                 )
@@ -45,6 +48,7 @@ array(
                 default: null
             )
             3: Param(
+                flags: 0
                 type: Identifier(
                     name: string
                 )
@@ -56,6 +60,7 @@ array(
                 default: null
             )
             4: Param(
+                flags: 0
                 type: Identifier(
                     name: iterable
                 )
@@ -67,6 +72,7 @@ array(
                 default: null
             )
             5: Param(
+                flags: 0
                 type: Identifier(
                     name: object
                 )
@@ -77,6 +83,18 @@ array(
                 )
                 default: null
             )
+            6: Param(
+                flags: 0
+                type: Identifier(
+                    name: mixed
+                )
+                byRef: false
+                variadic: false
+                var: Expr_Variable(
+                    name: g
+                )
+                default: null
+            )
         )
         returnType: Identifier(
             name: void

test/code/parser/stmt/function/byRef.test

@@ -13,6 +13,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: true
                 variadic: false
@@ -33,6 +34,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false

test/code/parser/stmt/function/defaultValues.test

@@ -22,6 +22,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -37,6 +38,7 @@ array(
                 )
             )
             1: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -48,6 +50,7 @@ array(
                 )
             )
             2: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -66,6 +69,7 @@ array(
                 )
             )
             3: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -79,6 +83,7 @@ array(
                 )
             )
             4: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -92,6 +97,7 @@ array(
                 )
             )
             5: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -104,6 +110,7 @@ array(
                 )
             )
             6: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -116,6 +123,7 @@ array(
                 )
             )
             7: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -136,6 +144,7 @@ array(
                 )
             )
             8: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false

test/code/parser/stmt/function/nullableTypes.test

@@ -14,6 +14,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: NullableType(
                     type: Name(
                         parts: array(
@@ -29,6 +30,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: NullableType(
                     type: Identifier(
                         name: string

test/code/parser/stmt/function/parameters_trailing_comma.test

@@ -0,0 +1,116 @@
+Trailing comma in parameter list
+-----
+<?php
+
+function foo($bar, ) {
+}
+-----
+!!php7
+array(
+    0: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: foo
+        )
+        params: array(
+            0: Param(
+                flags: 0
+                type: null
+                byRef: false
+                variadic: false
+                var: Expr_Variable(
+                    name: bar
+                )
+                default: null
+            )
+        )
+        returnType: null
+        stmts: array(
+        )
+    )
+)
+-----
+<?php
+
+class Foo
+{
+    function __construct($name, $value, )
+    {
+    }
+}
+-----
+!!php7
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: Identifier(
+            name: Foo
+        )
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassMethod(
+                flags: 0
+                byRef: false
+                name: Identifier(
+                    name: __construct
+                )
+                params: array(
+                    0: Param(
+                        flags: 0
+                        type: null
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: name
+                        )
+                        default: null
+                    )
+                    1: Param(
+                        flags: 0
+                        type: null
+                        byRef: false
+                        variadic: false
+                        var: Expr_Variable(
+                            name: value
+                        )
+                        default: null
+                    )
+                )
+                returnType: null
+                stmts: array(
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+fn($foo, ) => $bar;
+-----
+!!php7
+array(
+    0: Stmt_Expression(
+        expr: Expr_ArrowFunction(
+            static: false
+            byRef: false
+            params: array(
+                0: Param(
+                    flags: 0
+                    type: null
+                    byRef: false
+                    variadic: false
+                    var: Expr_Variable(
+                        name: foo
+                    )
+                    default: null
+                )
+            )
+            returnType: null
+            expr: Expr_Variable(
+                name: bar
+            )
+        )
+    )
+)

test/code/parser/stmt/function/typeDeclarations.test

@@ -12,6 +12,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -21,6 +22,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Identifier(
                     name: array
                 )
@@ -32,6 +34,7 @@ array(
                 default: null
             )
             2: Param(
+                flags: 0
                 type: Identifier(
                     name: callable
                 )
@@ -43,6 +46,7 @@ array(
                 default: null
             )
             3: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: E

test/code/parser/stmt/function/unionTypes.test

@@ -0,0 +1,93 @@
+Union types
+-----
+<?php
+
+class Test {
+    public A|iterable|null $prop;
+}
+
+function test(A|B $a): int|false {}
+-----
+!!php7
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: Identifier(
+            name: Test
+        )
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_Property(
+                flags: MODIFIER_PUBLIC (1)
+                type: UnionType(
+                    types: array(
+                        0: Name(
+                            parts: array(
+                                0: A
+                            )
+                        )
+                        1: Identifier(
+                            name: iterable
+                        )
+                        2: Identifier(
+                            name: null
+                        )
+                    )
+                )
+                props: array(
+                    0: Stmt_PropertyProperty(
+                        name: VarLikeIdentifier(
+                            name: prop
+                        )
+                        default: null
+                    )
+                )
+            )
+        )
+    )
+    1: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: test
+        )
+        params: array(
+            0: Param(
+                flags: 0
+                type: UnionType(
+                    types: array(
+                        0: Name(
+                            parts: array(
+                                0: A
+                            )
+                        )
+                        1: Name(
+                            parts: array(
+                                0: B
+                            )
+                        )
+                    )
+                )
+                byRef: false
+                variadic: false
+                var: Expr_Variable(
+                    name: a
+                )
+                default: null
+            )
+        )
+        returnType: UnionType(
+            types: array(
+                0: Identifier(
+                    name: int
+                )
+                1: Identifier(
+                    name: false
+                )
+            )
+        )
+        stmts: array(
+        )
+    )
+)

test/code/parser/stmt/function/variadic.test

@@ -14,6 +14,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -23,6 +24,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: true
@@ -43,6 +45,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -52,6 +55,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: null
                 byRef: true
                 variadic: true
@@ -72,6 +76,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -81,6 +86,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type
@@ -105,6 +111,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: false
@@ -114,6 +121,7 @@ array(
                 default: null
             )
             1: Param(
+                flags: 0
                 type: Name(
                     parts: array(
                         0: Type

test/code/parser/stmt/function/variadicDefaultValue.test

@@ -12,6 +12,7 @@ array(
         )
         params: array(
             0: Param(
+                flags: 0
                 type: null
                 byRef: false
                 variadic: true

test/code/parser/stmt/tryCatch_without_variable.test

@@ -0,0 +1,32 @@
+Try/Catch without variable
+-----
+<?php
+
+try {
+
+} catch (Exception) {
+
+}
+-----
+!!php7
+array(
+    0: Stmt_TryCatch(
+        stmts: array(
+        )
+        catches: array(
+            0: Stmt_Catch(
+                types: array(
+                    0: Name(
+                        parts: array(
+                            0: Exception
+                        )
+                    )
+                )
+                var: null
+                stmts: array(
+                )
+            )
+        )
+        finally: null
+    )
+)

test/code/prettyPrinter/expr/closure.test

@@ -2,7 +2,10 @@ Closures
 -----
 <?php
 
-$closureWithArgs = function ($arg1, $arg2) {
+$closureWithArgs = function (
+    $arg1,
+    $arg2
+) {
     $comment = 'closure body';
 };
 

test/code/prettyPrinter/stmt/property_promotion.test

@@ -0,0 +1,49 @@
+Property promotion
+-----
+<?php
+
+class Point
+{
+    public function __construct(
+        public float $x = 0.0,
+        protected array $y = [],
+        private string $z = 'hello',
+    ) {
+    }
+}
+-----
+!!php7
+class Point
+{
+    public function __construct(public float $x = 0.0, protected array $y = [], private string $z = 'hello')
+    {
+    }
+}
+-----
+<?php
+class Test
+{
+    public $z;
+    public function __construct(
+        public int $x,
+        /** @SomeAnnotation() */
+        public string $y = "123",
+        string $z = "abc"
+    )
+    {
+    }
+}
+-----
+!!php7
+class Test
+{
+    public $z;
+    public function __construct(
+        public int $x,
+        /** @SomeAnnotation() */
+        public string $y = "123",
+        string $z = "abc"
+    )
+    {
+    }
+}

test/code/prettyPrinter/stmt/tryCatch_without_variable.test

@@ -0,0 +1,17 @@
+tryCatch without variable
+-----
+<?php
+
+try {
+
+} catch (Exception) {
+
+} finally {
+
+}
+-----
+!!php7
+try {
+} catch (Exception) {
+} finally {
+}

test/code/prettyPrinter/stmt/union_types.test

@@ -0,0 +1,18 @@
+Union types
+-----
+<?php
+
+class Test {
+    public A|iterable|null $prop;
+}
+
+function test(A|B $a): int|false {}
+-----
+!!php7
+class Test
+{
+    public A|iterable|null $prop;
+}
+function test(A|B $a) : int|false
+{
+}

test_old/run.php

@@ -100,6 +100,8 @@ switch ($testType) {
 | Zend.tests.multibyte.multibyte_encoding_005
 # invalid code due to missing WS after opening tag
 | tests.run-test.bug75042-3
+# contains invalid chars, which we treat as parse error
+| Zend.tests.warning_during_heredoc_scan_ahead
 # pretty print difference due to INF vs 1e1000
 | ext.standard.tests.general_functions.bug27678
 | tests.lang.bug24640
@@ -231,6 +233,8 @@ foreach (new RecursiveIteratorIterator(
         echo $file, ":\n    Parse failed with message: {$e->getMessage()}\n";
 
         ++$parseFail;
+    } catch (Throwable $e) {
+        echo $file, ":\n    Unknown error occurred: $e\n";
     }
 }