Release PHP-Parser 4.2.4

This was recently fixed in PHP via
310708845f
so we should fix it here as well.

I'm not adding a dedicated test as it will fail until new PHP versions
are released. This is indirectly tested through the php-src tester.

.gitignore

@@ -2,3 +2,4 @@ vendor/
 composer.lock
 grammar/kmyacc.exe
 grammar/y.output
+.phpunit.result.cache

.travis.yml

@@ -16,7 +16,12 @@ php:
 
 install:
   - if [ $TRAVIS_PHP_VERSION = '7.0' ]; then composer require satooshi/php-coveralls '~1.0'; fi
-  - composer install --prefer-dist --ignore-platform-reqs
+  - |
+    if [ $TRAVIS_PHP_VERSION = 'nightly' ]; then
+      composer install --prefer-dist --ignore-platform-reqs;
+    else
+      composer install --prefer-dist;
+    fi
 
 matrix:
   allow_failures:

CHANGELOG.md

@@ -1,8 +1,32 @@
-Version 4.2.3-dev
+Version 4.2.5-dev
 -----------------
 
 Nothing yet.
 
+Version 4.2.4 (2019-09-01)
+--------------------------
+
+### Added
+
+* Added getProperties(), getConstants() and getTraitUses() to ClassLike. (#629, #630)
+
+### Fixed
+
+* Fixed flexible heredoc emulation to check for digits after the end label. This synchronizes
+  behavior with the upcoming PHP 7.3.10 release.
+
+Version 4.2.3 (2019-08-12)
+--------------------------
+
+### Added
+
+* [PHP 7.4] Add support for numeric literal separators. (#615)
+
+### Fixed
+
+* Fixed resolution of return types for arrow functions. (#613)
+* Fixed compatibility with PHP 7.4.
+
 Version 4.2.2 (2019-05-25)
 --------------------------
 

README.md

@@ -3,10 +3,10 @@ PHP Parser
 
 [![Build Status](https://travis-ci.org/nikic/PHP-Parser.svg?branch=master)](https://travis-ci.org/nikic/PHP-Parser) [![Coverage Status](https://coveralls.io/repos/github/nikic/PHP-Parser/badge.svg?branch=master)](https://coveralls.io/github/nikic/PHP-Parser?branch=master)
 
-This is a PHP 5.2 to PHP 7.3 parser written in PHP. Its purpose is to simplify static code analysis and
+This is a PHP 5.2 to PHP 7.4 parser written in PHP. Its purpose is to simplify static code analysis and
 manipulation.
 
-[**Documentation for version 4.x**][doc_master] (stable; for running on PHP >= 7.0; for parsing PHP 5.2 to PHP 7.3).
+[**Documentation for version 4.x**][doc_master] (stable; for running on PHP >= 7.0; for parsing PHP 5.2 to PHP 7.4).
 
 [Documentation for version 3.x][doc_3_x] (unsupported; for running on PHP >= 5.5; for parsing PHP 5.2 to PHP 7.2).
 

composer.json

@@ -17,7 +17,7 @@
         "ext-tokenizer": "*"
     },
     "require-dev": {
-        "phpunit/phpunit": "^6.5 || ^7.0"
+        "phpunit/phpunit": "^6.5 || ^7.0 || ^8.0"
     },
     "extra": {
         "branch-alias": {

doc/0_Introduction.markdown

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-This project is a PHP 5.2 to PHP 7.3 parser **written in PHP itself**.
+This project is a PHP 5.2 to PHP 7.4 parser **written in PHP itself**.
 
 What is this for?
 -----------------
@@ -26,11 +26,11 @@ programmatic PHP code analysis are incidentally PHP developers, not C developers
 What can it parse?
 ------------------
 
-The parser supports parsing PHP 5.2-7.3.
+The parser supports parsing PHP 5.2-7.4.
 
 As the parser is based on the tokens returned by `token_get_all` (which is only able to lex the PHP
 version it runs on), additionally a wrapper for emulating tokens from newer versions is provided.
-This allows to parse PHP 7.3 source code running on PHP 7.0, for example. This emulation is somewhat
+This allows to parse PHP 7.4 source code running on PHP 7.0, for example. This emulation is somewhat
 hacky and not perfect, but it should work well on any sane code.
 
 What output does it produce?
@@ -56,7 +56,7 @@ array(
 ```
 
 This matches the structure of the code: An echo statement, which takes two strings as expressions,
-with the values `Hi` and `World!`.
+with the values `Hi` and `World`.
 
 You can also see that the AST does not contain any whitespace information (but most comments are saved).
 So using it for formatting analysis is not possible.

doc/2_Usage_of_basic_components.markdown

@@ -77,7 +77,7 @@ A parser instance can be reused to parse multiple files.
 Node dumping
 ------------
 
-To dump the abstact syntax tree in human readable form, a `NodeDumper` can be used:
+To dump the abstract syntax tree in human readable form, a `NodeDumper` can be used:
 
 ```php
 <?php

doc/component/Walking_the_AST.markdown

@@ -312,7 +312,7 @@ $extendingClasses = $nodeFinder->find($stmts, function(Node $node) {
         && $node->extends !== null;
 });
 
-// Find first class occuring in the AST. Returns null if no class exists.
+// Find first class occurring in the AST. Returns null if no class exists.
 $class = $nodeFinder->findFirstInstanceOf($stmts, Node\Stmt\Class_::class);
 
 // Find first class that has name $name

lib/PhpParser/Lexer.php

@@ -37,10 +37,15 @@ class Lexer
         // map from internal tokens to PhpParser tokens
         $this->tokenMap = $this->createTokenMap();
 
+        // Compatibility define for PHP < 7.4
+        if (!defined('T_BAD_CHARACTER')) {
+            \define('T_BAD_CHARACTER', -1);
+        }
+
         // map of tokens to drop while lexing (the map is only used for isset lookup,
         // that's why the value is simply set to 1; the value is never actually used.)
         $this->dropTokens = array_fill_keys(
-            [\T_WHITESPACE, \T_OPEN_TAG, \T_COMMENT, \T_DOC_COMMENT], 1
+            [\T_WHITESPACE, \T_OPEN_TAG, \T_COMMENT, \T_DOC_COMMENT, \T_BAD_CHARACTER], 1
         );
 
         $defaultAttributes = ['comments', 'startLine', 'endLine'];
@@ -92,13 +97,9 @@ class Lexer
     }
 
     private function handleInvalidCharacterRange($start, $end, $line, ErrorHandler $errorHandler) {
+        $tokens = [];
         for ($i = $start; $i < $end; $i++) {
             $chr = $this->code[$i];
-            if ($chr === 'b' || $chr === 'B') {
-                // HHVM does not treat b" tokens correctly, so ignore these
-                continue;
-            }
-
             if ($chr === "\0") {
                 // PHP cuts error message after null byte, so need special case
                 $errorMsg = 'Unexpected null byte';
@@ -108,6 +109,7 @@ class Lexer
                 );
             }
 
+            $tokens[] = [\T_BAD_CHARACTER, $chr, $line];
             $errorHandler->handleError(new Error($errorMsg, [
                 'startLine' => $line,
                 'endLine' => $line,
@@ -115,6 +117,7 @@ class Lexer
                 'endFilePos' => $i,
             ]));
         }
+        return $tokens;
     }
 
     /**
@@ -155,16 +158,29 @@ class Lexer
 
         $filePos = 0;
         $line = 1;
-        foreach ($this->tokens as $token) {
+        $numTokens = \count($this->tokens);
+        for ($i = 0; $i < $numTokens; $i++) {
+            $token = $this->tokens[$i];
+
+            // Since PHP 7.4 invalid characters are represented by a T_BAD_CHARACTER token.
+            // In this case we only need to emit an error.
+            if ($token[0] === \T_BAD_CHARACTER) {
+                $this->handleInvalidCharacterRange($filePos, $filePos + 1, $line, $errorHandler);
+            }
+
             $tokenValue = \is_string($token) ? $token : $token[1];
             $tokenLen = \strlen($tokenValue);
 
             if (substr($this->code, $filePos, $tokenLen) !== $tokenValue) {
                 // Something is missing, must be an invalid character
                 $nextFilePos = strpos($this->code, $tokenValue, $filePos);
-                $this->handleInvalidCharacterRange(
+                $badCharTokens = $this->handleInvalidCharacterRange(
                     $filePos, $nextFilePos, $line, $errorHandler);
                 $filePos = (int) $nextFilePos;
+
+                array_splice($this->tokens, $i, 0, $badCharTokens);
+                $numTokens += \count($badCharTokens);
+                $i += \count($badCharTokens);
             }
 
             $filePos += $tokenLen;
@@ -187,8 +203,9 @@ class Lexer
                 $this->tokens[] = [$isDocComment ? \T_DOC_COMMENT : \T_COMMENT, $comment, $line];
             } else {
                 // Invalid characters at the end of the input
-                $this->handleInvalidCharacterRange(
+                $badCharTokens = $this->handleInvalidCharacterRange(
                     $filePos, \strlen($this->code), $line, $errorHandler);
+                $this->tokens = array_merge($this->tokens, $badCharTokens);
             }
             return;
         }

lib/PhpParser/Lexer/Emulative.php

@@ -7,17 +7,22 @@ use PhpParser\ErrorHandler;
 use PhpParser\Lexer;
 use PhpParser\Lexer\TokenEmulator\CoaleseEqualTokenEmulator;
 use PhpParser\Lexer\TokenEmulator\FnTokenEmulator;
+use PhpParser\Lexer\TokenEmulator\NumericLiteralSeparatorEmulator;
 use PhpParser\Lexer\TokenEmulator\TokenEmulatorInterface;
+use PhpParser\Parser\Tokens;
 
 class Emulative extends Lexer
 {
     const PHP_7_3 = '7.3.0dev';
     const PHP_7_4 = '7.4.0dev';
 
+    const T_COALESCE_EQUAL = 1007;
+    const T_FN = 1008;
+
     const FLEXIBLE_DOC_STRING_REGEX = <<<'REGEX'
 /<<<[ \t]*(['"]?)([a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*)\1\r?\n
 (?:.*\r?\n)*?
-(?<indentation>\h*)\2(?![a-zA-Z_\x80-\xff])(?<separator>(?:;?[\r\n])?)/x
+(?<indentation>\h*)\2(?![a-zA-Z0-9_\x80-\xff])(?<separator>(?:;?[\r\n])?)/x
 REGEX;
 
     /** @var mixed[] Patches used to reverse changes introduced in the code */
@@ -33,14 +38,12 @@ REGEX;
     {
         parent::__construct($options);
 
-        // prepare token emulators
         $this->tokenEmulators[] = new FnTokenEmulator();
         $this->tokenEmulators[] = new CoaleseEqualTokenEmulator();
+        $this->tokenEmulators[] = new NumericLiteralSeparatorEmulator();
 
-        // add emulated tokens here
-        foreach ($this->tokenEmulators as $emulativeToken) {
-            $this->tokenMap[$emulativeToken->getTokenId()] = $emulativeToken->getParserTokenId();
-        }
+        $this->tokenMap[self::T_COALESCE_EQUAL] = Tokens::T_COALESCE_EQUAL;
+        $this->tokenMap[self::T_FN] = Tokens::T_FN;
     }
 
     public function startLexing(string $code, ErrorHandler $errorHandler = null) {
@@ -57,14 +60,6 @@ REGEX;
         // 1. emulation of heredoc and nowdoc new syntax
         $preparedCode = $this->processHeredocNowdoc($code);
         parent::startLexing($preparedCode, $collector);
-
-        // add token emulation
-        foreach ($this->tokenEmulators as $emulativeToken) {
-            if ($emulativeToken->isEmulationNeeded($code)) {
-                $this->tokens = $emulativeToken->emulate($code, $this->tokens);
-            }
-        }
-
         $this->fixupTokens();
 
         $errors = $collector->getErrors();
@@ -74,6 +69,13 @@ REGEX;
                 $errorHandler->handleError($error);
             }
         }
+
+        // add token emulation
+        foreach ($this->tokenEmulators as $emulativeToken) {
+            if ($emulativeToken->isEmulationNeeded($code)) {
+                $this->tokens = $emulativeToken->emulate($code, $this->tokens);
+            }
+        }
     }
 
     private function isHeredocNowdocEmulationNeeded(string $code): bool

lib/PhpParser/Lexer/TokenEmulator/CoaleseEqualTokenEmulator.php

@@ -3,22 +3,9 @@
 namespace PhpParser\Lexer\TokenEmulator;
 
 use PhpParser\Lexer\Emulative;
-use PhpParser\Parser\Tokens;
 
 final class CoaleseEqualTokenEmulator implements TokenEmulatorInterface
 {
-    const T_COALESCE_EQUAL = 1007;
-
-    public function getTokenId(): int
-    {
-        return self::T_COALESCE_EQUAL;
-    }
-
-    public function getParserTokenId(): int
-    {
-        return Tokens::T_COALESCE_EQUAL;
-    }
-
     public function isEmulationNeeded(string $code) : bool
     {
         // skip version where this is supported
@@ -38,7 +25,7 @@ final class CoaleseEqualTokenEmulator implements TokenEmulatorInterface
             if (isset($tokens[$i + 1])) {
                 if ($tokens[$i][0] === T_COALESCE && $tokens[$i + 1] === '=') {
                     array_splice($tokens, $i, 2, [
-                        [self::T_COALESCE_EQUAL, '??=', $line]
+                        [Emulative::T_COALESCE_EQUAL, '??=', $line]
                     ]);
                     $c--;
                     continue;

lib/PhpParser/Lexer/TokenEmulator/FnTokenEmulator.php

@@ -3,22 +3,9 @@
 namespace PhpParser\Lexer\TokenEmulator;
 
 use PhpParser\Lexer\Emulative;
-use PhpParser\Parser\Tokens;
 
 final class FnTokenEmulator implements TokenEmulatorInterface
 {
-    const T_FN = 1008;
-
-    public function getTokenId(): int
-    {
-        return self::T_FN;
-    }
-
-    public function getParserTokenId(): int
-    {
-        return Tokens::T_FN;
-    }
-
     public function isEmulationNeeded(string $code) : bool
     {
         // skip version where this is supported
@@ -40,7 +27,7 @@ final class FnTokenEmulator implements TokenEmulatorInterface
                     continue;
                 }
 
-                $tokens[$i][0] = self::T_FN;
+                $tokens[$i][0] = Emulative::T_FN;
             }
         }
 

lib/PhpParser/Lexer/TokenEmulator/NumericLiteralSeparatorEmulator.php

@@ -0,0 +1,98 @@
+<?php declare(strict_types=1);
+
+namespace PhpParser\Lexer\TokenEmulator;
+
+use PhpParser\Lexer\Emulative;
+
+final class NumericLiteralSeparatorEmulator implements TokenEmulatorInterface
+{
+    const BIN = '(?:0b[01]+(?:_[01]+)*)';
+    const HEX = '(?:0x[0-9a-f]+(?:_[0-9a-f]+)*)';
+    const DEC = '(?:[0-9]+(?:_[0-9]+)*)';
+    const SIMPLE_FLOAT = '(?:' . self::DEC . '\.' . self::DEC . '?|\.' . self::DEC . ')';
+    const EXP = '(?:e[+-]?' . self::DEC . ')';
+    const FLOAT = '(?:' . self::SIMPLE_FLOAT . self::EXP . '?|' . self::DEC . self::EXP . ')';
+    const NUMBER = '~' . self::FLOAT . '|' . self::BIN . '|' . self::HEX . '|' . self::DEC . '~iA';
+
+    public function isEmulationNeeded(string $code) : bool
+    {
+        // skip version where this is supported
+        if (version_compare(\PHP_VERSION, Emulative::PHP_7_4, '>=')) {
+            return false;
+        }
+
+        return preg_match('~[0-9a-f]_[0-9a-f]~i', $code) !== false;
+    }
+
+    public function emulate(string $code, array $tokens): array
+    {
+        // We need to manually iterate and manage a count because we'll change
+        // the tokens array on the way
+        $codeOffset = 0;
+        for ($i = 0, $c = count($tokens); $i < $c; ++$i) {
+            $token = $tokens[$i];
+            $tokenLen = \strlen(\is_array($token) ? $token[1] : $token);
+
+            if ($token[0] !== T_LNUMBER && $token[0] !== T_DNUMBER) {
+                $codeOffset += $tokenLen;
+                continue;
+            }
+
+            $res = preg_match(self::NUMBER, $code, $matches, 0, $codeOffset);
+            assert($res, "No number at number token position");
+
+            $match = $matches[0];
+            $matchLen = \strlen($match);
+            if ($matchLen === $tokenLen) {
+                // Original token already holds the full number.
+                $codeOffset += $tokenLen;
+                continue;
+            }
+
+            $tokenKind = $this->resolveIntegerOrFloatToken($match);
+            $newTokens = [[$tokenKind, $match, $token[2]]];
+
+            $numTokens = 1;
+            $len = $tokenLen;
+            while ($matchLen > $len) {
+                $nextToken = $tokens[$i + $numTokens];
+                $nextTokenText = \is_array($nextToken) ? $nextToken[1] : $nextToken;
+                $nextTokenLen = \strlen($nextTokenText);
+
+                $numTokens++;
+                if ($matchLen < $len + $nextTokenLen) {
+                    // Split trailing characters into a partial token.
+                    assert(is_array($nextToken), "Partial token should be an array token");
+                    $partialText = substr($nextTokenText, $matchLen - $len);
+                    $newTokens[] = [$nextToken[0], $partialText, $nextToken[2]];
+                    break;
+                }
+
+                $len += $nextTokenLen;
+            }
+
+            array_splice($tokens, $i, $numTokens, $newTokens);
+            $c -= $numTokens - \count($newTokens);
+            $codeOffset += $matchLen;
+        }
+
+        return $tokens;
+    }
+
+    private function resolveIntegerOrFloatToken(string $str): int
+    {
+        $str = str_replace('_', '', $str);
+
+        if (stripos($str, '0b') === 0) {
+            $num = bindec($str);
+        } elseif (stripos($str, '0x') === 0) {
+            $num = hexdec($str);
+        } elseif (stripos($str, '0') === 0 && ctype_digit($str)) {
+            $num = octdec($str);
+        } else {
+            $num = +$str;
+        }
+
+        return is_float($num) ? T_DNUMBER : T_LNUMBER;
+    }
+}

lib/PhpParser/Lexer/TokenEmulator/TokenEmulatorInterface.php

@@ -2,12 +2,9 @@
 
 namespace PhpParser\Lexer\TokenEmulator;
 
+/** @internal */
 interface TokenEmulatorInterface
 {
-    public function getTokenId(): int;
-
-    public function getParserTokenId(): int;
-
     public function isEmulationNeeded(string $code): bool;
 
     /**

lib/PhpParser/Node/Scalar/DNumber.php

@@ -34,6 +34,8 @@ class DNumber extends Scalar
      * @return float The parsed number
      */
     public static function parse(string $str) : float {
+        $str = str_replace('_', '', $str);
+
         // if string contains any of .eE just cast it to float
         if (false !== strpbrk($str, '.eE')) {
             return (float) $str;

lib/PhpParser/Node/Scalar/LNumber.php

@@ -41,6 +41,8 @@ class LNumber extends Scalar
      * @return LNumber The constructed LNumber, including kind attribute
      */
     public static function fromString(string $str, array $attributes = [], bool $allowInvalidOctal = false) : LNumber {
+        $str = str_replace('_', '', $str);
+
         if ('0' !== $str[0] || '0' === $str) {
             $attributes['kind'] = LNumber::KIND_DEC;
             return new LNumber((int) $str, $attributes);

lib/PhpParser/Node/Scalar/String_.php

@@ -100,7 +100,7 @@ class String_ extends Scalar
                 if (isset(self::$replacements[$str])) {
                     return self::$replacements[$str];
                 } elseif ('x' === $str[0] || 'X' === $str[0]) {
-                    return chr(hexdec($str));
+                    return chr(hexdec(substr($str, 1)));
                 } elseif ('u' === $str[0]) {
                     return self::codePointToUtf8(hexdec($matches[2]));
                 } else {
@@ -134,7 +134,7 @@ class String_ extends Scalar
         }
         throw new Error('Invalid UTF-8 codepoint escape sequence: Codepoint too large');
     }
-    
+
     public function getType() : string {
         return 'Scalar_String';
     }

lib/PhpParser/Node/Stmt/ClassLike.php

@@ -14,6 +14,45 @@ abstract class ClassLike extends Node\Stmt
     /** @var Node\Stmt[] Statements */
     public $stmts;
 
+    /**
+     * @return TraitUse[]
+     */
+    public function getTraitUses() : array {
+        $traitUses = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof TraitUse) {
+                $traitUses[] = $stmt;
+            }
+        }
+        return $traitUses;
+    }
+
+    /**
+     * @return ClassConst[]
+     */
+    public function getConstants() : array {
+        $constants = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof ClassConst) {
+                $constants[] = $stmt;
+            }
+        }
+        return $constants;
+    }
+
+    /**
+     * @return Property[]
+     */
+    public function getProperties() : array {
+        $properties = [];
+        foreach ($this->stmts as $stmt) {
+            if ($stmt instanceof Property) {
+                $properties[] = $stmt;
+            }
+        }
+        return $properties;
+    }
+
     /**
      * Gets all methods defined directly in this class/interface/trait
      *

lib/PhpParser/Node/Stmt/Class_.php

@@ -98,7 +98,7 @@ class Class_ extends ClassLike
             throw new Error('Cannot use the final modifier on an abstract class member');
         }
     }
-    
+
     public function getType() : string {
         return 'Stmt_Class';
     }

lib/PhpParser/Node/Stmt/Interface_.php

@@ -28,7 +28,7 @@ class Interface_ extends ClassLike
     public function getSubNodeNames() : array {
         return ['name', 'extends', 'stmts'];
     }
-    
+
     public function getType() : string {
         return 'Stmt_Interface';
     }

lib/PhpParser/Node/Stmt/Trait_.php

@@ -23,7 +23,7 @@ class Trait_ extends ClassLike
     public function getSubNodeNames() : array {
         return ['name', 'stmts'];
     }
-    
+
     public function getType() : string {
         return 'Stmt_Trait';
     }

lib/PhpParser/Node/Stmt/TryCatch.php

@@ -18,7 +18,7 @@ class TryCatch extends Node\Stmt
      *
      * @param Node\Stmt[]   $stmts      Statements
      * @param Catch_[]      $catches    Catches
-     * @param null|Finally_ $finally    Optionaly finally node
+     * @param null|Finally_ $finally    Optional finally node
      * @param array         $attributes Additional attributes
      */
     public function __construct(array $stmts, array $catches, Finally_ $finally = null, array $attributes = []) {
@@ -31,7 +31,7 @@ class TryCatch extends Node\Stmt
     public function getSubNodeNames() : array {
         return ['stmts', 'catches', 'finally'];
     }
-    
+
     public function getType() : string {
         return 'Stmt_TryCatch';
     }

lib/PhpParser/NodeVisitor/NameResolver.php

@@ -91,6 +91,7 @@ class NameResolver extends NodeVisitorAbstract
             $this->resolveSignature($node);
         } elseif ($node instanceof Stmt\ClassMethod
                   || $node instanceof Expr\Closure
+                  || $node instanceof Expr\ArrowFunction
         ) {
             $this->resolveSignature($node);
         } elseif ($node instanceof Stmt\Property) {

lib/PhpParser/PrettyPrinter/Standard.php

@@ -159,8 +159,13 @@ class Standard extends PrettyPrinterAbstract
             return (string) $node->value;
         }
 
-        $sign = $node->value < 0 ? '-' : '';
-        $str = (string) $node->value;
+        if ($node->value < 0) {
+            $sign = '-';
+            $str = (string) -$node->value;
+        } else {
+            $sign = '';
+            $str = (string) $node->value;
+        }
         switch ($kind) {
             case Scalar\LNumber::KIND_BIN:
                 return $sign . '0b' . base_convert($str, 10, 2);

test/PhpParser/Builder/InterfaceTest.php

@@ -9,11 +9,8 @@ use PhpParser\Node\Stmt;
 
 class InterfaceTest extends \PHPUnit\Framework\TestCase
 {
-    /** @var Interface_ */
-    protected $builder;
-
-    protected function setUp() {
-        $this->builder = new Interface_('Contract');
+    protected function createInterfaceBuilder() {
+        return new Interface_('Contract');
     }
 
     private function dump($node) {
@@ -22,13 +19,14 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     }
 
     public function testEmpty() {
-        $contract = $this->builder->getNode();
+        $contract = $this->createInterfaceBuilder()->getNode();
         $this->assertInstanceOf(Stmt\Interface_::class, $contract);
         $this->assertEquals(new Node\Identifier('Contract'), $contract->name);
     }
 
     public function testExtending() {
-        $contract = $this->builder->extend('Space\Root1', 'Root2')->getNode();
+        $contract = $this->createInterfaceBuilder()
+            ->extend('Space\Root1', 'Root2')->getNode();
         $this->assertEquals(
             new Stmt\Interface_('Contract', [
                 'extends' => [
@@ -41,7 +39,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
 
     public function testAddMethod() {
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder->addStmt($method)->getNode();
+        $contract = $this->createInterfaceBuilder()->addStmt($method)->getNode();
         $this->assertSame([$method], $contract->stmts);
     }
 
@@ -49,7 +47,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
         $const = new Stmt\ClassConst([
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458.0))
         ]);
-        $contract = $this->builder->addStmt($const)->getNode();
+        $contract = $this->createInterfaceBuilder()->addStmt($const)->getNode();
         $this->assertSame(299792458.0, $contract->stmts[0]->consts[0]->value->value);
     }
 
@@ -58,7 +56,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458))
         ]);
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder
+        $contract = $this->createInterfaceBuilder()
             ->addStmt($method)
             ->addStmt($const)
             ->getNode()
@@ -69,7 +67,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     }
 
     public function testDocComment() {
-        $node = $this->builder
+        $node = $this->createInterfaceBuilder()
             ->setDocComment('/** Test */')
             ->getNode();
 
@@ -81,7 +79,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
     public function testInvalidStmtError() {
         $this->expectException(\LogicException::class);
         $this->expectExceptionMessage('Unexpected node of type "Stmt_PropertyProperty"');
-        $this->builder->addStmt(new Stmt\PropertyProperty('invalid'));
+        $this->createInterfaceBuilder()->addStmt(new Stmt\PropertyProperty('invalid'));
     }
 
     public function testFullFunctional() {
@@ -89,7 +87,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new Node\Const_('SPEED_OF_LIGHT', new DNumber(299792458))
         ]);
         $method = new Stmt\ClassMethod('doSomething');
-        $contract = $this->builder
+        $contract = $this->createInterfaceBuilder()
             ->addStmt($method)
             ->addStmt($const)
             ->getNode()

test/PhpParser/Builder/TraitTest.php

@@ -5,6 +5,12 @@ namespace PhpParser\Builder;
 use PhpParser\Comment;
 use PhpParser\Node\Name;
 use PhpParser\Node\Stmt;
+use PhpParser\Node\Stmt\Class_;
+use PhpParser\Node\Stmt\ClassConst;
+use PhpParser\Node\Stmt\ClassMethod;
+use PhpParser\Node\Stmt\Property;
+use PhpParser\Node\Stmt\PropertyProperty;
+use PhpParser\Node\Stmt\TraitUse;
 
 class TraitTest extends \PHPUnit\Framework\TestCase
 {
@@ -43,4 +49,43 @@ class TraitTest extends \PHPUnit\Framework\TestCase
             ->addStmt(new Stmt\Echo_([]))
         ;
     }
+
+    public function testGetMethods() {
+        $methods = [
+            new ClassMethod('foo'),
+            new ClassMethod('bar'),
+            new ClassMethod('fooBar'),
+        ];
+        $trait = new Stmt\Trait_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $methods[0],
+                new ClassConst([]),
+                $methods[1],
+                new Property(0, []),
+                $methods[2],
+            ]
+        ]);
+
+        $this->assertSame($methods, $trait->getMethods());
+    }
+
+    public function testGetProperties()
+    {
+        $properties = [
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo')]),
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('bar')]),
+        ];
+        $trait = new Stmt\Trait_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $properties[0],
+                new ClassConst([]),
+                $properties[1],
+                new ClassMethod('fooBar'),
+            ]
+        ]);
+
+        $this->assertSame($properties, $trait->getProperties());
+    }
 }

test/PhpParser/Lexer/EmulativeTest.php

@@ -123,10 +123,6 @@ class EmulativeTest extends LexerTest
 
     public function provideTestLexNewFeatures() {
         return [
-            // PHP 7.4
-            ['??=', [
-                [Tokens::T_COALESCE_EQUAL, '??='],
-            ]],
             ['yield from', [
                 [Tokens::T_YIELD_FROM, 'yield from'],
             ]],
@@ -169,7 +165,7 @@ class EmulativeTest extends LexerTest
                 [ord(';'), ';'],
             ]],
 
-            // Flexible heredoc/nowdoc
+            // PHP 7.3: Flexible heredoc/nowdoc
             ["<<<LABEL\nLABEL,", [
                 [Tokens::T_START_HEREDOC, "<<<LABEL\n"],
                 [Tokens::T_END_HEREDOC, "LABEL"],
@@ -205,6 +201,58 @@ class EmulativeTest extends LexerTest
                 [Tokens::T_END_HEREDOC, "    LABEL"],
                 [Tokens::T_STRING, "LABEL"],
             ]],
+
+            // PHP 7.4: Null coalesce equal
+            ['??=', [
+                [Tokens::T_COALESCE_EQUAL, '??='],
+            ]],
+
+            // PHP 7.4: Number literal separator
+            ['1_000', [
+                [Tokens::T_LNUMBER, '1_000'],
+            ]],
+            ['0xCAFE_F00D', [
+                [Tokens::T_LNUMBER, '0xCAFE_F00D'],
+            ]],
+            ['0b0101_1111', [
+                [Tokens::T_LNUMBER, '0b0101_1111'],
+            ]],
+            ['0137_041', [
+                [Tokens::T_LNUMBER, '0137_041'],
+            ]],
+            ['1_000.0', [
+                [Tokens::T_DNUMBER, '1_000.0'],
+            ]],
+            ['1_0.0', [
+                [Tokens::T_DNUMBER, '1_0.0']
+            ]],
+            ['1_000_000_000.0', [
+                [Tokens::T_DNUMBER, '1_000_000_000.0']
+            ]],
+            ['0e1_0', [
+                [Tokens::T_DNUMBER, '0e1_0']
+            ]],
+            ['1_0e+10', [
+                [Tokens::T_DNUMBER, '1_0e+10']
+            ]],
+            ['1_0e-10', [
+                [Tokens::T_DNUMBER, '1_0e-10']
+            ]],
+            ['0b1011010101001010_110101010010_10101101010101_0101101011001_110111100', [
+                [Tokens::T_DNUMBER, '0b1011010101001010_110101010010_10101101010101_0101101011001_110111100'],
+            ]],
+            ['0xFFFF_FFFF_FFFF_FFFF', [
+                [Tokens::T_DNUMBER, '0xFFFF_FFFF_FFFF_FFFF'],
+            ]],
+            ['1_000+1', [
+                [Tokens::T_LNUMBER, '1_000'],
+                [ord('+'), '+'],
+                [Tokens::T_LNUMBER, '1'],
+            ]],
+            ['1_0abc', [
+                [Tokens::T_LNUMBER, '1_0'],
+                [Tokens::T_STRING, 'abc'],
+            ]],
         ];
     }
 }

test/PhpParser/Node/Stmt/ClassTest.php

@@ -2,6 +2,8 @@
 
 namespace PhpParser\Node\Stmt;
 
+use PhpParser\Node\Scalar\String_;
+
 class ClassTest extends \PHPUnit\Framework\TestCase
 {
     public function testIsAbstract() {
@@ -20,6 +22,22 @@ class ClassTest extends \PHPUnit\Framework\TestCase
         $this->assertFalse($class->isFinal());
     }
 
+    public function testGetTraitUses() {
+        $traitUses = [
+            new TraitUse([new Trait_('foo')]),
+            new TraitUse([new Trait_('bar')]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                $traitUses[0],
+                new ClassMethod('fooBar'),
+                $traitUses[1],
+            ]
+        ]);
+
+        $this->assertSame($traitUses, $class->getTraitUses());
+    }
+
     public function testGetMethods() {
         $methods = [
             new ClassMethod('foo'),
@@ -40,6 +58,42 @@ class ClassTest extends \PHPUnit\Framework\TestCase
         $this->assertSame($methods, $class->getMethods());
     }
 
+    public function testGetConstants() {
+        $constants = [
+            new ClassConst([new \PhpParser\Node\Const_('foo', new String_('foo_value'))]),
+            new ClassConst([new \PhpParser\Node\Const_('bar', new String_('bar_value'))]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $constants[0],
+                new ClassMethod('fooBar'),
+                $constants[1],
+            ]
+        ]);
+
+        $this->assertSame($constants, $class->getConstants());
+    }
+
+    public function testGetProperties()
+    {
+        $properties = [
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('foo')]),
+            new Property(Class_::MODIFIER_PUBLIC, [new PropertyProperty('bar')]),
+        ];
+        $class = new Class_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $properties[0],
+                new ClassConst([]),
+                $properties[1],
+                new ClassMethod('fooBar'),
+            ]
+        ]);
+
+        $this->assertSame($properties, $class->getProperties());
+    }
+
     public function testGetMethod() {
         $methodConstruct = new ClassMethod('__CONSTRUCT');
         $methodTest = new ClassMethod('test');

test/PhpParser/Node/Stmt/InterfaceTest.php

@@ -3,6 +3,7 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
+use PhpParser\Node\Scalar\String_;
 
 class InterfaceTest extends \PHPUnit\Framework\TestCase
 {
@@ -11,7 +12,7 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
             new ClassMethod('foo'),
             new ClassMethod('bar'),
         ];
-        $interface = new Class_('Foo', [
+        $interface = new Interface_('Foo', [
             'stmts' => [
                 new Node\Stmt\ClassConst([new Node\Const_('C1', new Node\Scalar\String_('C1'))]),
                 $methods[0],
@@ -23,4 +24,21 @@ class InterfaceTest extends \PHPUnit\Framework\TestCase
 
         $this->assertSame($methods, $interface->getMethods());
     }
+
+    public function testGetConstants() {
+        $constants = [
+            new ClassConst([new \PhpParser\Node\Const_('foo', new String_('foo_value'))]),
+            new ClassConst([new \PhpParser\Node\Const_('bar', new String_('bar_value'))]),
+        ];
+        $class = new Interface_('Foo', [
+            'stmts' => [
+                new TraitUse([]),
+                $constants[0],
+                new ClassMethod('fooBar'),
+                $constants[1],
+            ]
+        ]);
+
+        $this->assertSame($constants, $class->getConstants());
+    }
 }

test/PhpParser/NodeTraverserTest.php

@@ -266,13 +266,17 @@ class NodeTraverserTest extends \PHPUnit\Framework\TestCase
         $traverser->addVisitor($visitor2);
         $traverser->addVisitor($visitor3);
 
+        $getVisitors = (function () {
+            return $this->visitors;
+        })->bindTo($traverser, NodeTraverser::class);
+
         $preExpected = [$visitor1, $visitor2, $visitor3];
-        $this->assertAttributeSame($preExpected, 'visitors', $traverser, 'The appropriate visitors have not been added');
+        $this->assertSame($preExpected, $getVisitors());
 
         $traverser->removeVisitor($visitor2);
 
         $postExpected = [0 => $visitor1, 2 => $visitor3];
-        $this->assertAttributeSame($postExpected, 'visitors', $traverser, 'The appropriate visitors are not present after removal');
+        $this->assertSame($postExpected, $getVisitors());
     }
 
     public function testNoCloneNodes() {

test/PhpParser/NodeVisitor/NameResolverTest.php

@@ -219,6 +219,10 @@ function(A $a) : A {};
 function fn3(?A $a) : ?A {}
 function fn4(?array $a) : ?array {}
 
+fn(array $a): array => $a;
+fn(A $a): A => $a;
+fn(?A $a): ?A => $a;
+
 A::b();
 A::$b;
 A::B;
@@ -263,6 +267,9 @@ function fn3(?\NS\A $a) : ?\NS\A
 function fn4(?array $a) : ?array
 {
 }
+fn(array $a): array => $a;
+fn(\NS\A $a): \NS\A => $a;
+fn(?\NS\A $a): ?\NS\A => $a;
 \NS\A::b();
 \NS\A::$b;
 \NS\A::B;

test/code/parser/errorHandling/lexerErrors.test

@@ -32,24 +32,25 @@ $a = 42;
 @@{ "\1" }@@
 $b = 24;
 -----
+!!positions
 Unexpected character "
" (ASCII 1) from 4:1 to 4:1
 array(
-    0: Stmt_Expression(
-        expr: Expr_Assign(
-            var: Expr_Variable(
+    0: Stmt_Expression[3:1 - 3:8](
+        expr: Expr_Assign[3:1 - 3:7](
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:7](
                 value: 42
             )
         )
     )
-    1: Stmt_Expression(
-        expr: Expr_Assign(
-            var: Expr_Variable(
+    1: Stmt_Expression[5:1 - 5:8](
+        expr: Expr_Assign[5:1 - 5:7](
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:7](
                 value: 24
             )
         )
@@ -62,24 +63,25 @@ $a = 42;
 @@{ "\0" }@@
 $b = 24;
 -----
+!!positions
 Unexpected null byte from 4:1 to 4:1
 array(
-    0: Stmt_Expression(
-        expr: Expr_Assign(
-            var: Expr_Variable(
+    0: Stmt_Expression[3:1 - 3:8](
+        expr: Expr_Assign[3:1 - 3:7](
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:7](
                 value: 42
             )
         )
     )
-    1: Stmt_Expression(
-        expr: Expr_Assign(
-            var: Expr_Variable(
+    1: Stmt_Expression[5:1 - 5:8](
+        expr: Expr_Assign[5:1 - 5:7](
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:7](
                 value: 24
             )
         )
@@ -94,35 +96,36 @@ $b = 2;
 @@{ "\2" }@@
 $c = 3;
 -----
-Unexpected character "@@{ "\1" }@@" (ASCII 1) from 4:1 to 4:1
-Unexpected character "@@{ "\2" }@@" (ASCII 2) from 6:1 to 6:1
+!!positions
+Unexpected character "
" (ASCII 1) from 4:1 to 4:1
+Unexpected character "" (ASCII 2) from 6:1 to 6:1
 array(
-    0: Stmt_Expression(
-        expr: Expr_Assign(
-            var: Expr_Variable(
+    0: Stmt_Expression[3:1 - 3:7](
+        expr: Expr_Assign[3:1 - 3:6](
+            var: Expr_Variable[3:1 - 3:2](
                 name: a
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[3:6 - 3:6](
                 value: 1
             )
         )
     )
-    1: Stmt_Expression(
-        expr: Expr_Assign(
-            var: Expr_Variable(
+    1: Stmt_Expression[5:1 - 5:7](
+        expr: Expr_Assign[5:1 - 5:6](
+            var: Expr_Variable[5:1 - 5:2](
                 name: b
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[5:6 - 5:6](
                 value: 2
             )
         )
     )
-    2: Stmt_Expression(
-        expr: Expr_Assign(
-            var: Expr_Variable(
+    2: Stmt_Expression[7:1 - 7:7](
+        expr: Expr_Assign[7:1 - 7:6](
+            var: Expr_Variable[7:1 - 7:2](
                 name: c
             )
-            expr: Scalar_LNumber(
+            expr: Scalar_LNumber[7:6 - 7:6](
                 value: 3
             )
         )

test/code/parser/scalar/numberSeparators.test

@@ -0,0 +1,199 @@
+Different integer syntaxes
+-----
+<?php
+
+6.674_083e-11;
+299_792_458;
+0xCAFE_F00D;
+0b0101_1111;
+0137_041;
+
+// already a valid constant name
+_100;
+
+// syntax errors
+100_;
+1__1;
+1_.0;
+1._0;
+0x_123;
+0b_101;
+1_e2;
+1e_2;
+-----
+Syntax error, unexpected T_STRING from 13:4 to 13:4
+Syntax error, unexpected T_STRING from 14:2 to 14:4
+Syntax error, unexpected T_STRING from 15:2 to 15:2
+Syntax error, unexpected T_STRING from 16:3 to 16:4
+Syntax error, unexpected T_STRING from 17:2 to 17:6
+Syntax error, unexpected T_STRING from 18:2 to 18:6
+Syntax error, unexpected T_STRING from 19:2 to 19:4
+Syntax error, unexpected T_STRING from 20:2 to 20:4
+array(
+    0: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 6.674083E-11
+        )
+    )
+    1: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 299792458
+        )
+    )
+    2: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 3405705229
+        )
+    )
+    3: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 95
+        )
+    )
+    4: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 48673
+        )
+    )
+    5: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _100
+                )
+                comments: array(
+                    0: // already a valid constant name
+                )
+            )
+            comments: array(
+                0: // already a valid constant name
+            )
+        )
+        comments: array(
+            0: // already a valid constant name
+        )
+    )
+    6: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 100
+            comments: array(
+                0: // syntax errors
+            )
+        )
+        comments: array(
+            0: // syntax errors
+        )
+    )
+    7: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _
+                )
+            )
+        )
+    )
+    8: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    9: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: __1
+                )
+            )
+        )
+    )
+    10: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    11: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _
+                )
+            )
+        )
+    )
+    12: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 0
+        )
+    )
+    13: Stmt_Expression(
+        expr: Scalar_DNumber(
+            value: 1
+        )
+    )
+    14: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _0
+                )
+            )
+        )
+    )
+    15: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 0
+        )
+    )
+    16: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: x_123
+                )
+            )
+        )
+    )
+    17: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 0
+        )
+    )
+    18: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: b_101
+                )
+            )
+        )
+    )
+    19: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    20: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: _e2
+                )
+            )
+        )
+    )
+    21: Stmt_Expression(
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    22: Stmt_Expression(
+        expr: Expr_ConstFetch(
+            name: Name(
+                parts: array(
+                    0: e_2
+                )
+            )
+        )
+    )
+)

test_old/run.php

@@ -231,6 +231,8 @@ foreach (new RecursiveIteratorIterator(
         echo $file, ":\n    Parse failed with message: {$e->getMessage()}\n";
 
         ++$parseFail;
+    } catch (Throwable $e) {
+        echo $file, ":\n    Unknown error occurred: $e\n";
     }
 }