Release PHP-Parser 3.0.3

Conflicts:
	lib/PhpParser/NodeVisitor.php

CHANGELOG.md

@@ -1,8 +1,120 @@
-Version 3.0.0-dev
+Version 3.0.4-dev
 -----------------
 
 Nothing yet.
 
+Version 3.0.3 (2017-02-03)
+--------------------------
+
+### Fixed
+
+* In `"$foo[0]"` the `0` is now parsed as an `LNumber` rather than `String`. (#325)
+* Ensure integers and floats are always pretty printed preserving semantics, even if the particular
+  value can only be manually constructed.
+* Throw a `LogicException` when trying to pretty-print an `Error` node. Previously this resulted in
+  an undefined method exception or fatal error.
+
+### Added
+
+* [PHP 7.1] Added support for negative interpolated offsets: `"$foo[-1]"`
+* Added `preserveOriginalNames` option to `NameResolver`. If this option is enabled, an
+  `originalName` attribute, containing the unresolved name, will be added to each resolved name.
+* Added `php-parse --with-positions` option, which dumps nodes with position information.
+
+### Deprecated
+
+* The XML serializer has been deprecated. In particular, the classes `Serializer\XML`,
+  `Unserializer\XML`, as well as the interfaces `Serializer` and `Unserializer` are deprecated.
+
+Version 3.0.2 (2016-12-06)
+--------------------------
+
+### Fixed
+
+* Fixed name resolution of nullable types. (#324)
+* Fixed pretty-printing of nullable types.
+
+Version 3.0.1 (2016-12-01)
+--------------------------
+
+### Fixed
+
+* Fixed handling of nested `list()`s: If the nested list was unkeyed, it was directly included in
+  the list items. If it was keyed, it was wrapped in `ArrayItem`. Now nested `List_` nodes are
+  always wrapped in `ArrayItem`s. (#321)
+
+Version 3.0.0 (2016-11-30)
+--------------------------
+
+### Added
+
+* Added support for dumping node positions in the NodeDumper through the `dumpPositions` option.
+* Added error recovery support for `$`, `new`, `Foo::`.
+
+Version 3.0.0-beta2 (2016-10-29)
+--------------------------------
+
+This release primarily improves our support for error recovery.
+
+### Added
+
+* Added `Node::setDocComment()` method.
+* Added `Error::getMessageWithColumnInfo()` method.
+* Added support for recovery from lexer errors.
+* Added support for recovering from "special" errors (i.e. non-syntax parse errors).
+* Added precise location information for lexer errors.
+* Added `ErrorHandler` interface, and `ErrorHandler\Throwing` and `ErrorHandler\Collecting` as
+  specific implementations. These provide a general mechanism for handling error recovery.
+* Added optional `ErrorHandler` argument to `Parser::parse()`, `Lexer::startLexing()` and
+  `NameResolver::__construct()`.
+* The `NameResolver` now adds a `namespacedName` attribute on name nodes that cannot be statically
+  resolved (unqualified unaliased function or constant names in namespaces).
+  
+### Fixed
+
+* Fixed attribute assignment for `GroupUse` prefix and variables in interpolated strings.
+
+### Changed
+
+* The constants on `NameTraverserInterface` have been moved into the `NameTraverser` class.
+* Due to the error handling changes, the `Parser` interface and `Lexer` API have changed.
+* The emulative lexer now directly postprocesses tokens, instead of using `~__EMU__~` sequences.
+  This changes the protected API of the lexer.
+* The `Name::slice()` method now returns `null` for empty slices, previously `new Name([])` was
+  used. `Name::concat()` now also supports concatenation with `null`.
+
+### Removed
+
+* Removed `Name::append()` and `Name::prepend()`. These mutable methods have been superseded by
+  the immutable `Name::concat()`.
+* Removed `Error::getRawLine()` and `Error::setRawLine()`. These methods have been superseded by
+  `Error::getStartLine()` and `Error::setStartLine()`.
+* Removed support for node cloning in the `NodeTraverser`.
+* Removed `$separator` argument from `Name::toString()`.
+* Removed `throw_on_error` parser option and `Parser::getErrors()` method. Use the `ErrorHandler`
+  mechanism instead.
+
+Version 3.0.0-beta1 (2016-09-16)
+--------------------------------
+
+### Added
+
+* [7.1] Function/method and parameter builders now support PHP 7.1 type hints (void, iterable and
+  nullable types).
+* Nodes and Comments now implement `JsonSerializable`. The node kind is stored in a `nodeType`
+  property.
+* The `InlineHTML` node now has an `hasLeadingNewline` attribute, that specifies whether the
+  preceding closing tag contained a newline. The pretty printer honors this attribute.
+* Partial parsing of `$obj->` (with missing property name) is now supported in error recovery mode.
+* The error recovery mode is now exposed in the `php-parse` script through the `--with-recovery`
+  or `-r` flags.
+
+The following changes are also part of PHP-Parser 2.1.1:
+
+* The PHP 7 parser will now generate a parse error for `$var =& new Obj` assignments.
+* Comments on free-standing code blocks will now be retained as comments on the first statement in
+  the code block.
+
 Version 3.0.0-alpha1 (2016-07-25)
 ---------------------------------
 
@@ -56,6 +168,19 @@ Additionally the following changes were made:
 * The deprecated `Comment::setLine()` and `Comment::setText()` methods have been removed.
 * The deprecated `Name::set()`, `Name::setFirst()` and `Name::setLast()` methods have been removed.
 
+Version 2.1.1 (2016-09-16)
+--------------------------
+
+### Changed
+
+* The pretty printer will now escape all control characters in the range `\x00-\x1F` inside double
+  quoted strings. If no special escape sequence is available, an octal escape will be used.
+* The quality of the error recovery has been improved. In particular unterminated expressions should
+  be handled more gracefully.
+* The PHP 7 parser will now generate a parse error for `$var =& new Obj` assignments.
+* Comments on free-standing code blocks will no be retained as comments on the first statement in
+  the code block.
+
 Version 2.1.0 (2016-04-19)
 --------------------------
 

README.md

@@ -6,7 +6,9 @@ PHP Parser
 This is a PHP 5.2 to PHP 7.1 parser written in PHP. Its purpose is to simplify static code analysis and
 manipulation.
 
-[**Documentation for version 2.x**][doc_master] (stable; for running on PHP >= 5.4; for parsing PHP 5.2 to PHP 7.0).
+[**Documentation for version 3.x**][doc_master] (stable; for running on PHP >= 5.5; for parsing PHP 5.2 to PHP 7.1).
+
+[Documentation for version 2.x][doc_2_x] (stable; for running on PHP >= 5.4; for parsing PHP 5.2 to PHP 7.0).
 
 [Documentation for version 1.x][doc_1_x] (unsupported; for running on PHP >= 5.3; for parsing PHP 5.2 to PHP 5.6).
 
@@ -89,7 +91,7 @@ Documentation
 
 Component documentation:
 
- 1. [Error](doc/component/Error.markdown)
+ 1. [Error handling](doc/component/Error_handling.markdown)
  2. [Lexer](doc/component/Lexer.markdown)
 
  [doc_1_x]: https://github.com/nikic/PHP-Parser/tree/1.x/doc

UPGRADE-3.0.md

@@ -1,8 +1,13 @@
 Upgrading from PHP-Parser 2.x to 3.0
 ====================================
 
-This version does not include any major API changes. Only specific details of the node
+The backwards-incompatible changes in this release may be summarized as follows:
-representation have changed in some cases.
+
+ * The specific details of the node representation have changed in some cases, primarily to
+   accomodate new PHP 7.1 features.
+ * There have been significant changes to the error recovery implementation. This may affect you,
+   if you used the error recovery mode or have a custom lexer implementation.
+ * A number of deprecated methods were removed.
 
 ### PHP version requirements
 
@@ -31,18 +36,125 @@ The following changes are unlikely to require code changes:
  * `void` and `iterable` types are now stored as strings if the PHP 7 parser is used. Previously
    these would have been represented as `Name` instances.
 
-### Removed methods
+### Changes to error recovery mode
 
-The following methods have been removed:
+Previously, error recovery mode was enabled by setting the `throwOnError` option to `false` when
+creating the parser, while collected errors were retrieved using the `getErrors()` method:
+
+```php
+$lexer = ...;
+$parser = (new ParserFactory)->create(ParserFactor::ONLY_PHP7, $lexer, [
+    'throwOnError' => true,
+]);
+
+$stmts = $parser->parse($code);
+$errors = $parser->getErrors();
+if ($errors) {
+    handleErrors($errors);
+}
+processAst($stmts);
+```
+
+Both the `throwOnError` option and the `getErrors()` method have been removed in PHP-Parser 3.0.
+Instead an instance of `ErrorHandler\Collecting` should be passed to the `parse()` method:
+
+```php
+$lexer = ...;
+$parser = (new ParserFactory)->create(ParserFactor::ONLY_PHP7, $lexer);
+
+$errorHandler = new ErrorHandler\Collecting;
+$stmts = $parser->parse($code, $errorHandler);
+if ($errorHandler->hasErrors()) {
+    handleErrors($errorHandler->getErrors());
+}
+processAst($stmts);
+```
+
+#### Multiple parser fallback in error recovery mode
+
+As a result of this change, if a `Multiple` parser is used (e.g. through the `ParserFactory` using
+`PREFER_PHP7` or `PREFER_PHP5`), it will now return the result of the first *non-throwing* parse. As
+parsing never throws in error recovery mode, the result from the first parser will always be
+returned.
+
+The PHP 7 parser is a superset of the PHP 5 parser, with the exceptions that `=& new` and
+`global $$foo->bar` are not supported (other differences are in representation only). The PHP 7
+parser will be able to recover from the error in both cases. For this reason, this change will
+likely pass unnoticed if you do not specifically test for this syntax.
+
+It is possible to restore the precise previous behavior with the following code:
+
+```php
+$lexer = ...;
+$parser7 = new Parser\Php7($lexer);
+$parser5 = new Parser\Php5($lexer);
+
+$errors7 = new ErrorHandler\Collecting();
+$stmts7 = $parser7->parse($code, $errors7);
+if ($errors7->hasErrors()) {
+    $errors5 = new ErrorHandler\Collecting();
+    $stmts5 = $parser5->parse($code, $errors5);
+    if (!$errors5->hasErrors()) {
+        // If PHP 7 parse has errors but PHP 5 parse has no errors, use PHP 5 result
+        return [$stmts5, $errors5];
+    }
+}
+// If PHP 7 succeeds or both fail use PHP 7 result
+return [$stmts7, $errors7];
+```
+
+#### Error handling in the lexer
+
+In order to support recovery from lexer errors, the signature of the `startLexing()` method changed
+to optionally accept an `ErrorHandler`:
+
+```php
+// OLD
+public function startLexing($code);
+// NEW
+public function startLexing($code, ErrorHandler $errorHandler = null);
+```
+
+If you use a custom lexer with overriden `startLexing()` method, it needs to be changed to accept
+the extra parameter. The value should be passed on to the parent method.
+
+#### Error checks in node constructors
+
+The constructors of certain nodes used to contain additional checks for semantic errors, such as
+creating a try block without either catch or finally. These checks have been moved from the node
+constructors into the parser. This allows recovery from such errors, as well as representing the
+resulting (invalid) AST.
+
+This means that certain error conditions are no longer checked for manually constructed nodes.
+
+### Removed methods, arguments, options
+
+The following methods, arguments or options have been removed:
 
  * `Comment::setLine()`, `Comment::setText()`: Create new `Comment` instances instead.
- * `Name::set()`, `Name::setFirst()`, `Name::setLast()`: Create new `Name` instances instead. For
+ * `Name::set()`, `Name::setFirst()`, `Name::setLast()`, `Name::append()`, `Name::prepend()`:
-   the latter two a combination of `Name::concat()` and `Name::slice()` can be used.
+    Use `Name::concat()` in combination with `Name::slice()` instead.
+ * `Error::getRawLine()`, `Error::setRawLine()`. Use `Error::getStartLine()` and
+   `Error::setStartLine()` instead.
+ * `Parser::getErrors()`. Use `ErrorHandler\Collecting` instead.
+ * `$separator` argument of `Name::toString()`. Use `strtr()` instead, if you really need it.
+ * `$cloneNodes` argument of `NodeTraverser::__construct()`. Explicitly clone nodes in the visitor
+   instead.
+ * `throwOnError` parser option. Use `ErrorHandler\Collecting` instead.
 
 ### Miscellaneous
 
+ * The `NameResolver` will now resolve unqualified function and constant names in the global
+   namespace into fully qualified names. For example `foo()` in the global namespace resolves to
+   `\foo()`. For names where no static resolution is possible, a `namespacedName` attribute is
+   added now, containing the namespaced variant of the name.
  * All methods on `PrettyPrinter\Standard` are now protected. Previoulsy most of them were public.
    The pretty printer should only be invoked using the `prettyPrint()`, `prettyPrintFile()` and
    `prettyPrintExpr()` methods.
  * The node dumper now prints numeric values that act as enums/flags in a string representation.
    If node dumper results are used in tests, updates may be needed to account for this.
+ * The constants on `NameTraverserInterface` have been moved into the `NameTraverser` class.
+ * The emulative lexer now directly postprocesses tokens, instead of using `~__EMU__~` sequences.
+   This changes the protected API of the emulative lexer.
+ * The `Name::slice()` method now returns `null` for empty slices, previously `new Name([])` was
+   used. `Name::concat()` now also supports concatenation with `null`.

bin/php-parse

@@ -29,8 +29,14 @@ if (empty($files)) {
 $lexer = new PhpParser\Lexer\Emulative(array('usedAttributes' => array(
     'startLine', 'endLine', 'startFilePos', 'endFilePos', 'comments'
 )));
-$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7, $lexer);
+$parser = (new PhpParser\ParserFactory)->create(
-$dumper = new PhpParser\NodeDumper(['dumpComments' => true]);
+    PhpParser\ParserFactory::PREFER_PHP7,
+    $lexer
+);
+$dumper = new PhpParser\NodeDumper([
+    'dumpComments' => true,
+    'dumpPositions' => $attributes['with-positions'],
+]);
 $prettyPrinter = new PhpParser\PrettyPrinter\Standard;
 $serializer = new PhpParser\Serializer\XML;
 
@@ -50,26 +56,29 @@ foreach ($files as $file) {
         echo "====> File $file:\n";
     }
 
+    if ($attributes['with-recovery']) {
+        $errorHandler = new PhpParser\ErrorHandler\Collecting;
+        $stmts = $parser->parse($code, $errorHandler);
+        foreach ($errorHandler->getErrors() as $error) {
+            $message = formatErrorMessage($error, $code, $attributes['with-column-info']);
+            echo $message . "\n";
+        }
+        if (null === $stmts) {
+            continue;
+        }
+    } else {
         try {
             $stmts = $parser->parse($code);
-    } catch (PhpParser\Error $e) {
+        } catch (PhpParser\Error $error) {
-        if ($attributes['with-column-info'] && $e->hasColumnInfo()) {
+            $message = formatErrorMessage($error, $code, $attributes['with-column-info']);
-            $startLine = $e->getStartLine();
-            $endLine = $e->getEndLine();
-            $startColumn = $e->getStartColumn($code);
-            $endColumn   = $e->getEndColumn($code);
-            $message .= $e->getRawMessage() . " from $startLine:$startColumn to $endLine:$endColumn";
-        } else {
-            $message = $e->getMessage();
-        }
-
             die($message . "\n");
         }
+    }
 
     foreach ($operations as $operation) {
         if ('dump' === $operation) {
             echo "==> Node dump:\n";
-            echo $dumper->dump($stmts), "\n";
+            echo $dumper->dump($stmts, $code), "\n";
         } elseif ('pretty-print' === $operation) {
             echo "==> Pretty print:\n";
             echo $prettyPrinter->prettyPrintFile($stmts), "\n";
@@ -86,6 +95,14 @@ foreach ($files as $file) {
     }
 }
 
+function formatErrorMessage(PhpParser\Error $e, $code, $withColumnInfo) {
+    if ($withColumnInfo && $e->hasColumnInfo()) {
+        return $e->getMessageWithColumnInfo($code);
+    } else {
+        return $e->getMessage();
+    }
+}
+
 function showHelp($error = '') {
     if ($error) {
         echo $error . "\n\n";
@@ -103,6 +120,8 @@ Operations is a list of the following options (--dump by default):
         --var-dump          var_dump() nodes (for exact structure)
     -N, --resolve-names     Resolve names using NodeVisitor\NameResolver
     -c, --with-column-info  Show column-numbers for errors (if available)
+    -P, --with-positions    Show positions in node dumps
+    -r, --with-recovery     Use parsing with error recovery
     -h, --help              Display this page
 
 Example:
@@ -120,6 +139,8 @@ function parseArgs($args) {
     $files = array();
     $attributes = array(
         'with-column-info' => false,
+        'with-posititions' => false,
+        'with-recovery' => false,
     );
 
     array_shift($args);
@@ -153,6 +174,14 @@ function parseArgs($args) {
             case '-c';
                 $attributes['with-column-info'] = true;
                 break;
+            case '--with-positions':
+            case '-P':
+                $attributes['with-positions'] = true;
+                break;
+            case '--with-recovery':
+            case '-r':
+                $attributes['with-recovery'] = true;
+                break;
             case '--help':
             case '-h';
                 showHelp();

composer.json

@@ -14,7 +14,7 @@
         "ext-tokenizer": "*"
     },
     "require-dev": {
-        "phpunit/phpunit": "~4.0"
+        "phpunit/phpunit": "~4.0|~5.0"
     },
     "autoload": {
         "psr-4": {

doc/3_Other_node_tree_representations.markdown

@@ -8,7 +8,7 @@ Simple serialization
 
 It is possible to serialize the node tree using `serialize()` and also unserialize it using
 `unserialize()`. The output is not human readable and not easily processable from anything
-but PHP, but it is compact and generates fast. The main application thus is in caching.
+but PHP, but it is compact and generates quickly. The main application thus is in caching.
 
 Human readable dumping
 ----------------------
@@ -86,6 +86,134 @@ array(
 )
 ```
 
+JSON encoding
+-------------
+
+Nodes (and comments) implement the `JsonSerializable` interface. As such, it is possible to JSON
+encode the AST directly using `json_encode()`:
+
+```php
+$code = <<<'CODE'
+<?php
+
+function printLine($msg) {
+    echo $msg, "\n";
+}
+
+printLine('Hello World!!!');
+CODE;
+
+$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7);
+$nodeDumper = new PhpParser\NodeDumper;
+
+try {
+    $stmts = $parser->parse($code);
+
+    echo json_encode($stmts, JSON_PRETTY_PRINT), "\n";
+} catch (PhpParser\Error $e) {
+    echo 'Parse Error: ', $e->getMessage();
+}
+```
+
+This will result in the following output (which includes attributes):
+
+```json
+[
+    {
+        "nodeType": "Stmt_Function",
+        "byRef": false,
+        "name": "printLine",
+        "params": [
+            {
+                "nodeType": "Param",
+                "type": null,
+                "byRef": false,
+                "variadic": false,
+                "name": "msg",
+                "default": null,
+                "attributes": {
+                    "startLine": 3,
+                    "endLine": 3
+                }
+            }
+        ],
+        "returnType": null,
+        "stmts": [
+            {
+                "nodeType": "Stmt_Echo",
+                "exprs": [
+                    {
+                        "nodeType": "Expr_Variable",
+                        "name": "msg",
+                        "attributes": {
+                            "startLine": 4,
+                            "endLine": 4
+                        }
+                    },
+                    {
+                        "nodeType": "Scalar_String",
+                        "value": "\n",
+                        "attributes": {
+                            "startLine": 4,
+                            "endLine": 4,
+                            "kind": 2
+                        }
+                    }
+                ],
+                "attributes": {
+                    "startLine": 4,
+                    "endLine": 4
+                }
+            }
+        ],
+        "attributes": {
+            "startLine": 3,
+            "endLine": 5
+        }
+    },
+    {
+        "nodeType": "Expr_FuncCall",
+        "name": {
+            "nodeType": "Name",
+            "parts": [
+                "printLine"
+            ],
+            "attributes": {
+                "startLine": 7,
+                "endLine": 7
+            }
+        },
+        "args": [
+            {
+                "nodeType": "Arg",
+                "value": {
+                    "nodeType": "Scalar_String",
+                    "value": "Hello World!!!",
+                    "attributes": {
+                        "startLine": 7,
+                        "endLine": 7,
+                        "kind": 1
+                    }
+                },
+                "byRef": false,
+                "unpack": false,
+                "attributes": {
+                    "startLine": 7,
+                    "endLine": 7
+                }
+            }
+        ],
+        "attributes": {
+            "startLine": 7,
+            "endLine": 7
+        }
+    }
+]
+```
+
+There is currently no mechanism to convert JSON back into a node tree. Furthermore, not all ASTs
+can be JSON encoded. In particular, JSON only supports UTF-8 strings.
+
 Serialization to XML
 --------------------
 

doc/component/Error_handling.markdown

@@ -35,6 +35,8 @@ the source code of the parsed file. An example for printing an error:
 if ($e->hasColumnInfo()) {
     echo $e->getRawMessage() . ' from ' . $e->getStartLine() . ':' . $e->getStartColumn($code)
         . ' to ' . $e->getEndLine() . ':' . $e->getEndColumn($code);
+    // or:
+    echo $e->getMessageWithColumnInfo();
 } else {
     echo $e->getMessage();
 }
@@ -46,27 +48,23 @@ file.
 Error recovery
 --------------
 
-> **EXPERIMENTAL**
+The error behavior of the parser (and other components) is controlled by an `ErrorHandler`. Whenever an error is
+encountered, `ErrorHandler::handleError()` is invoked. The default error handling strategy is `ErrorHandler\Throwing`,
+which will immediately throw when an error is encountered.
 
-By default the parser will throw an exception upon encountering the first error during parsing. An alternative mode is
+To instead collect all encountered errors into an array, while trying to continue parsing the rest of the source code,
-also supported, in which the parser will remember the error, but try to continue parsing the rest of the source code.
+an instance of `ErrorHandler\Collecting` can be passed to the `Parser::parse()` method. A usage example:
-
-To enable this mode the `throwOnError` parser option needs to be disabled. Any errors that occurred during parsing can
-then be retrieved using `$parser->getErrors()`. The `$parser->parse()` method will either return a partial syntax tree
-or `null` if recovery fails.
-
-A usage example:
 
 ```php
-$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7, null, array(
+$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::ONLY_PHP7);
-    'throwOnError' => false,
+$errorHandler = new PhpParser\ErrorHandler\Collecting;
-));
 
-$stmts = $parser->parse($code);
+$stmts = $parser->parse($code, $errorHandler);
-$errors = $parser->getErrors();
 
-foreach ($errors as $error) {
+if ($errorHandler->hasErrors()) {
+    foreach ($errorHandler->getErrors() as $error) {
         // $error is an ordinary PhpParser\Error
+    }
 }
 
 if (null !== $stmts) {
@@ -74,4 +72,4 @@ if (null !== $stmts) {
 }
 ```
 
-The error recovery implementation is experimental -- it currently won't be able to recover from many types of errors.
+The `NameResolver` visitor also accepts an `ErrorHandler` as a constructor argument.

doc/component/Lexer.markdown

@@ -95,13 +95,14 @@ Lexer extension
 
 A lexer has to define the following public interface:
 
-    void startLexing(string $code);
+    void startLexing(string $code, ErrorHandler $errorHandler = null);
     array getTokens();
     string handleHaltCompiler();
     int getNextToken(string &$value = null, array &$startAttributes = null, array &$endAttributes = null);
 
 The `startLexing()` method is invoked with the source code that is to be lexed (including the opening tag) whenever the
-`parse()` method of the parser is called. It can be used to reset state or preprocess the source code or tokens.
+`parse()` method of the parser is called. It can be used to reset state or preprocess the source code or tokens. The
+passes `ErrorHandler` should be used to report lexing errors.
 
 The `getTokens()` method returns the current token array, in the usual `token_get_all()` format. This method is not
 used by the parser (which uses `getNextToken()`), but is useful in combination with the token position attributes.
@@ -122,9 +123,10 @@ node and the `$endAttributes` from the last token that is part of the node.
 E.g. if the tokens `T_FUNCTION T_STRING ... '{' ... '}'` constitute a node, then the `$startAttributes` from the
 `T_FUNCTION` token will be taken and the `$endAttributes` from the `'}'` token.
 
-An application of custom attributes is storing the original formatting of literals: The parser does not retain
+An application of custom attributes is storing the exact original formatting of literals: While the parser does retain
-information about the formatting of integers (like decimal vs. hexadecimal) or strings (like used quote type or used
+some information about the formatting of integers (like decimal vs. hexadecimal) or strings (like used quote type), it
-escape sequences). This can be remedied by storing the original value in an attribute:
+does not preserve the exact original formatting (e.g. leading zeros for integers or escape sequences in strings). This
+can be remedied by storing the original value in an attribute:
 
 ```php
 use PhpParser\Lexer;
@@ -136,6 +138,7 @@ class KeepOriginalValueLexer extends Lexer // or Lexer\Emulative
         $tokenId = parent::getNextToken($value, $startAttributes, $endAttributes);
 
         if ($tokenId == Tokens::T_CONSTANT_ENCAPSED_STRING   // non-interpolated string
+            || $tokenId == Tokens::T_ENCAPSED_AND_WHITESPACE // interpolated string
             || $tokenId == Tokens::T_LNUMBER                 // integer
             || $tokenId == Tokens::T_DNUMBER                 // floating point number
         ) {

grammar/README.md

@@ -6,7 +6,7 @@ What do all those files mean?
  * `tokens.y`:           Tokens definition shared between PHP 5 and PHP 7 grammars
  * `parser.template`:    A `kmyacc` parser prototype file for PHP
  * `tokens.template`:    A `kmyacc` prototype file for the `Tokens` class
- * `rebuildParser.php`: Preprocesses the grammar and builds the parser using `kmyacc`
+ * `rebuildParsers.php`: Preprocesses the grammar and builds the parser using `kmyacc`
 
 .phpy pseudo language
 =====================
@@ -16,13 +16,13 @@ applied to it:
 
  * Nodes are created using the syntax `Name[..., ...]`. This is transformed into
    `new Name(..., ..., attributes())`
- * Some function-like constructs are resolved (see `rebuildParser.php` for a list)
+ * Some function-like constructs are resolved (see `rebuildParsers.php` for a list)
 
 Building the parser
 ===================
 
 In order to rebuild the parser, you need [moriyoshi's fork of kmyacc](https://github.com/moriyoshi/kmyacc-forked).
-After you compiled/installed it, run the `rebuildParser.php` script.
+After you compiled/installed it, run the `rebuildParsers.php` script.
 
 By default only the `Parser.php` is built. If you want to additionally emit debug symbols and create `y.output`, run the
 script with `--debug`. If you want to retain the preprocessed grammar pass `--keep-tmp-grammar`.

grammar/php5.y

@@ -55,9 +55,12 @@ top_statement:
     | class_declaration_statement                           { $$ = $1; }
     | T_HALT_COMPILER
           { $$ = Stmt\HaltCompiler[$this->lexer->handleHaltCompiler()]; }
-    | T_NAMESPACE namespace_name ';'                        { $$ = Stmt\Namespace_[$2, null]; }
+    | T_NAMESPACE namespace_name ';'
-    | T_NAMESPACE namespace_name '{' top_statement_list '}' { $$ = Stmt\Namespace_[$2, $4]; }
+          { $$ = Stmt\Namespace_[$2, null]; $this->checkNamespace($$); }
-    | T_NAMESPACE '{' top_statement_list '}'                { $$ = Stmt\Namespace_[null,     $3]; }
+    | T_NAMESPACE namespace_name '{' top_statement_list '}'
+          { $$ = Stmt\Namespace_[$2, $4]; $this->checkNamespace($$); }
+    | T_NAMESPACE '{' top_statement_list '}'
+          { $$ = Stmt\Namespace_[null, $3]; $this->checkNamespace($$); }
     | T_USE use_declarations ';'                            { $$ = Stmt\Use_[$2, Stmt\Use_::TYPE_NORMAL]; }
     | T_USE use_type use_declarations ';'                   { $$ = Stmt\Use_[$3, $2]; }
     | group_use_declaration ';'                             { $$ = $1; }
@@ -72,13 +75,13 @@ use_type:
 /* Using namespace_name_parts here to avoid s/r conflict on T_NS_SEPARATOR */
 group_use_declaration:
       T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
-          { $$ = Stmt\GroupUse[Name[$3], $6, $2]; }
+          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, $2]; }
     | T_USE use_type T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
-          { $$ = Stmt\GroupUse[Name[$4], $7, $2]; }
+          { $$ = Stmt\GroupUse[new Name($4, stackAttributes(#4)), $7, $2]; }
     | T_USE namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[Name[$2], $5, Stmt\Use_::TYPE_UNKNOWN]; }
+          { $$ = Stmt\GroupUse[new Name($2, stackAttributes(#2)), $5, Stmt\Use_::TYPE_UNKNOWN]; }
     | T_USE T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[Name[$3], $6, Stmt\Use_::TYPE_UNKNOWN]; }
+          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, Stmt\Use_::TYPE_UNKNOWN]; }
 ;
 
 unprefixed_use_declarations:
@@ -98,8 +101,10 @@ inline_use_declarations:
 ;
 
 unprefixed_use_declaration:
-      namespace_name                                        { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; }
+      namespace_name
-    | namespace_name T_AS T_STRING                          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; }
+          { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #1); }
+    | namespace_name T_AS T_STRING
+          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #3); }
 ;
 
 use_declaration:
@@ -150,7 +155,7 @@ inner_statement:
 ;
 
 non_empty_statement:
-      '{' inner_statement_list '}'                          { $$ = $2; }
+      '{' inner_statement_list '}'                          { $$ = $2; prependLeadingComments($$); }
     | T_IF parentheses_expr statement elseif_list else_single
           { $$ = Stmt\If_[$2, ['stmts' => toArray($3), 'elseifs' => $4, 'else' => $5]]; }
     | T_IF parentheses_expr ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
@@ -179,7 +184,7 @@ non_empty_statement:
           { $$ = Stmt\Foreach_[$3, $7[0], ['keyVar' => $5, 'byRef' => $7[1], 'stmts' => $9]]; }
     | T_DECLARE '(' declare_list ')' declare_statement      { $$ = Stmt\Declare_[$3, $5]; }
     | T_TRY '{' inner_statement_list '}' catches optional_finally
-          { $$ = Stmt\TryCatch[$3, $5, $6]; }
+          { $$ = Stmt\TryCatch[$3, $5, $6]; $this->checkTryCatch($$); }
     | T_THROW expr ';'                                      { $$ = Stmt\Throw_[$2]; }
     | T_GOTO T_STRING ';'                                   { $$ = Stmt\Goto_[$2]; }
     | T_STRING ':'                                          { $$ = Stmt\Label[$1]; }
@@ -231,9 +236,11 @@ function_declaration_statement:
 
 class_declaration_statement:
       class_entry_type T_STRING extends_from implements_list '{' class_statement_list '}'
-          { $$ = Stmt\Class_[$2, ['type' => $1, 'extends' => $3, 'implements' => $4, 'stmts' => $6]]; }
+          { $$ = Stmt\Class_[$2, ['type' => $1, 'extends' => $3, 'implements' => $4, 'stmts' => $6]];
+            $this->checkClass($$, #2); }
     | T_INTERFACE T_STRING interface_extends_list '{' class_statement_list '}'
-          { $$ = Stmt\Interface_[$2, ['extends' => $3, 'stmts' => $5]]; }
+          { $$ = Stmt\Interface_[$2, ['extends' => $3, 'stmts' => $5]];
+            $this->checkInterface($$, #2); }
     | T_TRAIT T_STRING '{' class_statement_list '}'
           { $$ = Stmt\Trait_[$2, ['stmts' => $4]]; }
 ;
@@ -362,9 +369,9 @@ non_empty_parameter_list:
 
 parameter:
       optional_param_type optional_ref optional_ellipsis T_VARIABLE
-          { $$ = Node\Param[parseVar($4), null, $1, $2, $3]; }
+          { $$ = Node\Param[parseVar($4), null, $1, $2, $3]; $this->checkParam($$); }
     | optional_param_type optional_ref optional_ellipsis T_VARIABLE '=' static_scalar
-          { $$ = Node\Param[parseVar($4), $6, $1, $2, $3]; }
+          { $$ = Node\Param[parseVar($4), $6, $1, $2, $3]; $this->checkParam($$); }
 ;
 
 type:
@@ -427,10 +434,12 @@ class_statement_list:
 ;
 
 class_statement:
-      variable_modifiers property_declaration_list ';'      { $$ = Stmt\Property[$1, $2]; }
+      variable_modifiers property_declaration_list ';'
+          { $$ = Stmt\Property[$1, $2]; $this->checkProperty($$, #1); }
     | T_CONST class_const_list ';'                          { $$ = Stmt\ClassConst[$2, 0]; }
     | method_modifiers T_FUNCTION optional_ref identifier '(' parameter_list ')' optional_return_type method_body
-          { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]]; }
+          { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]];
+            $this->checkClassMethod($$, #1); }
     | T_USE name_list trait_adaptations                     { $$ = Stmt\TraitUse[$2, $3]; }
 ;
 
@@ -482,7 +491,7 @@ method_modifiers:
 
 non_empty_member_modifiers:
       member_modifier                                       { $$ = $1; }
-    | non_empty_member_modifiers member_modifier            { Stmt\Class_::verifyModifier($1, $2); $$ = $1 | $2; }
+    | non_empty_member_modifiers member_modifier            { $this->checkModifier($1, $2, #2); $$ = $1 | $2; }
 ;
 
 member_modifier:
@@ -640,7 +649,8 @@ scalar_dereference:
 
 anonymous_class:
       T_CLASS ctor_arguments extends_from implements_list '{' class_statement_list '}'
-          { $$ = array(Stmt\Class_[null, ['type' => 0, 'extends' => $3, 'implements' => $4, 'stmts' => $6]], $2); }
+          { $$ = array(Stmt\Class_[null, ['type' => 0, 'extends' => $3, 'implements' => $4, 'stmts' => $6]], $2);
+            $this->checkClass($$[0], -1); }
 
 new_expr:
       T_NEW class_name_reference ctor_arguments             { $$ = Expr\New_[$2, $3]; }
@@ -743,7 +753,7 @@ ctor_arguments:
 ;
 
 common_scalar:
-      T_LNUMBER                                             { $$ = Scalar\LNumber::fromString($1, attributes(), true); }
+      T_LNUMBER                                             { $$ = $this->parseLNumber($1, attributes(), true); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
     | T_CONSTANT_ENCAPSED_STRING
           { $attrs = attributes(); $attrs['kind'] = strKind($1);
@@ -915,6 +925,7 @@ object_property:
       T_STRING                                              { $$ = $1; }
     | '{' expr '}'                                          { $$ = $2; }
     | variable_without_objects                              { $$ = $1; }
+    | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 list_expr:
@@ -928,7 +939,7 @@ list_expr_elements:
 
 list_expr_element:
       variable                                              { $$ = Expr\ArrayItem[$1, null, false]; }
-    | list_expr                                             { $$ = $1; }
+    | list_expr                                             { $$ = Expr\ArrayItem[$1, null, false]; }
     | /* empty */                                           { $$ = null; }
 ;
 
@@ -960,10 +971,14 @@ encaps_string_part:
       T_ENCAPSED_AND_WHITESPACE                             { $$ = Scalar\EncapsedStringPart[$1]; }
 ;
 
-encaps_var:
+encaps_base_var:
       T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
-    | T_VARIABLE '[' encaps_var_offset ']'                  { $$ = Expr\ArrayDimFetch[Expr\Variable[parseVar($1)], $3]; }
+;
-    | T_VARIABLE T_OBJECT_OPERATOR T_STRING                 { $$ = Expr\PropertyFetch[Expr\Variable[parseVar($1)], $3]; }
+
+encaps_var:
+      encaps_base_var                                       { $$ = $1; }
+    | encaps_base_var '[' encaps_var_offset ']'             { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | encaps_base_var T_OBJECT_OPERATOR T_STRING            { $$ = Expr\PropertyFetch[$1, $3]; }
     | T_DOLLAR_OPEN_CURLY_BRACES expr '}'                   { $$ = Expr\Variable[$2]; }
     | T_DOLLAR_OPEN_CURLY_BRACES T_STRING_VARNAME '}'       { $$ = Expr\Variable[$2]; }
     | T_DOLLAR_OPEN_CURLY_BRACES T_STRING_VARNAME '[' expr ']' '}'
@@ -973,7 +988,7 @@ encaps_var:
 
 encaps_var_offset:
       T_STRING                                              { $$ = Scalar\String_[$1]; }
-    | T_NUM_STRING                                          { $$ = Scalar\String_[$1]; }
+    | T_NUM_STRING                                          { $$ = $this->parseNumString($1, attributes()); }
     | T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
 ;
 

grammar/php7.y

@@ -55,9 +55,12 @@ top_statement:
     | class_declaration_statement                           { $$ = $1; }
     | T_HALT_COMPILER
           { $$ = Stmt\HaltCompiler[$this->lexer->handleHaltCompiler()]; }
-    | T_NAMESPACE namespace_name ';'                        { $$ = Stmt\Namespace_[$2, null]; }
+    | T_NAMESPACE namespace_name ';'
-    | T_NAMESPACE namespace_name '{' top_statement_list '}' { $$ = Stmt\Namespace_[$2, $4]; }
+          { $$ = Stmt\Namespace_[$2, null]; $this->checkNamespace($$); }
-    | T_NAMESPACE '{' top_statement_list '}'                { $$ = Stmt\Namespace_[null,     $3]; }
+    | T_NAMESPACE namespace_name '{' top_statement_list '}'
+          { $$ = Stmt\Namespace_[$2, $4]; $this->checkNamespace($$); }
+    | T_NAMESPACE '{' top_statement_list '}'
+          { $$ = Stmt\Namespace_[null, $3]; $this->checkNamespace($$); }
     | T_USE use_declarations ';'                            { $$ = Stmt\Use_[$2, Stmt\Use_::TYPE_NORMAL]; }
     | T_USE use_type use_declarations ';'                   { $$ = Stmt\Use_[$3, $2]; }
     | group_use_declaration ';'                             { $$ = $1; }
@@ -72,13 +75,13 @@ use_type:
 /* Using namespace_name_parts here to avoid s/r conflict on T_NS_SEPARATOR */
 group_use_declaration:
       T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
-          { $$ = Stmt\GroupUse[Name[$3], $6, $2]; }
+          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, $2]; }
     | T_USE use_type T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
-          { $$ = Stmt\GroupUse[Name[$4], $7, $2]; }
+          { $$ = Stmt\GroupUse[new Name($4, stackAttributes(#4)), $7, $2]; }
     | T_USE namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[Name[$2], $5, Stmt\Use_::TYPE_UNKNOWN]; }
+          { $$ = Stmt\GroupUse[new Name($2, stackAttributes(#2)), $5, Stmt\Use_::TYPE_UNKNOWN]; }
     | T_USE T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
-          { $$ = Stmt\GroupUse[Name[$3], $6, Stmt\Use_::TYPE_UNKNOWN]; }
+          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, Stmt\Use_::TYPE_UNKNOWN]; }
 ;
 
 unprefixed_use_declarations:
@@ -98,8 +101,10 @@ inline_use_declarations:
 ;
 
 unprefixed_use_declaration:
-      namespace_name                                        { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; }
+      namespace_name
-    | namespace_name T_AS T_STRING                          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; }
+          { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #1); }
+    | namespace_name T_AS T_STRING
+          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #3); }
 ;
 
 use_declaration:
@@ -150,7 +155,7 @@ inner_statement:
 ;
 
 non_empty_statement:
-      '{' inner_statement_list '}'                          { $$ = $2; }
+      '{' inner_statement_list '}'                          { $$ = $2; prependLeadingComments($$); }
     | T_IF '(' expr ')' statement elseif_list else_single
           { $$ = Stmt\If_[$3, ['stmts' => toArray($5), 'elseifs' => $6, 'else' => $7]]; }
     | T_IF '(' expr ')' ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
@@ -175,7 +180,7 @@ non_empty_statement:
           { $$ = Stmt\Foreach_[$3, $7[0], ['keyVar' => $5, 'byRef' => $7[1], 'stmts' => $9]]; }
     | T_DECLARE '(' declare_list ')' declare_statement      { $$ = Stmt\Declare_[$3, $5]; }
     | T_TRY '{' inner_statement_list '}' catches optional_finally
-          { $$ = Stmt\TryCatch[$3, $5, $6]; }
+          { $$ = Stmt\TryCatch[$3, $5, $6]; $this->checkTryCatch($$); }
     | T_THROW expr ';'                                      { $$ = Stmt\Throw_[$2]; }
     | T_GOTO T_STRING ';'                                   { $$ = Stmt\Goto_[$2]; }
     | T_STRING ':'                                          { $$ = Stmt\Label[$1]; }
@@ -232,9 +237,11 @@ function_declaration_statement:
 
 class_declaration_statement:
       class_entry_type T_STRING extends_from implements_list '{' class_statement_list '}'
-          { $$ = Stmt\Class_[$2, ['type' => $1, 'extends' => $3, 'implements' => $4, 'stmts' => $6]]; }
+          { $$ = Stmt\Class_[$2, ['type' => $1, 'extends' => $3, 'implements' => $4, 'stmts' => $6]];
+            $this->checkClass($$, #2); }
     | T_INTERFACE T_STRING interface_extends_list '{' class_statement_list '}'
-          { $$ = Stmt\Interface_[$2, ['extends' => $3, 'stmts' => $5]]; }
+          { $$ = Stmt\Interface_[$2, ['extends' => $3, 'stmts' => $5]];
+            $this->checkInterface($$, #2); }
     | T_TRAIT T_STRING '{' class_statement_list '}'
           { $$ = Stmt\Trait_[$2, ['stmts' => $4]]; }
 ;
@@ -364,9 +371,9 @@ non_empty_parameter_list:
 
 parameter:
       optional_param_type optional_ref optional_ellipsis T_VARIABLE
-          { $$ = Node\Param[parseVar($4), null, $1, $2, $3]; }
+          { $$ = Node\Param[parseVar($4), null, $1, $2, $3]; $this->checkParam($$); }
     | optional_param_type optional_ref optional_ellipsis T_VARIABLE '=' expr
-          { $$ = Node\Param[parseVar($4), $6, $1, $2, $3]; }
+          { $$ = Node\Param[parseVar($4), $6, $1, $2, $3]; $this->checkParam($$); }
 ;
 
 type_expr:
@@ -431,10 +438,13 @@ class_statement_list:
 ;
 
 class_statement:
-      variable_modifiers property_declaration_list ';'      { $$ = Stmt\Property[$1, $2]; }
+      variable_modifiers property_declaration_list ';'
-    | method_modifiers T_CONST class_const_list ';'         { $$ = Stmt\ClassConst[$3, $1]; }
+          { $$ = Stmt\Property[$1, $2]; $this->checkProperty($$, #1); }
+    | method_modifiers T_CONST class_const_list ';'
+          { $$ = Stmt\ClassConst[$3, $1]; $this->checkClassConst($$, #1); }
     | method_modifiers T_FUNCTION optional_ref identifier '(' parameter_list ')' optional_return_type method_body
-          { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]]; }
+          { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]];
+            $this->checkClassMethod($$, #1); }
     | T_USE name_list trait_adaptations                     { $$ = Stmt\TraitUse[$2, $3]; }
 ;
 
@@ -486,7 +496,7 @@ method_modifiers:
 
 non_empty_member_modifiers:
       member_modifier                                       { $$ = $1; }
-    | non_empty_member_modifiers member_modifier            { Stmt\Class_::verifyModifier($1, $2); $$ = $1 | $2; }
+    | non_empty_member_modifiers member_modifier            { $this->checkModifier($1, $2, #2); $$ = $1 | $2; }
 ;
 
 member_modifier:
@@ -524,7 +534,6 @@ expr:
     | array_short_syntax '=' expr                           { $$ = Expr\Assign[$1, $3]; }
     | variable '=' expr                                     { $$ = Expr\Assign[$1, $3]; }
     | variable '=' '&' variable                             { $$ = Expr\AssignRef[$1, $4]; }
-    | variable '=' '&' new_expr                             { $$ = Expr\AssignRef[$1, $4]; }
     | new_expr                                              { $$ = $1; }
     | T_CLONE expr                                          { $$ = Expr\Clone_[$2]; }
     | variable T_PLUS_EQUAL expr                            { $$ = Expr\AssignOp\Plus      [$1, $3]; }
@@ -614,7 +623,8 @@ expr:
 
 anonymous_class:
       T_CLASS ctor_arguments extends_from implements_list '{' class_statement_list '}'
-          { $$ = array(Stmt\Class_[null, ['type' => 0, 'extends' => $3, 'implements' => $4, 'stmts' => $6]], $2); }
+          { $$ = array(Stmt\Class_[null, ['type' => 0, 'extends' => $3, 'implements' => $4, 'stmts' => $6]], $2);
+            $this->checkClass($$[0], -1); }
 
 new_expr:
       T_NEW class_name_reference ctor_arguments             { $$ = Expr\New_[$2, $3]; }
@@ -657,6 +667,7 @@ name:
 class_name_reference:
       class_name                                            { $$ = $1; }
     | new_variable                                          { $$ = $1; }
+    | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 class_name_or_var:
@@ -685,6 +696,10 @@ constant:
       name                                                  { $$ = Expr\ConstFetch[$1]; }
     | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM identifier
           { $$ = Expr\ClassConstFetch[$1, $3]; }
+    /* We interpret and isolated FOO:: as an unfinished class constant fetch. It could also be
+       an unfinished static property fetch or unfinished scoped call. */
+    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM error
+          { $$ = Expr\ClassConstFetch[$1, Expr\Error[]]; $this->errorState = 2; }
 ;
 
 array_short_syntax:
@@ -704,7 +719,7 @@ dereferencable_scalar:
 ;
 
 scalar:
-      T_LNUMBER                                             { $$ = Scalar\LNumber::fromString($1, attributes()); }
+      T_LNUMBER                                             { $$ = $this->parseLNumber($1, attributes()); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
     | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
     | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
@@ -767,6 +782,7 @@ simple_variable:
       T_VARIABLE                                            { $$ = parseVar($1); }
     | '$' '{' expr '}'                                      { $$ = $3; }
     | '$' simple_variable                                   { $$ = Expr\Variable[$2]; }
+    | '$' error                                             { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 static_member:
@@ -793,6 +809,7 @@ property_name:
       T_STRING                                              { $$ = $1; }
     | '{' expr '}'	                                        { $$ = $2; }
     | simple_variable	                                    { $$ = Expr\Variable[$1]; }
+    | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 list_expr:
@@ -806,7 +823,7 @@ list_expr_elements:
 
 list_expr_element:
       variable                                              { $$ = Expr\ArrayItem[$1, null, false]; }
-    | list_expr                                             { $$ = $1; }
+    | list_expr                                             { $$ = Expr\ArrayItem[$1, null, false]; }
     | expr T_DOUBLE_ARROW variable                          { $$ = Expr\ArrayItem[$3, $1, false]; }
     | expr T_DOUBLE_ARROW list_expr                         { $$ = Expr\ArrayItem[$3, $1, false]; }
     | /* empty */                                           { $$ = null; }
@@ -841,10 +858,14 @@ encaps_string_part:
       T_ENCAPSED_AND_WHITESPACE                             { $$ = Scalar\EncapsedStringPart[$1]; }
 ;
 
-encaps_var:
+encaps_base_var:
       T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
-    | T_VARIABLE '[' encaps_var_offset ']'                  { $$ = Expr\ArrayDimFetch[Expr\Variable[parseVar($1)], $3]; }
+;
-    | T_VARIABLE T_OBJECT_OPERATOR T_STRING                 { $$ = Expr\PropertyFetch[Expr\Variable[parseVar($1)], $3]; }
+
+encaps_var:
+      encaps_base_var                                       { $$ = $1; }
+    | encaps_base_var '[' encaps_var_offset ']'             { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | encaps_base_var T_OBJECT_OPERATOR T_STRING            { $$ = Expr\PropertyFetch[$1, $3]; }
     | T_DOLLAR_OPEN_CURLY_BRACES expr '}'                   { $$ = Expr\Variable[$2]; }
     | T_DOLLAR_OPEN_CURLY_BRACES T_STRING_VARNAME '}'       { $$ = Expr\Variable[$2]; }
     | T_DOLLAR_OPEN_CURLY_BRACES T_STRING_VARNAME '[' expr ']' '}'
@@ -854,7 +875,8 @@ encaps_var:
 
 encaps_var_offset:
       T_STRING                                              { $$ = Scalar\String_[$1]; }
-    | T_NUM_STRING                                          { $$ = Scalar\String_[$1]; }
+    | T_NUM_STRING                                          { $$ = $this->parseNumString($1, attributes()); }
+    | '-' T_NUM_STRING                                      { $$ = $this->parseNumString('-' . $2, attributes()); }
     | T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
 ;
 

grammar/rebuildParsers.php

@@ -124,6 +124,12 @@ function resolveMacros($code) {
                 return '$this->startAttributeStack[#1] + $this->endAttributes';
             }
 
+            if ('stackAttributes' == $name) {
+                assertArgs(1, $args, $name);
+                return '$this->startAttributeStack[' . $args[0] . ']'
+                     . ' + $this->endAttributeStack[' . $args[0] . ']';
+            }
+
             if ('init' == $name) {
                 return '$$ = array(' . implode(', ', $args) . ')';
             }
@@ -195,6 +201,15 @@ function resolveMacros($code) {
                      . $args[0] . '[\'docLabel\'] = $matches[1];';
             }
 
+            if ('prependLeadingComments' == $name) {
+                assertArgs(1, $args, $name);
+
+                return '$attrs = $this->startAttributeStack[#1]; $stmts = ' . $args[0] . '; '
+                . 'if (!empty($attrs[\'comments\']) && isset($stmts[0])) {'
+                . '$stmts[0]->setAttribute(\'comments\', '
+                . 'array_merge($attrs[\'comments\'], $stmts[0]->getAttribute(\'comments\', []))); }';
+            }
+
             return $matches[0];
         },
         $code

lib/PhpParser/Builder/Declaration.php

@@ -3,8 +3,6 @@
 namespace PhpParser\Builder;
 
 use PhpParser;
-use PhpParser\Node;
-use PhpParser\Node\Stmt;
 
 abstract class Declaration extends PhpParser\BuilderAbstract
 {

lib/PhpParser/Builder/FunctionLike.php

@@ -4,12 +4,13 @@ namespace PhpParser\Builder;
 
 use PhpParser;
 use PhpParser\Node;
-use PhpParser\Node\Stmt;
 
 abstract class FunctionLike extends Declaration
 {
     protected $returnByRef = false;
     protected $params = array();
+
+    /** @var string|Node\Name|Node\NullableType|null */
     protected $returnType = null;
 
     /**
@@ -60,18 +61,14 @@ abstract class FunctionLike extends Declaration
     /**
      * Sets the return type for PHP 7.
      *
-     * @param string|Node\Name $type One of array, callable, string, int, float, bool,
+     * @param string|Node\Name|Node\NullableType $type One of array, callable, string, int, float, bool, iterable,
      *                               or a class/interface name.
      *
      * @return $this The builder instance (for fluid interface)
      */
     public function setReturnType($type)
     {
-        if (in_array($type, array('array', 'callable', 'string', 'int', 'float', 'bool'))) {
+        $this->returnType = $this->normalizeType($type);
-            $this->returnType = $type;
-        } else {
-            $this->returnType = $this->normalizeName($type);
-        }
 
         return $this;
     }

lib/PhpParser/Builder/Method.php

@@ -10,6 +10,8 @@ class Method extends FunctionLike
 {
     protected $name;
     protected $flags = 0;
+
+    /** @var array|null */
     protected $stmts = array();
 
     /**

lib/PhpParser/Builder/Param.php

@@ -10,7 +10,10 @@ class Param extends PhpParser\BuilderAbstract
     protected $name;
 
     protected $default = null;
+
+    /** @var string|Node\Name|Node\NullableType|null */
     protected $type = null;
+
     protected $byRef = false;
 
     /**
@@ -38,15 +41,14 @@ class Param extends PhpParser\BuilderAbstract
     /**
      * Sets type hint for the parameter.
      *
-     * @param string|Node\Name $type Type hint to use
+     * @param string|Node\Name|Node\NullableType $type Type hint to use
      *
      * @return $this The builder instance (for fluid interface)
      */
     public function setTypeHint($type) {
-        if (in_array($type, array('array', 'callable', 'string', 'int', 'float', 'bool'))) {
+        $this->type = $this->normalizeType($type);
-            $this->type = $type;
+        if ($this->type === 'void') {
-        } else {
+            throw new \LogicException('Parameter type cannot be void');
-            $this->type = $this->normalizeName($type);
         }
 
         return $this;

lib/PhpParser/Builder/Trait_.php

@@ -3,7 +3,6 @@
 namespace PhpParser\Builder;
 
 use PhpParser;
-use PhpParser\Node\Name;
 use PhpParser\Node\Stmt;
 
 class Trait_ extends Declaration

lib/PhpParser/BuilderAbstract.php

@@ -2,11 +2,12 @@
 
 namespace PhpParser;
 
-use PhpParser\Node\Name;
-use PhpParser\Node\Expr;
-use PhpParser\Node\Stmt;
-use PhpParser\Node\Scalar;
 use PhpParser\Comment;
+use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
+use PhpParser\Node\NullableType;
+use PhpParser\Node\Scalar;
+use PhpParser\Node\Stmt;
 
 abstract class BuilderAbstract implements Builder {
     /**
@@ -53,6 +54,49 @@ abstract class BuilderAbstract implements Builder {
         throw new \LogicException('Name must be a string or an instance of PhpParser\Node\Name');
     }
 
+    /**
+     * Normalizes a type: Converts plain-text type names into proper AST representation.
+     *
+     * In particular, builtin types are left as strings, custom types become Names and nullables
+     * are wrapped in NullableType nodes.
+     *
+     * @param Name|string|NullableType $type The type to normalize
+     *
+     * @return Name|string|NullableType The normalized type
+     */
+    protected function normalizeType($type) {
+        if (!is_string($type)) {
+            if (!$type instanceof Name && !$type instanceof NullableType) {
+                throw new \LogicException(
+                    'Type must be a string, or an instance of Name or NullableType');
+            }
+            return $type;
+        }
+
+        $nullable = false;
+        if (strlen($type) > 0 && $type[0] === '?') {
+            $nullable = true;
+            $type = substr($type, 1);
+        }
+
+        $builtinTypes = array(
+            'array', 'callable', 'string', 'int', 'float', 'bool', 'iterable', 'void'
+        );
+
+        $lowerType = strtolower($type);
+        if (in_array($lowerType, $builtinTypes)) {
+            $type = $lowerType;
+        } else {
+            $type = $this->normalizeName($type);
+        }
+
+        if ($nullable && $type === 'void') {
+            throw new \LogicException('void type cannot be nullable');
+        }
+
+        return $nullable ? new Node\NullableType($type) : $type;
+    }
+
     /**
      * Normalizes a value: Converts nulls, booleans, integers,
      * floats, strings and arrays into their respective nodes

lib/PhpParser/Comment.php

@@ -2,7 +2,7 @@
 
 namespace PhpParser;
 
-class Comment
+class Comment implements \JsonSerializable
 {
     protected $text;
     protected $line;
@@ -126,4 +126,15 @@ class Comment
         }
         return $shortestPrefixLen;
     }
+
+    public function jsonSerialize() {
+        // Technically not a node, but we make it look like one anyway
+        $type = $this instanceof Comment\Doc ? 'Comment_Doc' : 'Comment';
+        return [
+            'nodeType' => $type,
+            'text' => $this->text,
+            'line' => $this->line,
+            'filePos' => $this->filePos,
+        ];
+    }
 }

lib/PhpParser/Error.php

@@ -61,6 +61,16 @@ class Error extends \RuntimeException
         return $this->attributes;
     }
 
+    /**
+     * Sets the attributes of the node/token the error occured at.
+     *
+     * @param array $attributes
+     */
+    public function setAttributes(array $attributes) {
+        $this->attributes = $attributes;
+        $this->updateMessage();
+    }
+
     /**
      * Sets the line of the PHP file the error occurred in.
      *
@@ -120,6 +130,14 @@ class Error extends \RuntimeException
         return $this->toColumn($code, $this->attributes['endFilePos']);
     }
 
+    public function getMessageWithColumnInfo($code) {
+        return sprintf(
+            '%s from %d:%d to %d:%d', $this->getRawMessage(),
+            $this->getStartLine(), $this->getStartColumn($code),
+            $this->getEndLine(), $this->getEndColumn($code)
+        );
+    }
+
     private function toColumn($code, $pos) {
         if ($pos > strlen($code)) {
             throw new \RuntimeException('Invalid position information');
@@ -145,14 +163,4 @@ class Error extends \RuntimeException
             $this->message .= ' on line ' . $this->getStartLine();
         }
     }
-
-    /** @deprecated Use getStartLine() instead */
-    public function getRawLine() {
-        return $this->getStartLine();
-    }
-
-    /** @deprecated Use setStartLine() instead */
-    public function setRawLine($line) {
-        $this->setStartLine($line);
-    }
 }

lib/PhpParser/ErrorHandler.php

@@ -0,0 +1,13 @@
+<?php
+
+namespace PhpParser;
+
+interface ErrorHandler
+{
+    /**
+     * Handle an error generated during lexing, parsing or some other operation.
+     *
+     * @param Error $error The error that needs to be handled
+     */
+    public function handleError(Error $error);
+}

lib/PhpParser/ErrorHandler/Collecting.php

@@ -0,0 +1,46 @@
+<?php
+
+namespace PhpParser\ErrorHandler;
+
+use PhpParser\Error;
+use PhpParser\ErrorHandler;
+
+/**
+ * Error handler that collects all errors into an array.
+ *
+ * This allows graceful handling of errors.
+ */
+class Collecting implements ErrorHandler
+{
+    /** @var Error[] Collected errors */
+    private $errors = [];
+
+    public function handleError(Error $error) {
+        $this->errors[] = $error;
+    }
+
+    /**
+     * Get collected errors.
+     *
+     * @return Error[]
+     */
+    public function getErrors() {
+        return $this->errors;
+    }
+
+    /**
+     * Check whether there are any errors.
+     *
+     * @return bool
+     */
+    public function hasErrors() {
+        return !empty($this->errors);
+    }
+
+    /**
+     * Reset/clear collected errors.
+     */
+    public function clearErrors() {
+        $this->errors = [];
+    }
+}

lib/PhpParser/ErrorHandler/Throwing.php

@@ -0,0 +1,18 @@
+<?php
+
+namespace PhpParser\ErrorHandler;
+
+use PhpParser\Error;
+use PhpParser\ErrorHandler;
+
+/**
+ * Error handler that handles all errors by throwing them.
+ *
+ * This is the default strategy used by all components.
+ */
+class Throwing implements ErrorHandler
+{
+    public function handleError(Error $error) {
+        throw $error;
+    }
+}

lib/PhpParser/Lexer.php

@@ -2,7 +2,6 @@
 
 namespace PhpParser;
 
-use PhpParser\Node\Scalar\LNumber;
 use PhpParser\Parser\Tokens;
 
 class Lexer
@@ -12,6 +11,7 @@ class Lexer
     protected $pos;
     protected $line;
     protected $filePos;
+    protected $prevCloseTagHasNewline;
 
     protected $tokenMap;
     protected $dropTokens;
@@ -48,25 +48,36 @@ class Lexer
     /**
      * Initializes the lexer for lexing the provided source code.
      *
-     * @param string $code The source code to lex
+     * This function does not throw if lexing errors occur. Instead, errors may be retrieved using
+     * the getErrors() method.
      *
-     * @throws Error on lexing errors (unterminated comment or unexpected character)
+     * @param string $code The source code to lex
+     * @param ErrorHandler|null $errorHandler Error handler to use for lexing errors. Defaults to
+     *                                        ErrorHandler\Throwing
      */
-    public function startLexing($code) {
+    public function startLexing($code, ErrorHandler $errorHandler = null) {
-        $scream = ini_set('xdebug.scream', '0');
+        if (null === $errorHandler) {
-
+            $errorHandler = new ErrorHandler\Throwing();
-        $this->resetErrors();
-        $this->tokens = @token_get_all($code);
-        $this->handleErrors();
-
-        if (false !== $scream) {
-            ini_set('xdebug.scream', $scream);
         }
 
         $this->code = $code; // keep the code around for __halt_compiler() handling
         $this->pos  = -1;
         $this->line =  1;
         $this->filePos = 0;
+
+        // If inline HTML occurs without preceding code, treat it as if it had a leading newline.
+        // This ensures proper composability, because having a newline is the "safe" assumption.
+        $this->prevCloseTagHasNewline = true;
+
+        $scream = ini_set('xdebug.scream', '0');
+
+        $this->resetErrors();
+        $this->tokens = @token_get_all($code);
+        $this->handleErrors($errorHandler);
+
+        if (false !== $scream) {
+            ini_set('xdebug.scream', $scream);
+        }
     }
 
     protected function resetErrors() {
@@ -80,32 +91,109 @@ class Lexer
         }
     }
 
-    protected function handleErrors() {
+    private function handleInvalidCharacterRange($start, $end, $line, ErrorHandler $errorHandler) {
+        for ($i = $start; $i < $end; $i++) {
+            $chr = $this->code[$i];
+            if ($chr === 'b' || $chr === 'B') {
+                // HHVM does not treat b" tokens correctly, so ignore these
+                continue;
+            }
+
+            if ($chr === "\0") {
+                // PHP cuts error message after null byte, so need special case
+                $errorMsg = 'Unexpected null byte';
+            } else {
+                $errorMsg = sprintf(
+                    'Unexpected character "%s" (ASCII %d)', $chr, ord($chr)
+                );
+            }
+
+            $errorHandler->handleError(new Error($errorMsg, [
+                'startLine' => $line,
+                'endLine' => $line,
+                'startFilePos' => $i,
+                'endFilePos' => $i,
+            ]));
+        }
+    }
+
+    private function isUnterminatedComment($token) {
+        return ($token[0] === T_COMMENT || $token[0] === T_DOC_COMMENT)
+            && substr($token[1], 0, 2) === '/*'
+            && substr($token[1], -2) !== '*/';
+    }
+
+    private function errorMayHaveOccurred() {
+        if (defined('HHVM_VERSION')) {
+            // In HHVM token_get_all() does not throw warnings, so we need to conservatively
+            // assume that an error occurred
+            return true;
+        }
+
         $error = error_get_last();
-        if (null === $error) {
+        return null !== $error
+            && false === strpos($error['message'], 'Undefined variable');
+    }
+
+    protected function handleErrors(ErrorHandler $errorHandler) {
+        if (!$this->errorMayHaveOccurred()) {
             return;
         }
 
-        if (preg_match(
+        // PHP's error handling for token_get_all() is rather bad, so if we want detailed
-            '~^Unterminated comment starting line ([0-9]+)$~',
+        // error information we need to compute it ourselves. Invalid character errors are
-            $error['message'], $matches
+        // detected by finding "gaps" in the token array. Unterminated comments are detected
-        )) {
+        // by checking if a trailing comment has a "*/" at the end.
-            throw new Error('Unterminated comment', (int) $matches[1]);
+
+        $filePos = 0;
+        $line = 1;
+        foreach ($this->tokens as $i => $token) {
+            $tokenValue = \is_string($token) ? $token : $token[1];
+            $tokenLen = \strlen($tokenValue);
+
+            if (substr($this->code, $filePos, $tokenLen) !== $tokenValue) {
+                // Something is missing, must be an invalid character
+                $nextFilePos = strpos($this->code, $tokenValue, $filePos);
+                $this->handleInvalidCharacterRange(
+                    $filePos, $nextFilePos, $line, $errorHandler);
+                $filePos = $nextFilePos;
             }
 
-        if (preg_match(
+            $filePos += $tokenLen;
-            '~^Unexpected character in input:  \'(.)\' \(ASCII=([0-9]+)\)~s',
+            $line += substr_count($tokenValue, "\n");
-            $error['message'], $matches
-        )) {
-            throw new Error(sprintf(
-                'Unexpected character "%s" (ASCII %d)',
-                $matches[1], $matches[2]
-            ));
         }
 
-        // PHP cuts error message after null byte, so need special case
+        if ($filePos !== \strlen($this->code)) {
-        if (preg_match('~^Unexpected character in input:  \'$~', $error['message'])) {
+            if (substr($this->code, $filePos, 2) === '/*') {
-            throw new Error('Unexpected null byte');
+                // Unlike PHP, HHVM will drop unterminated comments entirely
+                $comment = substr($this->code, $filePos);
+                $errorHandler->handleError(new Error('Unterminated comment', [
+                    'startLine' => $line,
+                    'endLine' => $line + substr_count($comment, "\n"),
+                    'startFilePos' => $filePos,
+                    'endFilePos' => $filePos + \strlen($comment),
+                ]));
+
+                // Emulate the PHP behavior
+                $isDocComment = isset($comment[3]) && $comment[3] === '*';
+                $this->tokens[] = [$isDocComment ? T_DOC_COMMENT : T_COMMENT, $comment, $line];
+            } else {
+                // Invalid characters at the end of the input
+                $this->handleInvalidCharacterRange(
+                    $filePos, \strlen($this->code), $line, $errorHandler);
+            }
+            return;
+        }
+
+        // Check for unterminated comment
+        $lastToken = $this->tokens[count($this->tokens) - 1];
+        if ($this->isUnterminatedComment($lastToken)) {
+            $errorHandler->handleError(new Error('Unterminated comment', [
+                'startLine' => $line - substr_count($lastToken[1], "\n"),
+                'endLine' => $line,
+                'startFilePos' => $filePos - \strlen($lastToken[1]),
+                'endFilePos' => $filePos,
+            ]));
         }
     }
 
@@ -166,6 +254,11 @@ class Lexer
             } elseif (!isset($this->dropTokens[$token[0]])) {
                 $value = $token[1];
                 $id = $this->tokenMap[$token[0]];
+                if (T_CLOSE_TAG === $token[0]) {
+                    $this->prevCloseTagHasNewline = false !== strpos($token[1], "\n");
+                } else if (T_INLINE_HTML === $token[0]) {
+                    $startAttributes['hasLeadingNewline'] = $this->prevCloseTagHasNewline;
+                }
 
                 $this->line += substr_count($value, "\n");
                 $this->filePos += \strlen($value);

lib/PhpParser/Lexer/Emulative.php

@@ -2,11 +2,9 @@
 
 namespace PhpParser\Lexer;
 
+use PhpParser\ErrorHandler;
 use PhpParser\Parser\Tokens;
 
-/**
- * ATTENTION: This code is WRITE-ONLY. Do not try to read it.
- */
 class Emulative extends \PhpParser\Lexer
 {
     protected $newKeywords;
@@ -53,131 +51,106 @@ class Emulative extends \PhpParser\Lexer
         $this->tokenMap[self::T_POW_EQUAL] = Tokens::T_POW_EQUAL;
     }
 
-    public function startLexing($code) {
+    public function startLexing($code, ErrorHandler $errorHandler = null) {
         $this->inObjectAccess = false;
 
-        $preprocessedCode = $this->preprocessCode($code);
+        parent::startLexing($code, $errorHandler);
-        parent::startLexing($preprocessedCode);
+        if ($this->requiresEmulation($code)) {
-        if ($preprocessedCode !== $code) {
+            $this->emulateTokens();
-            $this->postprocessTokens();
         }
-
-        // Set code property back to the original code, so __halt_compiler()
-        // handling and (start|end)FilePos attributes use the correct offsets
-        $this->code = $code;
     }
 
     /*
-     * Replaces new features in the code by ~__EMU__{NAME}__{DATA}__~ sequences.
+     * Checks if the code is potentially using features that require emulation.
-     * ~LABEL~ is never valid PHP code, that's why we can (to some degree) safely
-     * use it here.
-     * Later when preprocessing the tokens these sequences will either be replaced
-     * by real tokens or replaced with their original content (e.g. if they occurred
-     * inside a string, i.e. a place where they don't have a special meaning).
      */
-    protected function preprocessCode($code) {
+    protected function requiresEmulation($code) {
         if (version_compare(PHP_VERSION, self::PHP_7_0, '>=')) {
-            return $code;
+            return false;
         }
 
-        $code = str_replace('??', '~__EMU__COALESCE__~', $code);
+        if (preg_match('(\?\?|<=>|yield[ \n\r\t]+from)', $code)) {
-        $code = str_replace('<=>', '~__EMU__SPACESHIP__~', $code);
+            return true;
-        $code = preg_replace_callback('(yield[ \n\r\t]+from)', function($matches) {
+        }
-            // Encoding $0 in order to preserve exact whitespace
-            return '~__EMU__YIELDFROM__' . bin2hex($matches[0]) . '__~';
-        }, $code);
 
         if (version_compare(PHP_VERSION, self::PHP_5_6, '>=')) {
-            return $code;
+            return false;
         }
 
-        $code = str_replace('...', '~__EMU__ELLIPSIS__~', $code);
+        return preg_match('(\.\.\.|(?<!/)\*\*(?!/))', $code);
-        $code = preg_replace('((?<!/)\*\*=)', '~__EMU__POWEQUAL__~', $code);
-        $code = preg_replace('((?<!/)\*\*(?!/))', '~__EMU__POW__~', $code);
-
-        return $code;
     }
 
     /*
-     * Replaces the ~__EMU__...~ sequences with real tokens or their original
+     * Emulates tokens for newer PHP versions.
-     * value.
      */
-    protected function postprocessTokens() {
+    protected function emulateTokens() {
-        // we need to manually iterate and manage a count because we'll change
+        // We need to manually iterate and manage a count because we'll change
         // the tokens array on the way
+        $line = 1;
         for ($i = 0, $c = count($this->tokens); $i < $c; ++$i) {
-            // first check that the following tokens are of form ~LABEL~,
+            $replace = null;
-            // then match the __EMU__... sequence.
+            if (isset($this->tokens[$i + 1])) {
-            if ('~' === $this->tokens[$i]
+                if ($this->tokens[$i] === '?' && $this->tokens[$i + 1] === '?') {
-                && isset($this->tokens[$i + 2])
+                    array_splice($this->tokens, $i, 2, array(
-                && '~' === $this->tokens[$i + 2]
+                        array(self::T_COALESCE, '??', $line)
-                && T_STRING === $this->tokens[$i + 1][0]
+                    ));
-                && preg_match('(^__EMU__([A-Z]++)__(?:([A-Za-z0-9]++)__)?$)', $this->tokens[$i + 1][1], $matches)
+                    $c--;
-            ) {
+                    continue;
-                if ('ELLIPSIS' === $matches[1]) {
-                    $replace = array(
-                        array(self::T_ELLIPSIS, '...', $this->tokens[$i + 1][2])
-                    );
-                } else if ('POW' === $matches[1]) {
-                    $replace = array(
-                        array(self::T_POW, '**', $this->tokens[$i + 1][2])
-                    );
-                } else if ('POWEQUAL' === $matches[1]) {
-                    $replace = array(
-                        array(self::T_POW_EQUAL, '**=', $this->tokens[$i + 1][2])
-                    );
-                } else if ('COALESCE' === $matches[1]) {
-                    $replace = array(
-                        array(self::T_COALESCE, '??', $this->tokens[$i + 1][2])
-                    );
-                } else if ('SPACESHIP' === $matches[1]) {
-                    $replace = array(
-                        array(self::T_SPACESHIP, '<=>', $this->tokens[$i + 1][2]),
-                    );
-                } else if ('YIELDFROM' === $matches[1]) {
-                    $content = hex2bin($matches[2]);
-                    $replace = array(
-                        array(self::T_YIELD_FROM, $content, $this->tokens[$i + 1][2] - substr_count($content, "\n"))
-                    );
-                } else {
-                    throw new \RuntimeException('Invalid __EMU__ sequence');
                 }
-
+                if ($this->tokens[$i][0] === T_IS_SMALLER_OR_EQUAL
-                array_splice($this->tokens, $i, 3, $replace);
+                    && $this->tokens[$i + 1] === '>'
-                $c -= 3 - count($replace);
-            // for multichar tokens (e.g. strings) replace any ~__EMU__...~ sequences
-            // in their content with the original character sequence
-            } elseif (is_array($this->tokens[$i])
-                      && 0 !== strpos($this->tokens[$i][1], '__EMU__')
                 ) {
-                $this->tokens[$i][1] = preg_replace_callback(
+                    array_splice($this->tokens, $i, 2, array(
-                    '(~__EMU__([A-Z]++)__(?:([A-Za-z0-9]++)__)?~)',
+                        array(self::T_SPACESHIP, '<=>', $line)
-                    array($this, 'restoreContentCallback'),
+                    ));
-                    $this->tokens[$i][1]
+                    $c--;
-                );
+                    continue;
                 }
+                if ($this->tokens[$i] === '*' && $this->tokens[$i + 1] === '*') {
+                    array_splice($this->tokens, $i, 2, array(
+                        array(self::T_POW, '**', $line)
+                    ));
+                    $c--;
+                    continue;
+                }
+                if ($this->tokens[$i] === '*' && $this->tokens[$i + 1][0] === T_MUL_EQUAL) {
+                    array_splice($this->tokens, $i, 2, array(
+                        array(self::T_POW_EQUAL, '**=', $line)
+                    ));
+                    $c--;
+                    continue;
                 }
             }
 
-    /*
+            if (isset($this->tokens[$i + 2])) {
-     * This method is a callback for restoring EMU sequences in
+                if ($this->tokens[$i][0] === T_YIELD && $this->tokens[$i + 1][0] === T_WHITESPACE
-     * multichar tokens (like strings) to their original value.
+                    && $this->tokens[$i + 2][0] === T_STRING
-     */
+                    && !strcasecmp($this->tokens[$i + 2][1], 'from')
-    public function restoreContentCallback(array $matches) {
+                ) {
-        if ('ELLIPSIS' === $matches[1]) {
+                    array_splice($this->tokens, $i, 3, array(
-            return '...';
+                        array(
-        } else if ('POW' === $matches[1]) {
+                            self::T_YIELD_FROM,
-            return '**';
+                            $this->tokens[$i][1] . $this->tokens[$i + 1][1] . $this->tokens[$i + 2][1],
-        } else if ('POWEQUAL' === $matches[1]) {
+                            $line
-            return '**=';
+                        )
-        } else if ('COALESCE' === $matches[1]) {
+                    ));
-            return '??';
+                    $c -= 2;
-        } else if ('SPACESHIP' === $matches[1]) {
+                    $line += substr_count($this->tokens[$i][1], "\n");
-            return '<=>';
+                    continue;
-        } else if ('YIELDFROM' === $matches[1]) {
+                }
-            return hex2bin($matches[2]);
+                if ($this->tokens[$i] === '.' && $this->tokens[$i + 1] === '.'
-        } else {
+                    && $this->tokens[$i + 2] === '.'
-            return $matches[0];
+                ) {
+                    array_splice($this->tokens, $i, 3, array(
+                        array(self::T_ELLIPSIS, '...', $line)
+                    ));
+                    $c -= 2;
+                    continue;
+                }
+            }
+
+            if (\is_array($this->tokens[$i])) {
+                $line += substr_count($this->tokens[$i][1], "\n");
+            }
         }
     }
 

lib/PhpParser/Node.php

@@ -41,6 +41,15 @@ interface Node
      */
     public function getDocComment();
 
+    /**
+     * Sets the doc comment of the node.
+     *
+     * This will either replace an existing doc comment or add it to the comments array.
+     *
+     * @param Comment\Doc $docComment Doc comment to set
+     */
+    public function setDocComment(Comment\Doc $docComment);
+
     /**
      * Sets an attribute on a node.
      *

lib/PhpParser/Node/Expr/ArrayDimFetch.php

@@ -24,7 +24,7 @@ class ArrayDimFetch extends Expr
         $this->dim = $dim;
     }
 
-    public function getSubnodeNames() {
+    public function getSubNodeNames() {
         return array('var', 'dim');
     }
 }

lib/PhpParser/Node/Expr/ClassConstFetch.php

@@ -2,21 +2,21 @@
 
 namespace PhpParser\Node\Expr;
 
-use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
 
 class ClassConstFetch extends Expr
 {
     /** @var Name|Expr Class name */
     public $class;
-    /** @var string Constant name */
+    /** @var string|Error Constant name */
     public $name;
 
     /**
      * Constructs a class const fetch node.
      *
      * @param Name|Expr    $class      Class name
-     * @param string    $name       Constant name
+     * @param string|Error $name       Constant name
      * @param array        $attributes Additional attributes
      */
     public function __construct($class, $name, array $attributes = array()) {

lib/PhpParser/Node/Expr/Closure.php

@@ -16,7 +16,7 @@ class Closure extends Expr implements FunctionLike
     public $params;
     /** @var ClosureUse[] use()s */
     public $uses;
-    /** @var null|string|Node\Name Return type */
+    /** @var null|string|Node\Name|Node\NullableType Return type */
     public $returnType;
     /** @var Node[] Statements */
     public $stmts;

lib/PhpParser/Node/Expr/ConstFetch.php

@@ -2,8 +2,8 @@
 
 namespace PhpParser\Node\Expr;
 
-use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
 
 class ConstFetch extends Expr
 {

lib/PhpParser/Node/Expr/Error.php

@@ -0,0 +1,27 @@
+<?php
+
+namespace PhpParser\Node\Expr;
+
+use PhpParser\Node\Expr;
+
+/**
+ * Error node used during parsing with error recovery.
+ *
+ * An error node may be placed at a position where an expression is required, but an error occurred.
+ * Error nodes will not be present if the parser is run in throwOnError mode (the default).
+ */
+class Error extends Expr
+{
+    /**
+     * Constructs an error node.
+     *
+     * @param array $attributes Additional attributes
+     */
+    public function __construct(array $attributes = array()) {
+        parent::__construct($attributes);
+    }
+
+    public function getSubNodeNames() {
+        return array();
+    }
+}

lib/PhpParser/Node/Expr/Instanceof_.php

@@ -2,8 +2,8 @@
 
 namespace PhpParser\Node\Expr;
 
-use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
 
 class Instanceof_ extends Expr
 {

lib/PhpParser/Node/Expr/StaticPropertyFetch.php

@@ -2,8 +2,8 @@
 
 namespace PhpParser\Node\Expr;
 
-use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
 
 class StaticPropertyFetch extends Expr
 {

lib/PhpParser/Node/FunctionLike.php

@@ -23,7 +23,7 @@ interface FunctionLike extends Node
     /**
      * Get the declared return type or null
      * 
-     * @return null|string|Node\Name
+     * @return null|string|Node\Name|Node\NullableType
      */
     public function getReturnType();
 

lib/PhpParser/Node/Name.php

@@ -6,22 +6,21 @@ use PhpParser\NodeAbstract;
 
 class Name extends NodeAbstract
 {
-    /** @var string[] Parts of the name */
+    /**
+     * @var string[] Parts of the name
+     * @deprecated Avoid directly accessing $parts, use methods instead.
+     */
     public $parts;
 
     /**
      * Constructs a name node.
      *
-     * @param string|array $parts      Parts of the name (or name as string)
+     * @param string|array|self $name       Name as string, part array or Name instance (copy ctor)
      * @param array             $attributes Additional attributes
      */
-    public function __construct($parts, array $attributes = array()) {
+    public function __construct($name, array $attributes = array()) {
-        if (!is_array($parts)) {
-            $parts = explode('\\', $parts);
-        }
-
         parent::__construct($attributes);
-        $this->parts = $parts;
+        $this->parts = self::prepareName($name);
     }
 
     public function getSubNodeNames() {
@@ -83,14 +82,13 @@ class Name extends NodeAbstract
     }
 
     /**
-     * Returns a string representation of the name by imploding the namespace parts with a separator.
+     * Returns a string representation of the name by imploding the namespace parts with the
-     *
+     * namespace separator.
-     * @param string $separator The separator to use (defaults to the namespace separator \)
      *
      * @return string String representation
      */
-    public function toString($separator = '\\') {
+    public function toString() {
-        return implode($separator, $this->parts);
+        return implode('\\', $this->parts);
     }
 
     /**
@@ -103,43 +101,21 @@ class Name extends NodeAbstract
         return implode('\\', $this->parts);
     }
 
-    /**
-     * Prepends a name to this name.
-     *
-     * @deprecated Use Name::concat($name1, $name2) instead
-     *
-     * @param string|array|self $name Name to prepend
-     */
-    public function prepend($name) {
-        $this->parts = array_merge(self::prepareName($name), $this->parts);
-    }
-
-    /**
-     * Appends a name to this name.
-     *
-     * @deprecated Use Name::concat($name1, $name2) instead
-     *
-     * @param string|array|self $name Name to append
-     */
-    public function append($name) {
-        $this->parts = array_merge($this->parts, self::prepareName($name));
-    }
-
     /**
      * Gets a slice of a name (similar to array_slice).
      *
      * This method returns a new instance of the same type as the original and with the same
      * attributes.
      *
-     * If the slice is empty, a Name with an empty parts array is returned. While this is
+     * If the slice is empty, null is returned. The null value will be correctly handled in
-     * meaningless in itself, it works correctly in conjunction with concat().
+     * concatenations using concat().
      *
      * Offset and length have the same meaning as in array_slice().
      *
      * @param int      $offset Offset to start the slice at (may be negative)
      * @param int|null $length Length of the slice (may be negative)
      *
-     * @return static Sliced name
+     * @return static|null Sliced name
      */
     public function slice($offset, $length = null) {
         $numParts = count($this->parts);
@@ -158,6 +134,11 @@ class Name extends NodeAbstract
             }
         }
 
+        if ($realLength === 0) {
+            // Empty slice is represented as null
+            return null;
+        }
+
         return new static(array_slice($this->parts, $realOffset, $realLength), $this->attributes);
     }
 
@@ -167,17 +148,30 @@ class Name extends NodeAbstract
      * The type of the generated instance depends on which class this method is called on, for
      * example Name\FullyQualified::concat() will yield a Name\FullyQualified instance.
      *
-     * @param string|array|self $name1      The first name
+     * If one of the arguments is null, a new instance of the other name will be returned. If both
-     * @param string|array|self $name2      The second name
+     * arguments are null, null will be returned. As such, writing
+     *     Name::concat($namespace, $shortName)
+     * where $namespace is a Name node or null will work as expected.
+     *
+     * @param string|array|self|null $name1      The first name
+     * @param string|array|self|null $name2      The second name
      * @param array                  $attributes Attributes to assign to concatenated name
      *
-     * @return static Concatenated name
+     * @return static|null Concatenated name
      */
     public static function concat($name1, $name2, array $attributes = []) {
+        if (null === $name1 && null === $name2) {
+            return null;
+        } elseif (null === $name1) {
+            return new static(self::prepareName($name2), $attributes);
+        } else if (null === $name2) {
+            return new static(self::prepareName($name1), $attributes);
+        } else {
             return new static(
                 array_merge(self::prepareName($name1), self::prepareName($name2)), $attributes
             );
         }
+    }
 
     /**
      * Prepares a (string, array or Name node) name for use in name changing methods by converting
@@ -188,16 +182,16 @@ class Name extends NodeAbstract
      * @return array Prepared name
      */
     private static function prepareName($name) {
-        if (is_string($name)) {
+        if (\is_string($name)) {
             return explode('\\', $name);
-        } elseif (is_array($name)) {
+        } elseif (\is_array($name)) {
             return $name;
         } elseif ($name instanceof self) {
             return $name->parts;
         }
 
         throw new \InvalidArgumentException(
-            'When changing a name you need to pass either a string, an array or a Name node'
+            'Expected string, array of parts or Name instance'
         );
     }
 }

lib/PhpParser/Node/NullableType.php

@@ -12,12 +12,12 @@ class NullableType extends NodeAbstract
     /**
      * Constructs a nullable type (wrapping another type).
      *
-     * @param string|Name $flags       Type
+     * @param string|Name $type       Type
      * @param array       $attributes Additional attributes
      */
-    public function __construct($flags, array $attributes = array()) {
+    public function __construct($type, array $attributes = array()) {
         parent::__construct($attributes);
-        $this->type = $flags;
+        $this->type = $type;
     }
 
     public function getSubNodeNames() {

lib/PhpParser/Node/Param.php

@@ -2,12 +2,11 @@
 
 namespace PhpParser\Node;
 
-use PhpParser\Error;
 use PhpParser\NodeAbstract;
 
 class Param extends NodeAbstract
 {
-    /** @var null|string|Name Typehint */
+    /** @var null|string|Name|NullableType Typehint */
     public $type;
     /** @var bool Whether parameter is passed by reference */
     public $byRef;
@@ -23,7 +22,7 @@ class Param extends NodeAbstract
      *
      * @param string                        $name       Name
      * @param null|Expr                     $default    Default value
-     * @param null|string|Name $type       Typehint
+     * @param null|string|Name|NullableType $type       Typehint
      * @param bool                          $byRef      Whether is passed by reference
      * @param bool                          $variadic   Whether this is a variadic argument
      * @param array                         $attributes Additional attributes
@@ -35,10 +34,6 @@ class Param extends NodeAbstract
         $this->variadic = $variadic;
         $this->name = $name;
         $this->default = $default;
-
-        if ($variadic && null !== $default) {
-            throw new Error('Variadic parameter cannot have a default value', $default->getAttributes());
-        }
     }
 
     public function getSubNodeNames() {

lib/PhpParser/Node/Scalar/Encapsed.php

@@ -2,11 +2,12 @@
 
 namespace PhpParser\Node\Scalar;
 
+use PhpParser\Node\Expr;
 use PhpParser\Node\Scalar;
 
 class Encapsed extends Scalar
 {
-    /** @var array Encaps list */
+    /** @var Expr[] list of string parts */
     public $parts;
 
     /**

lib/PhpParser/Node/Stmt/ClassConst.php

@@ -3,7 +3,6 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
-use PhpParser\Error;
 
 class ClassConst extends Node\Stmt
 {
@@ -20,16 +19,6 @@ class ClassConst extends Node\Stmt
      * @param array         $attributes Additional attributes
      */
     public function __construct(array $consts, $flags = 0, array $attributes = array()) {
-        if ($flags & Class_::MODIFIER_STATIC) {
-            throw new Error("Cannot use 'static' as constant modifier");
-        }
-        if ($flags & Class_::MODIFIER_ABSTRACT) {
-            throw new Error("Cannot use 'abstract' as constant modifier");
-        }
-        if ($flags & Class_::MODIFIER_FINAL) {
-            throw new Error("Cannot use 'final' as constant modifier");
-        }
-
         parent::__construct($attributes);
         $this->flags = $flags;
         $this->consts = $consts;

lib/PhpParser/Node/Stmt/ClassLike.php

@@ -5,7 +5,7 @@ namespace PhpParser\Node\Stmt;
 use PhpParser\Node;
 
 abstract class ClassLike extends Node\Stmt {
-    /** @var string Name */
+    /** @var string|null Name */
     public $name;
     /** @var Node[] Statements */
     public $stmts;

lib/PhpParser/Node/Stmt/ClassMethod.php

@@ -4,7 +4,6 @@ namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
 use PhpParser\Node\FunctionLike;
-use PhpParser\Error;
 
 class ClassMethod extends Node\Stmt implements FunctionLike
 {
@@ -16,7 +15,7 @@ class ClassMethod extends Node\Stmt implements FunctionLike
     public $name;
     /** @var Node\Param[] Parameters */
     public $params;
-    /** @var null|string|Node\Name Return type */
+    /** @var null|string|Node\Name|Node\NullableType Return type */
     public $returnType;
     /** @var Node[] Statements */
     public $stmts;
@@ -46,17 +45,6 @@ class ClassMethod extends Node\Stmt implements FunctionLike
         $this->params = isset($subNodes['params']) ? $subNodes['params'] : array();
         $this->returnType = isset($subNodes['returnType']) ? $subNodes['returnType'] : null;
         $this->stmts = array_key_exists('stmts', $subNodes) ? $subNodes['stmts'] : array();
-
-        if ($this->flags & Class_::MODIFIER_STATIC) {
-            switch (strtolower($this->name)) {
-                case '__construct':
-                    throw new Error(sprintf('Constructor %s() cannot be static', $this->name));
-                case '__destruct':
-                    throw new Error(sprintf('Destructor %s() cannot be static', $this->name));
-                case '__clone':
-                    throw new Error(sprintf('Clone method %s() cannot be static', $this->name));
-            }
-        }
     }
 
     public function getSubNodeNames() {

lib/PhpParser/Node/Stmt/Class_.php

@@ -2,8 +2,8 @@
 
 namespace PhpParser\Node\Stmt;
 
-use PhpParser\Node;
 use PhpParser\Error;
+use PhpParser\Node;
 
 class Class_ extends ClassLike
 {
@@ -52,26 +52,6 @@ class Class_ extends ClassLike
         $this->extends = isset($subNodes['extends']) ? $subNodes['extends'] : null;
         $this->implements = isset($subNodes['implements']) ? $subNodes['implements'] : array();
         $this->stmts = isset($subNodes['stmts']) ? $subNodes['stmts'] : array();
-
-        if (null !== $this->name && isset(self::$specialNames[strtolower($this->name)])) {
-            throw new Error(sprintf('Cannot use \'%s\' as class name as it is reserved', $this->name));
-        }
-
-        if (isset(self::$specialNames[strtolower($this->extends)])) {
-            throw new Error(
-                sprintf('Cannot use \'%s\' as class name as it is reserved', $this->extends),
-                $this->extends->getAttributes()
-            );
-        }
-
-        foreach ($this->implements as $interface) {
-            if (isset(self::$specialNames[strtolower($interface)])) {
-                throw new Error(
-                    sprintf('Cannot use \'%s\' as interface name as it is reserved', $interface),
-                    $interface->getAttributes()
-                );
-            }
-        }
     }
 
     public function getSubNodeNames() {

lib/PhpParser/Node/Stmt/Declare_.php

@@ -3,6 +3,7 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
+
 class Declare_ extends Node\Stmt
 {
     /** @var DeclareDeclare[] List of declares */

lib/PhpParser/Node/Stmt/Function_.php

@@ -13,7 +13,7 @@ class Function_ extends Node\Stmt implements FunctionLike
     public $name;
     /** @var Node\Param[] Parameters */
     public $params;
-    /** @var null|string|Node\Name Return type */
+    /** @var null|string|Node\Name|Node\NullableType Return type */
     public $returnType;
     /** @var Node[] Statements */
     public $stmts;

lib/PhpParser/Node/Stmt/GroupUse.php

@@ -2,8 +2,8 @@
 
 namespace PhpParser\Node\Stmt;
 
-use PhpParser\Node\Stmt;
 use PhpParser\Node\Name;
+use PhpParser\Node\Stmt;
 
 class GroupUse extends Stmt
 {

lib/PhpParser/Node/Stmt/Interface_.php

@@ -3,19 +3,12 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
-use PhpParser\Error;
 
 class Interface_ extends ClassLike
 {
     /** @var Node\Name[] Extended interfaces */
     public $extends;
 
-    protected static $specialNames = array(
-        'self'   => true,
-        'parent' => true,
-        'static' => true,
-    );
-
     /**
      * Constructs a class node.
      *
@@ -30,19 +23,6 @@ class Interface_ extends ClassLike
         $this->name = $name;
         $this->extends = isset($subNodes['extends']) ? $subNodes['extends'] : array();
         $this->stmts = isset($subNodes['stmts']) ? $subNodes['stmts'] : array();
-
-        if (isset(self::$specialNames[strtolower($this->name)])) {
-            throw new Error(sprintf('Cannot use \'%s\' as class name as it is reserved', $this->name));
-        }
-
-        foreach ($this->extends as $interface) {
-            if (isset(self::$specialNames[strtolower($interface)])) {
-                throw new Error(
-                    sprintf('Cannot use \'%s\' as interface name as it is reserved', $interface),
-                    $interface->getAttributes()
-                );
-            }
-        }
     }
 
     public function getSubNodeNames() {

lib/PhpParser/Node/Stmt/Namespace_.php

@@ -3,7 +3,6 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
-use PhpParser\Error;
 
 class Namespace_ extends Node\Stmt
 {
@@ -12,12 +11,6 @@ class Namespace_ extends Node\Stmt
     /** @var Node[] Statements */
     public $stmts;
 
-    protected static $specialNames = array(
-        'self'   => true,
-        'parent' => true,
-        'static' => true,
-    );
-
     /**
      * Constructs a namespace node.
      *
@@ -29,21 +22,6 @@ class Namespace_ extends Node\Stmt
         parent::__construct($attributes);
         $this->name = $name;
         $this->stmts = $stmts;
-
-        if (isset(self::$specialNames[strtolower($this->name)])) {
-            throw new Error(
-                sprintf('Cannot use \'%s\' as namespace name', $this->name),
-                $this->name->getAttributes()
-            );
-        }
-
-        if (null !== $this->stmts) {
-            foreach ($this->stmts as $stmt) {
-                if ($stmt instanceof self) {
-                    throw new Error('Namespace declarations cannot be nested', $stmt->getAttributes());
-                }
-            }
-        }
     }
 
     public function getSubNodeNames() {

lib/PhpParser/Node/Stmt/Property.php

@@ -3,7 +3,6 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
-use PhpParser\Error;
 
 class Property extends Node\Stmt
 {
@@ -23,14 +22,6 @@ class Property extends Node\Stmt
      * @param array              $attributes Additional attributes
      */
     public function __construct($flags, array $props, array $attributes = array()) {
-        if ($flags & Class_::MODIFIER_ABSTRACT) {
-            throw new Error('Properties cannot be declared abstract');
-        }
-
-        if ($flags & Class_::MODIFIER_FINAL) {
-            throw new Error('Properties cannot be declared final');
-        }
-
         parent::__construct($attributes);
         $this->flags = $flags;
         $this->type = $flags;

lib/PhpParser/Node/Stmt/TryCatch.php

@@ -3,7 +3,6 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
-use PhpParser\Error;
 
 class TryCatch extends Node\Stmt
 {
@@ -23,10 +22,6 @@ class TryCatch extends Node\Stmt
      * @param array|null    $attributes Additional attributes
      */
     public function __construct(array $stmts, array $catches, Finally_ $finally = null, array $attributes = array()) {
-        if (empty($catches) && null === $finally) {
-            throw new Error('Cannot use try without catch or finally');
-        }
-
         parent::__construct($attributes);
         $this->stmts = $stmts;
         $this->catches = $catches;

lib/PhpParser/Node/Stmt/UseUse.php

@@ -3,7 +3,6 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
-use PhpParser\Error;
 
 class UseUse extends Node\Stmt
 {
@@ -27,13 +26,6 @@ class UseUse extends Node\Stmt
             $alias = $name->getLast();
         }
 
-        if ('self' == strtolower($alias) || 'parent' == strtolower($alias)) {
-            throw new Error(sprintf(
-                'Cannot use %s as %s because \'%2$s\' is a special class name',
-                $name, $alias
-            ));
-        }
-
         parent::__construct($attributes);
         $this->type = $type;
         $this->name = $name;

lib/PhpParser/NodeAbstract.php

@@ -2,7 +2,7 @@
 
 namespace PhpParser;
 
-abstract class NodeAbstract implements Node
+abstract class NodeAbstract implements Node, \JsonSerializable
 {
     protected $attributes;
 
@@ -63,6 +63,28 @@ abstract class NodeAbstract implements Node
         return $lastComment;
     }
 
+    /**
+     * Sets the doc comment of the node.
+     *
+     * This will either replace an existing doc comment or add it to the comments array.
+     *
+     * @param Comment\Doc $docComment Doc comment to set
+     */
+    public function setDocComment(Comment\Doc $docComment) {
+        $comments = $this->getAttribute('comments', []);
+
+        $numComments = count($comments);
+        if ($numComments > 0 && $comments[$numComments - 1] instanceof Comment\Doc) {
+            // Replace existing doc comment
+            $comments[$numComments - 1] = $docComment;
+        } else {
+            // Append new comment
+            $comments[] = $docComment;
+        }
+
+        $this->setAttribute('comments', $comments);
+    }
+
     public function setAttribute($key, $value) {
         $this->attributes[$key] = $value;
     }
@@ -82,4 +104,8 @@ abstract class NodeAbstract implements Node
     public function getAttributes() {
         return $this->attributes;
     }
+
+    public function jsonSerialize() {
+        return ['nodeType' => $this->getType()] + get_object_vars($this);
+    }
 }

lib/PhpParser/NodeDumper.php

@@ -11,27 +11,46 @@ use PhpParser\Node\Stmt\UseUse;
 class NodeDumper
 {
     private $dumpComments;
+    private $dumpPositions;
+    private $code;
 
     /**
      * Constructs a NodeDumper.
      *
-     * @param array $options Boolean option 'dumpComments' controls whether comments should be
+     * Supported options:
-     *                       dumped
+     *  * bool dumpComments: Whether comments should be dumped.
+     *  * bool dumpPositions: Whether line/offset information should be dumped. To dump offset
+     *                        information, the code needs to be passed to dump().
+     *
+     * @param array $options Options (see description)
      */
     public function __construct(array $options = []) {
         $this->dumpComments = !empty($options['dumpComments']);
+        $this->dumpPositions = !empty($options['dumpPositions']);
     }
 
     /**
      * Dumps a node or array.
      *
      * @param array|Node  $node Node or array to dump
+     * @param string|null $code Code corresponding to dumped AST. This only needs to be passed if
+     *                          the dumpPositions option is enabled and the dumping of node offsets
+     *                          is desired.
      *
      * @return string Dumped value
      */
-    public function dump($node) {
+    public function dump($node, $code = null) {
+        $this->code = $code;
+        return $this->dumpRecursive($node);
+    }
+
+    protected function dumpRecursive($node) {
         if ($node instanceof Node) {
-            $r = $node->getType() . '(';
+            $r = $node->getType();
+            if ($this->dumpPositions && null !== $p = $this->dumpPosition($node)) {
+                $r .= $p;
+            }
+            $r .= '(';
 
             foreach ($node->getSubNodeNames() as $key) {
                 $r .= "\n    " . $key . ': ';
@@ -55,12 +74,12 @@ class NodeDumper
                         $r .= $value;
                     }
                 } else {
-                    $r .= str_replace("\n", "\n    ", $this->dump($value));
+                    $r .= str_replace("\n", "\n    ", $this->dumpRecursive($value));
                 }
             }
 
             if ($this->dumpComments && $comments = $node->getAttribute('comments')) {
-                $r .= "\n    comments: " . str_replace("\n", "\n    ", $this->dump($comments));
+                $r .= "\n    comments: " . str_replace("\n", "\n    ", $this->dumpRecursive($comments));
             }
         } elseif (is_array($node)) {
             $r = 'array(';
@@ -77,7 +96,7 @@ class NodeDumper
                 } elseif (is_scalar($value)) {
                     $r .= $value;
                 } else {
-                    $r .= str_replace("\n", "\n    ", $this->dump($value));
+                    $r .= str_replace("\n", "\n    ", $this->dumpRecursive($value));
                 }
             }
         } elseif ($node instanceof Comment) {
@@ -144,4 +163,34 @@ class NodeDumper
         }
         return $map[$type] . ' (' . $type . ')';
     }
+
+    protected function dumpPosition(Node $node) {
+        if (!$node->hasAttribute('startLine') || !$node->hasAttribute('endLine')) {
+            return null;
+        }
+
+        $start = $node->getAttribute('startLine');
+        $end = $node->getAttribute('endLine');
+        if ($node->hasAttribute('startFilePos') && $node->hasAttribute('endFilePos')
+            && null !== $this->code
+        ) {
+            $start .= ':' . $this->toColumn($this->code, $node->getAttribute('startFilePos'));
+            $end .= ':' . $this->toColumn($this->code, $node->getAttribute('endFilePos'));
+        }
+        return "[$start - $end]";
+    }
+
+    // Copied from Error class
+    private function toColumn($code, $pos) {
+        if ($pos > strlen($code)) {
+            throw new \RuntimeException('Invalid position information');
+        }
+
+        $lineStartPos = strrpos($code, "\n", $pos - strlen($code));
+        if (false === $lineStartPos) {
+            $lineStartPos = -1;
+        }
+
+        return $pos - $lineStartPos;
+    }
 }

lib/PhpParser/NodeTraverser.php

@@ -5,23 +5,42 @@ namespace PhpParser;
 class NodeTraverser implements NodeTraverserInterface
 {
     /**
-     * @var NodeVisitor[] Visitors
+     * If NodeVisitor::enterNode() returns DONT_TRAVERSE_CHILDREN, child nodes
+     * of the current node will not be traversed for any visitors.
+     *
+     * For subsequent visitors enterNode() will still be called on the current
+     * node and leaveNode() will also be invoked for the current node.
      */
-    protected $visitors;
+    const DONT_TRAVERSE_CHILDREN = 1;
 
     /**
-     * @var bool
+     * If NodeVisitor::enterNode() or NodeVisitor::leaveNode() returns
+     * STOP_TRAVERSAL, traversal is aborted.
+     *
+     * The afterTraverse() method will still be invoked.
      */
-    private $cloneNodes;
+    const STOP_TRAVERSAL = 2;
+
+    /**
+     * If NodeVisitor::leaveNode() returns REMOVE_NODE for a node that occurs
+     * in an array, it will be removed from the array.
+     *
+     * For subsequent visitors leaveNode() will still be invoked for the
+     * removed node.
+     */
+    const REMOVE_NODE = false;
+
+    /** @var NodeVisitor[] Visitors */
+    protected $visitors;
+
+    /** @var bool Whether traversal should be stopped */
+    protected $stopTraversal;
 
     /**
      * Constructs a node traverser.
-     *
-     * @param bool $cloneNodes Should the traverser clone the nodes when traversing the AST
      */
-    public function __construct($cloneNodes = false) {
+    public function __construct() {
         $this->visitors = array();
-        $this->cloneNodes = $cloneNodes;
     }
 
     /**
@@ -55,6 +74,8 @@ class NodeTraverser implements NodeTraverserInterface
      * @return Node[] Traversed array of nodes
      */
     public function traverse(array $nodes) {
+        $this->stopTraversal = false;
+
         foreach ($this->visitors as $visitor) {
             if (null !== $return = $visitor->beforeTraverse($nodes)) {
                 $nodes = $return;
@@ -73,21 +94,23 @@ class NodeTraverser implements NodeTraverserInterface
     }
 
     protected function traverseNode(Node $node) {
-        if ($this->cloneNodes) {
-            $node = clone $node;
-        }
-
         foreach ($node->getSubNodeNames() as $name) {
             $subNode =& $node->$name;
 
             if (is_array($subNode)) {
                 $subNode = $this->traverseArray($subNode);
+                if ($this->stopTraversal) {
+                    break;
+                }
             } elseif ($subNode instanceof Node) {
                 $traverseChildren = true;
                 foreach ($this->visitors as $visitor) {
                     $return = $visitor->enterNode($subNode);
                     if (self::DONT_TRAVERSE_CHILDREN === $return) {
                         $traverseChildren = false;
+                    } else if (self::STOP_TRAVERSAL === $return) {
+                        $this->stopTraversal = true;
+                        break 2;
                     } else if (null !== $return) {
                         $subNode = $return;
                     }
@@ -95,10 +118,17 @@ class NodeTraverser implements NodeTraverserInterface
 
                 if ($traverseChildren) {
                     $subNode = $this->traverseNode($subNode);
+                    if ($this->stopTraversal) {
+                        break;
+                    }
                 }
 
                 foreach ($this->visitors as $visitor) {
-                    if (null !== $return = $visitor->leaveNode($subNode)) {
+                    $return = $visitor->leaveNode($subNode);
+                    if (self::STOP_TRAVERSAL === $return) {
+                        $this->stopTraversal = true;
+                        break 2;
+                    } else if (null !== $return) {
                         if (is_array($return)) {
                             throw new \LogicException(
                                 'leaveNode() may only return an array ' .
@@ -120,12 +150,18 @@ class NodeTraverser implements NodeTraverserInterface
         foreach ($nodes as $i => &$node) {
             if (is_array($node)) {
                 $node = $this->traverseArray($node);
+                if ($this->stopTraversal) {
+                    break;
+                }
             } elseif ($node instanceof Node) {
                 $traverseChildren = true;
                 foreach ($this->visitors as $visitor) {
                     $return = $visitor->enterNode($node);
                     if (self::DONT_TRAVERSE_CHILDREN === $return) {
                         $traverseChildren = false;
+                    } else if (self::STOP_TRAVERSAL === $return) {
+                        $this->stopTraversal = true;
+                        break 2;
                     } else if (null !== $return) {
                         $node = $return;
                     }
@@ -133,6 +169,9 @@ class NodeTraverser implements NodeTraverserInterface
 
                 if ($traverseChildren) {
                     $node = $this->traverseNode($node);
+                    if ($this->stopTraversal) {
+                        break;
+                    }
                 }
 
                 foreach ($this->visitors as $visitor) {
@@ -141,6 +180,9 @@ class NodeTraverser implements NodeTraverserInterface
                     if (self::REMOVE_NODE === $return) {
                         $doNodes[] = array($i, array());
                         break;
+                    } else if (self::STOP_TRAVERSAL === $return) {
+                        $this->stopTraversal = true;
+                        break 2;
                     } elseif (is_array($return)) {
                         $doNodes[] = array($i, $return);
                         break;

lib/PhpParser/NodeTraverserInterface.php

@@ -4,24 +4,6 @@ namespace PhpParser;
 
 interface NodeTraverserInterface
 {
-    /**
-     * If NodeVisitor::enterNode() returns DONT_TRAVERSE_CHILDREN, child nodes
-     * of the current node will not be traversed for any visitors.
-     *
-     * For subsequent visitors enterNode() will still be called on the current
-     * node and leaveNode() will also be invoked for the current node.
-     */
-    const DONT_TRAVERSE_CHILDREN = 1;
-
-    /**
-     * If NodeVisitor::leaveNode() returns REMOVE_NODE for a node that occurs
-     * in an array, it will be removed from the array.
-     *
-     * For subsequent visitors leaveNode() will still be invoked for the
-     * removed node.
-     */
-    const REMOVE_NODE = false;
-
     /**
      * Adds a visitor.
      *

lib/PhpParser/NodeVisitor.php

@@ -21,12 +21,18 @@ interface NodeVisitor
      * Called when entering a node.
      *
      * Return value semantics:
-     *  * null:      $node stays as-is
+     *  * null
-     *  * otherwise: $node is set to the return value
+     *        => $node stays as-is
+     *  * NodeTraverser::DONT_TRAVERSE_CHILDREN
+     *        => Children of $node are not traversed. $node stays as-is
+     *  * NodeTraverser::STOP_TRAVERSAL
+     *        => Traversal is aborted. $node stays as-is
+     *  * otherwise
+     *        => $node is set to the return value
      *
      * @param Node $node Node
      *
-     * @return null|Node Node
+     * @return null|int|Node Node
      */
     public function enterNode(Node $node);
 
@@ -34,14 +40,20 @@ interface NodeVisitor
      * Called when leaving a node.
      *
      * Return value semantics:
-     *  * null:      $node stays as-is
+     *  * null
-     *  * false:     $node is removed from the parent array
+     *        => $node stays as-is
-     *  * array:     The return value is merged into the parent array (at the position of the $node)
+     *  * NodeTraverser::REMOVE_NODE
-     *  * otherwise: $node is set to the return value
+     *        => $node is removed from the parent array
+     *  * NodeTraverser::STOP_TRAVERSAL
+     *        => Traversal is aborted. $node stays as-is
+     *  * array (of Nodes)
+     *        => The return value is merged into the parent array (at the position of the $node)
+     *  * otherwise
+     *        => $node is set to the return value
      *
      * @param Node $node Node
      *
-     * @return null|Node|false|Node[] Node
+     * @return null|false|int|Node|Node[] Node
      */
     public function leaveNode(Node $node);
 

lib/PhpParser/NodeVisitor/NameResolver.php

@@ -2,13 +2,14 @@
 
 namespace PhpParser\NodeVisitor;
 
-use PhpParser\NodeVisitorAbstract;
 use PhpParser\Error;
+use PhpParser\ErrorHandler;
 use PhpParser\Node;
+use PhpParser\Node\Expr;
 use PhpParser\Node\Name;
 use PhpParser\Node\Name\FullyQualified;
-use PhpParser\Node\Expr;
 use PhpParser\Node\Stmt;
+use PhpParser\NodeVisitorAbstract;
 
 class NameResolver extends NodeVisitorAbstract
 {
@@ -18,6 +19,26 @@ class NameResolver extends NodeVisitorAbstract
     /** @var array Map of format [aliasType => [aliasName => originalName]] */
     protected $aliases;
 
+    /** @var ErrorHandler Error handler */
+    protected $errorHandler;
+
+    /** @var bool Whether to preserve original names */
+    protected $preserveOriginalNames;
+
+    /**
+     * Constructs a name resolution visitor.
+     *
+     * Options: If "preserveOriginalNames" is enabled, an "originalName" attribute will be added to
+     * all name nodes that underwent resolution.
+     *
+     * @param ErrorHandler|null $errorHandler Error handler
+     * @param array $options Options
+     */
+    public function __construct(ErrorHandler $errorHandler = null, array $options = []) {
+        $this->errorHandler = $errorHandler ?: new ErrorHandler\Throwing;
+        $this->preserveOriginalNames = !empty($options['preserveOriginalNames']);
+    }
+
     public function beforeTraverse(array $nodes) {
         $this->resetState();
     }
@@ -99,7 +120,10 @@ class NameResolver extends NodeVisitorAbstract
                     }
                 }
             }
-
+        } elseif ($node instanceof Node\NullableType) {
+            if ($node->type instanceof Name) {
+                $node->type = $this->resolveClassName($node->type);
+            }
         }
     }
 
@@ -132,13 +156,14 @@ class NameResolver extends NodeVisitorAbstract
                 Stmt\Use_::TYPE_CONSTANT => 'const ',
             );
 
-            throw new Error(
+            $this->errorHandler->handleError(new Error(
                 sprintf(
                     'Cannot use %s%s as %s because the name is already in use',
                     $typeStringMap[$type], $name, $use->alias
                 ),
-                $use->getLine()
+                $use->getAttributes()
-            );
+            ));
+            return;
         }
 
         $this->aliases[$type][$aliasName] = $name;
@@ -157,15 +182,21 @@ class NameResolver extends NodeVisitorAbstract
     }
 
     protected function resolveClassName(Name $name) {
+        if ($this->preserveOriginalNames) {
+            // Save the original name
+            $originalName = $name;
+            $name = clone $originalName;
+            $name->setAttribute('originalName', $originalName);
+        }
+
         // don't resolve special class names
         if (in_array(strtolower($name->toString()), array('self', 'parent', 'static'))) {
             if (!$name->isUnqualified()) {
-                throw new Error(
+                $this->errorHandler->handleError(new Error(
                     sprintf("'\\%s' is an invalid class name", $name->toString()),
-                    $name->getLine()
+                    $name->getAttributes()
-                );
+                ));
             }
-
             return $name;
         }
 
@@ -181,15 +212,18 @@ class NameResolver extends NodeVisitorAbstract
             return FullyQualified::concat($alias, $name->slice(1), $name->getAttributes());
         }
 
-        if (null !== $this->namespace) {
         // if no alias exists prepend current namespace
         return FullyQualified::concat($this->namespace, $name, $name->getAttributes());
     }
 
-        return new FullyQualified($name->parts, $name->getAttributes());
+    protected function resolveOtherName(Name $name, $type) {
+        if ($this->preserveOriginalNames) {
+            // Save the original name
+            $originalName = $name;
+            $name = clone $originalName;
+            $name->setAttribute('originalName', $originalName);
         }
 
-    protected function resolveOtherName(Name $name, $type) {
         // fully qualified names are already resolved
         if ($name->isFullyQualified()) {
             return $name;
@@ -208,28 +242,28 @@ class NameResolver extends NodeVisitorAbstract
                 $aliasName = $name->getFirst();
             }
 
-            if (!isset($this->aliases[$type][$aliasName])) {
+            if (isset($this->aliases[$type][$aliasName])) {
-                // unqualified, unaliased names cannot be resolved at compile-time
-                return $name;
-            }
-
                 // resolve unqualified aliases
                 return new FullyQualified($this->aliases[$type][$aliasName], $name->getAttributes());
             }
 
-        if (null !== $this->namespace) {
+            if (null === $this->namespace) {
+                // outside of a namespace unaliased unqualified is same as fully qualified
+                return new FullyQualified($name, $name->getAttributes());
+            }
+
+            // unqualified names inside a namespace cannot be resolved at compile-time
+            // add the namespaced version of the name as an attribute
+            $name->setAttribute('namespacedName',
+                FullyQualified::concat($this->namespace, $name, $name->getAttributes()));
+            return $name;
+        }
+
         // if no alias exists prepend current namespace
         return FullyQualified::concat($this->namespace, $name, $name->getAttributes());
     }
 
-        return new FullyQualified($name->parts, $name->getAttributes());
-    }
-
     protected function addNamespacedName(Node $node) {
-        if (null !== $this->namespace) {
         $node->namespacedName = Name::concat($this->namespace, $node->name);
-        } else {
-            $node->namespacedName = new Name($node->name);
-        }
     }
 }

lib/PhpParser/Parser.php

@@ -7,18 +7,11 @@ interface Parser {
      * Parses PHP code into a node tree.
      *
      * @param string $code The source code to parse
+     * @param ErrorHandler|null $errorHandler Error handler to use for lexer/parser errors, defaults
+     *                                        to ErrorHandler\Throwing.
      *
      * @return Node[]|null Array of statements (or null if the 'throwOnError' option is disabled and the parser was
      *                     unable to recover from an error).
      */
-    public function parse($code);
+    public function parse($code, ErrorHandler $errorHandler = null);
-
-    /**
-     * Get array of errors that occurred during the last parse.
-     *
-     * This method may only return multiple errors if the 'throwOnError' option is disabled.
-     *
-     * @return Error[]
-     */
-    public function getErrors();
 }

lib/PhpParser/Parser/Multiple.php

@@ -3,61 +3,52 @@
 namespace PhpParser\Parser;
 
 use PhpParser\Error;
+use PhpParser\ErrorHandler;
 use PhpParser\Parser;
 
 class Multiple implements Parser {
     /** @var Parser[] List of parsers to try, in order of preference */
     private $parsers;
-    /** @var Error[] Errors collected during last parse */
-    private $errors;
 
     /**
      * Create a parser which will try multiple parsers in an order of preference.
      *
      * Parsers will be invoked in the order they're provided to the constructor. If one of the
-     * parsers runs without errors, it's output is returned. Otherwise the errors (and
+     * parsers runs without throwing, it's output is returned. Otherwise the exception that the
-     * PhpParser\Error exception) of the first parser are used.
+     * first parser generated is thrown.
      *
      * @param Parser[] $parsers
      */
     public function __construct(array $parsers) {
         $this->parsers = $parsers;
-        $this->errors = [];
     }
 
-    public function parse($code) {
+    public function parse($code, ErrorHandler $errorHandler = null) {
-        list($firstStmts, $firstErrors, $firstError) = $this->tryParse($this->parsers[0], $code);
+        if (null === $errorHandler) {
-        if ($firstErrors === []) {
+            $errorHandler = new ErrorHandler\Throwing;
-            $this->errors = [];
+        }
+
+        list($firstStmts, $firstError) = $this->tryParse($this->parsers[0], $errorHandler, $code);
+        if ($firstError === null) {
             return $firstStmts;
         }
 
         for ($i = 1, $c = count($this->parsers); $i < $c; ++$i) {
-            list($stmts, $errors) = $this->tryParse($this->parsers[$i], $code);
+            list($stmts, $error) = $this->tryParse($this->parsers[$i], $errorHandler, $code);
-            if ($errors === []) {
+            if ($error === null) {
-                $this->errors = [];
                 return $stmts;
             }
         }
 
-        $this->errors = $firstErrors;
-        if ($firstError) {
         throw $firstError;
     }
-        return $firstStmts;
-    }
 
-    public function getErrors() {
+    private function tryParse(Parser $parser, ErrorHandler $errorHandler, $code) {
-        return $this->errors;
-    }
-
-    private function tryParse(Parser $parser, $code) {
         $stmts = null;
         $error = null;
         try {
-            $stmts = $parser->parse($code);
+            $stmts = $parser->parse($code, $errorHandler);
         } catch (Error $error) {}
-        $errors = $parser->getErrors();
+        return [$stmts, $error];
-        return [$stmts, $errors, $error];
     }
 }

lib/PhpParser/ParserAbstract.php

@@ -7,6 +7,17 @@ namespace PhpParser;
  * turn is based on work by Masato Bito.
  */
 use PhpParser\Node\Name;
+use PhpParser\Node\Param;
+use PhpParser\Node\Scalar\LNumber;
+use PhpParser\Node\Scalar\String_;
+use PhpParser\Node\Stmt\Class_;
+use PhpParser\Node\Stmt\ClassConst;
+use PhpParser\Node\Stmt\ClassMethod;
+use PhpParser\Node\Stmt\Interface_;
+use PhpParser\Node\Stmt\Namespace_;
+use PhpParser\Node\Stmt\Property;
+use PhpParser\Node\Stmt\TryCatch;
+use PhpParser\Node\Stmt\UseUse;
 
 abstract class ParserAbstract implements Parser
 {
@@ -86,52 +97,54 @@ abstract class ParserAbstract implements Parser
     protected $semStack;
     /** @var array[] Start attribute stack */
     protected $startAttributeStack;
+    /** @var array[] End attribute stack */
+    protected $endAttributeStack;
     /** @var array End attributes of last *shifted* token */
     protected $endAttributes;
     /** @var array Start attributes of last *read* token */
     protected $lookaheadStartAttributes;
 
-    /** @var bool Whether to throw on first error */
+    /** @var ErrorHandler Error handler */
-    protected $throwOnError;
+    protected $errorHandler;
     /** @var Error[] Errors collected during last parse */
     protected $errors;
+    /** @var int Error state, used to avoid error floods */
+    protected $errorState;
 
     /**
      * Creates a parser instance.
      *
      * @param Lexer $lexer A lexer
-     * @param array $options Options array. The boolean 'throwOnError' option determines whether an exception should be
+     * @param array $options Options array. Currently no options are supported.
-     *                       thrown on first error, or if the parser should try to continue parsing the remaining code
-     *                       and build a partial AST.
      */
     public function __construct(Lexer $lexer, array $options = array()) {
         $this->lexer = $lexer;
         $this->errors = array();
-        $this->throwOnError = isset($options['throwOnError']) ? $options['throwOnError'] : true;
-    }
 
-    /**
+        if (isset($options['throwOnError'])) {
-     * Get array of errors that occurred during the last parse.
+            throw new \LogicException(
-     *
+                '"throwOnError" is no longer supported, use "errorHandler" instead');
-     * This method may only return multiple errors if the 'throwOnError' option is disabled.
+        }
-     *
-     * @return Error[]
-     */
-    public function getErrors() {
-        return $this->errors;
     }
 
     /**
      * Parses PHP code into a node tree.
      *
+     * If a non-throwing error handler is used, the parser will continue parsing after an error
+     * occurred and attempt to build a partial AST.
+     *
      * @param string $code The source code to parse
+     * @param ErrorHandler|null $errorHandler Error handler to use for lexer/parser errors, defaults
+     *                                        to ErrorHandler\Throwing.
      *
      * @return Node[]|null Array of statements (or null if the 'throwOnError' option is disabled and the parser was
      *                     unable to recover from an error).
      */
-    public function parse($code) {
+    public function parse($code, ErrorHandler $errorHandler = null) {
-        $this->lexer->startLexing($code);
+        $this->errorHandler = $errorHandler ?: new ErrorHandler\Throwing;
-        $this->errors = array();
+
+        // Initialize the lexer
+        $this->lexer->startLexing($code, $this->errorHandler);
 
         // We start off with no lookahead-token
         $symbol = self::SYMBOL_NONE;
@@ -143,9 +156,9 @@ abstract class ParserAbstract implements Parser
         $endAttributes = '*POISON';
         $this->endAttributes = $endAttributes;
 
-        // In order to figure out the attributes for the starting token, we have to keep
+        // Keep stack of start and end attributes
-        // them in a stack
         $this->startAttributeStack = array();
+        $this->endAttributeStack = array($endAttributes);
 
         // Start off in the initial state and keep a stack of previous states
         $state = 0;
@@ -157,7 +170,7 @@ abstract class ParserAbstract implements Parser
         // Current position in the stack(s)
         $this->stackPos = 0;
 
-        $errorState = 0;
+        $this->errorState = 0;
 
         for (;;) {
             //$this->traceNewState($state, $symbol);
@@ -187,6 +200,7 @@ abstract class ParserAbstract implements Parser
                     // This is necessary to assign some meaningful attributes to /* empty */ productions. They'll get
                     // the attributes of the next token, even though they don't contain it themselves.
                     $this->startAttributeStack[$this->stackPos+1] = $startAttributes;
+                    $this->endAttributeStack[$this->stackPos+1] = $endAttributes;
                     $this->lookaheadStartAttributes = $startAttributes;
 
                     //$this->traceRead($symbol);
@@ -213,11 +227,12 @@ abstract class ParserAbstract implements Parser
                         $stateStack[$this->stackPos] = $state = $action;
                         $this->semStack[$this->stackPos] = $tokenValue;
                         $this->startAttributeStack[$this->stackPos] = $startAttributes;
+                        $this->endAttributeStack[$this->stackPos] = $endAttributes;
                         $this->endAttributes = $endAttributes;
                         $symbol = self::SYMBOL_NONE;
 
-                        if ($errorState) {
+                        if ($this->errorState) {
-                            --$errorState;
+                            --$this->errorState;
                         }
 
                         if ($action < $this->YYNLSTATES) {
@@ -250,16 +265,13 @@ abstract class ParserAbstract implements Parser
                             $e->setStartLine($startAttributes['startLine']);
                         }
 
-                        $this->errors[] = $e;
+                        $this->emitError($e);
-                        if ($this->throwOnError) {
+                        // Can't recover from this type of error
-                            throw $e;
-                        } else {
-                            // Currently can't recover from "special" errors
                         return null;
                     }
-                    }
 
                     /* Goto - shift nonterminal */
+                    $lastEndAttributes = $this->endAttributeStack[$this->stackPos];
                     $this->stackPos -= $this->ruleToLength[$rule];
                     $nonTerminal = $this->ruleToNonTerminal[$rule];
                     $idx = $this->gotoBase[$nonTerminal] + $stateStack[$this->stackPos];
@@ -272,20 +284,17 @@ abstract class ParserAbstract implements Parser
                     ++$this->stackPos;
                     $stateStack[$this->stackPos]     = $state;
                     $this->semStack[$this->stackPos] = $this->semValue;
+                    $this->endAttributeStack[$this->stackPos] = $lastEndAttributes;
                 } else {
                     /* error */
-                    switch ($errorState) {
+                    switch ($this->errorState) {
                         case 0:
                             $msg = $this->getErrorMessage($symbol, $state);
-                            $error = new Error($msg, $startAttributes + $endAttributes);
+                            $this->emitError(new Error($msg, $startAttributes + $endAttributes));
-                            $this->errors[] = $error;
-                            if ($this->throwOnError) {
-                                throw $error;
-                            }
                             // Break missing intentionally
                         case 1:
                         case 2:
-                            $errorState = 3;
+                            $this->errorState = 3;
 
                             // Pop until error-expecting state uncovered
                             while (!(
@@ -304,7 +313,9 @@ abstract class ParserAbstract implements Parser
                             }
 
                             //$this->traceShift($this->errorSymbol);
-                            $stateStack[++$this->stackPos] = $state = $action;
+                            ++$this->stackPos;
+                            $stateStack[$this->stackPos] = $state = $action;
+                            $this->endAttributes = $this->endAttributeStack[$this->stackPos];
                             break;
 
                         case 3:
@@ -331,6 +342,10 @@ abstract class ParserAbstract implements Parser
         throw new \RuntimeException('Reached end of parser loop');
     }
 
+    protected function emitError(Error $error) {
+        $this->errorHandler->handleError($error);
+    }
+
     protected function getErrorMessage($symbol, $state) {
         $expectedString = '';
         if ($expected = $this->getExpectedTokens($state)) {
@@ -353,6 +368,7 @@ abstract class ParserAbstract implements Parser
             ) {
                 if ($this->action[$idx] != $this->unexpectedTokenRule
                     && $this->action[$idx] != $this->defaultAction
+                    && $symbol != $this->errorSymbol
                 ) {
                     if (count($expected) == 4) {
                         /* Too many expected tokens */
@@ -413,6 +429,7 @@ abstract class ParserAbstract implements Parser
      * @return Node[]
      */
     protected function handleNamespaces(array $stmts) {
+        $hasErrored = false;
         $style = $this->getNamespacingStyle($stmts);
         if (null === $style) {
             // not namespaced, nothing to do
@@ -423,8 +440,11 @@ abstract class ParserAbstract implements Parser
             foreach ($stmts as $stmt) {
                 if ($stmt instanceof Node\Stmt\Namespace_) {
                     $afterFirstNamespace = true;
-                } elseif (!$stmt instanceof Node\Stmt\HaltCompiler && $afterFirstNamespace) {
+                } elseif (!$stmt instanceof Node\Stmt\HaltCompiler
-                    throw new Error('No code may exist outside of namespace {}', $stmt->getLine());
+                        && $afterFirstNamespace && !$hasErrored) {
+                    $this->emitError(new Error(
+                        'No code may exist outside of namespace {}', $stmt->getAttributes()));
+                    $hasErrored = true; // Avoid one error for every statement
                 }
             }
             return $stmts;
@@ -434,9 +454,15 @@ abstract class ParserAbstract implements Parser
             $targetStmts =& $resultStmts;
             foreach ($stmts as $stmt) {
                 if ($stmt instanceof Node\Stmt\Namespace_) {
+                    if ($stmt->stmts === null) {
                         $stmt->stmts = array();
                         $targetStmts =& $stmt->stmts;
                         $resultStmts[] = $stmt;
+                    } else {
+                        // This handles the invalid case of mixed style namespaces
+                        $resultStmts[] = $stmt;
+                        $targetStmts =& $resultStmts;
+                    }
                 } elseif ($stmt instanceof Node\Stmt\HaltCompiler) {
                     // __halt_compiler() is not moved into the namespace
                     $resultStmts[] = $stmt;
@@ -457,10 +483,18 @@ abstract class ParserAbstract implements Parser
                 if (null === $style) {
                     $style = $currentStyle;
                     if ($hasNotAllowedStmts) {
-                        throw new Error('Namespace declaration statement has to be the very first statement in the script', $stmt->getLine());
+                        $this->emitError(new Error(
+                            'Namespace declaration statement has to be the very first statement in the script',
+                            $stmt->getLine() // Avoid marking the entire namespace as an error
+                        ));
                     }
                 } elseif ($style !== $currentStyle) {
-                    throw new Error('Cannot mix bracketed namespace declarations with unbracketed namespace declarations', $stmt->getLine());
+                    $this->emitError(new Error(
+                        'Cannot mix bracketed namespace declarations with unbracketed namespace declarations',
+                        $stmt->getLine() // Avoid marking the entire namespace as an error
+                    ));
+                    // Treat like semicolon style for namespace normalization
+                    return 'semicolon';
                 }
                 continue;
             }
@@ -500,4 +534,190 @@ abstract class ParserAbstract implements Parser
         $lowerName = strtolower($name->toString());
         return isset($scalarTypes[$lowerName]) ? $lowerName : $name;
     }
+
+    protected static $specialNames = array(
+        'self'   => true,
+        'parent' => true,
+        'static' => true,
+    );
+
+    protected function getAttributesAt($pos) {
+        return $this->startAttributeStack[$pos] + $this->endAttributeStack[$pos];
+    }
+
+    protected function parseLNumber($str, $attributes, $allowInvalidOctal = false) {
+        try {
+            return LNumber::fromString($str, $attributes, $allowInvalidOctal);
+        } catch (Error $error) {
+            $this->emitError($error);
+            // Use dummy value
+            return new LNumber(0, $attributes);
+        }
+    }
+
+    protected function parseNumString($str, $attributes) {
+        if (!preg_match('/^(?:0|-?[1-9][0-9]*)$/', $str)) {
+            return new String_($str, $attributes);
+        }
+
+        $num = +$str;
+        if (!is_int($num)) {
+            return new String_($str, $attributes);
+        }
+
+        return new LNumber($num, $attributes);
+    }
+
+    protected function checkModifier($a, $b, $modifierPos) {
+        // Jumping through some hoops here because verifyModifier() is also used elsewhere
+        try {
+            Class_::verifyModifier($a, $b);
+        } catch (Error $error) {
+            $error->setAttributes($this->getAttributesAt($modifierPos));
+            $this->emitError($error);
+        }
+    }
+
+    protected function checkParam(Param $node) {
+        if ($node->variadic && null !== $node->default) {
+            $this->emitError(new Error(
+                'Variadic parameter cannot have a default value',
+                $node->default->getAttributes()
+            ));
+        }
+    }
+
+    protected function checkTryCatch(TryCatch $node) {
+        if (empty($node->catches) && null === $node->finally) {
+            $this->emitError(new Error(
+                'Cannot use try without catch or finally', $node->getAttributes()
+            ));
+        }
+    }
+
+    protected function checkNamespace(Namespace_ $node) {
+        if (isset(self::$specialNames[strtolower($node->name)])) {
+            $this->emitError(new Error(
+                sprintf('Cannot use \'%s\' as namespace name', $node->name),
+                $node->name->getAttributes()
+            ));
+        }
+
+        if (null !== $node->stmts) {
+            foreach ($node->stmts as $stmt) {
+                if ($stmt instanceof Namespace_) {
+                    $this->emitError(new Error(
+                        'Namespace declarations cannot be nested', $stmt->getAttributes()
+                    ));
+                }
+            }
+        }
+    }
+
+    protected function checkClass(Class_ $node, $namePos) {
+        if (null !== $node->name && isset(self::$specialNames[strtolower($node->name)])) {
+            $this->emitError(new Error(
+                sprintf('Cannot use \'%s\' as class name as it is reserved', $node->name),
+                $this->getAttributesAt($namePos)
+            ));
+        }
+
+        if (isset(self::$specialNames[strtolower($node->extends)])) {
+            $this->emitError(new Error(
+                sprintf('Cannot use \'%s\' as class name as it is reserved', $node->extends),
+                $node->extends->getAttributes()
+            ));
+        }
+
+        foreach ($node->implements as $interface) {
+            if (isset(self::$specialNames[strtolower($interface)])) {
+                $this->emitError(new Error(
+                    sprintf('Cannot use \'%s\' as interface name as it is reserved', $interface),
+                    $interface->getAttributes()
+                ));
+            }
+        }
+    }
+
+    protected function checkInterface(Interface_ $node, $namePos) {
+        if (null !== $node->name && isset(self::$specialNames[strtolower($node->name)])) {
+            $this->emitError(new Error(
+                sprintf('Cannot use \'%s\' as class name as it is reserved', $node->name),
+                $this->getAttributesAt($namePos)
+            ));
+        }
+
+        foreach ($node->extends as $interface) {
+            if (isset(self::$specialNames[strtolower($interface)])) {
+                $this->emitError(new Error(
+                    sprintf('Cannot use \'%s\' as interface name as it is reserved', $interface),
+                    $interface->getAttributes()
+                ));
+            }
+        }
+    }
+
+    protected function checkClassMethod(ClassMethod $node, $modifierPos) {
+        if ($node->flags & Class_::MODIFIER_STATIC) {
+            switch (strtolower($node->name)) {
+                case '__construct':
+                    $this->emitError(new Error(
+                        sprintf('Constructor %s() cannot be static', $node->name),
+                        $this->getAttributesAt($modifierPos)));
+                    break;
+                case '__destruct':
+                    $this->emitError(new Error(
+                        sprintf('Destructor %s() cannot be static', $node->name),
+                        $this->getAttributesAt($modifierPos)));
+                    break;
+                case '__clone':
+                    $this->emitError(new Error(
+                        sprintf('Clone method %s() cannot be static', $node->name),
+                        $this->getAttributesAt($modifierPos)));
+                    break;
+            }
+        }
+    }
+
+    protected function checkClassConst(ClassConst $node, $modifierPos) {
+        if ($node->flags & Class_::MODIFIER_STATIC) {
+            $this->emitError(new Error(
+                "Cannot use 'static' as constant modifier",
+                $this->getAttributesAt($modifierPos)));
+        }
+        if ($node->flags & Class_::MODIFIER_ABSTRACT) {
+            $this->emitError(new Error(
+                "Cannot use 'abstract' as constant modifier",
+                $this->getAttributesAt($modifierPos)));
+        }
+        if ($node->flags & Class_::MODIFIER_FINAL) {
+            $this->emitError(new Error(
+                "Cannot use 'final' as constant modifier",
+                $this->getAttributesAt($modifierPos)));
+        }
+    }
+
+    protected function checkProperty(Property $node, $modifierPos) {
+        if ($node->flags & Class_::MODIFIER_ABSTRACT) {
+            $this->emitError(new Error('Properties cannot be declared abstract',
+                $this->getAttributesAt($modifierPos)));
+        }
+
+        if ($node->flags & Class_::MODIFIER_FINAL) {
+            $this->emitError(new Error('Properties cannot be declared final',
+                $this->getAttributesAt($modifierPos)));
+        }
+    }
+
+    protected function checkUseUse(UseUse $node, $namePos) {
+        if ('self' == strtolower($node->alias) || 'parent' == strtolower($node->alias)) {
+            $this->emitError(new Error(
+                sprintf(
+                    'Cannot use %s as %s because \'%2$s\' is a special class name',
+                    $node->name, $node->alias
+                ),
+                $this->getAttributesAt($namePos)
+            ));
+        }
+    }
 }

lib/PhpParser/PrettyPrinter/Standard.php

@@ -2,16 +2,16 @@
 
 namespace PhpParser\PrettyPrinter;
 
-use PhpParser\PrettyPrinterAbstract;
 use PhpParser\Node;
-use PhpParser\Node\Scalar;
-use PhpParser\Node\Scalar\MagicConst;
 use PhpParser\Node\Expr;
 use PhpParser\Node\Expr\AssignOp;
 use PhpParser\Node\Expr\BinaryOp;
 use PhpParser\Node\Expr\Cast;
-use PhpParser\Node\Stmt;
 use PhpParser\Node\Name;
+use PhpParser\Node\Scalar;
+use PhpParser\Node\Scalar\MagicConst;
+use PhpParser\Node\Stmt;
+use PhpParser\PrettyPrinterAbstract;
 
 class Standard extends PrettyPrinterAbstract
 {
@@ -34,7 +34,7 @@ class Standard extends PrettyPrinterAbstract
     }
 
     protected function pNullableType(Node\NullableType $node) {
-        return '?' . $node->type;
+        return '?' . $this->pType($node->type);
     }
 
     // Names
@@ -141,21 +141,42 @@ class Standard extends PrettyPrinterAbstract
     }
 
     protected function pScalar_LNumber(Scalar\LNumber $node) {
+        if ($node->value === -\PHP_INT_MAX-1) {
+            // PHP_INT_MIN cannot be represented as a literal,
+            // because the sign is not part of the literal
+            return '(-' . \PHP_INT_MAX . '-1)';
+        }
+
+        $kind = $node->getAttribute('kind', Scalar\LNumber::KIND_DEC);
+        if (Scalar\LNumber::KIND_DEC === $kind) {
+            return (string) $node->value;
+        }
+
+        $sign = $node->value < 0 ? '-' : '';
         $str = (string) $node->value;
-        switch ($node->getAttribute('kind', Scalar\LNumber::KIND_DEC)) {
+        switch ($kind) {
             case Scalar\LNumber::KIND_BIN:
-                return '0b' . base_convert($str, 10, 2);
+                return $sign . '0b' . base_convert($str, 10, 2);
             case Scalar\LNumber::KIND_OCT:
-                return '0' . base_convert($str, 10, 8);
+                return $sign . '0' . base_convert($str, 10, 8);
-            case Scalar\LNumber::KIND_DEC:
-                return $str;
             case Scalar\LNumber::KIND_HEX:
-                return '0x' . base_convert($str, 10, 16);
+                return $sign . '0x' . base_convert($str, 10, 16);
         }
         throw new \Exception('Invalid number kind');
     }
 
     protected function pScalar_DNumber(Scalar\DNumber $node) {
+        if (!is_finite($node->value)) {
+            if ($node->value === \INF) {
+                return '\INF';
+            } elseif ($node->value === -\INF) {
+                return '-\INF';
+            } else {
+                return '\NAN';
+            }
+        }
+
+        // Try to find a short full-precision representation
         $stringValue = sprintf('%.16G', $node->value);
         if ($node->value !== (double) $stringValue) {
             $stringValue = sprintf('%.17G', $node->value);
@@ -464,6 +485,10 @@ class Standard extends PrettyPrinterAbstract
 
     // Other
 
+    protected function pExpr_Error(Expr\Error $node) {
+        throw new \LogicException('Cannot pretty-print AST with Error nodes');
+    }
+
     protected function pExpr_Variable(Expr\Variable $node) {
         if ($node->name instanceof Expr) {
             return '${' . $this->p($node->name) . '}';
@@ -497,7 +522,8 @@ class Standard extends PrettyPrinterAbstract
     }
 
     protected function pExpr_ClassConstFetch(Expr\ClassConstFetch $node) {
-        return $this->p($node->class) . '::' . $node->name;
+        return $this->p($node->class) . '::'
+             . (is_string($node->name) ? $node->name : $this->p($node->name));
     }
 
     protected function pExpr_PropertyFetch(Expr\PropertyFetch $node) {
@@ -789,7 +815,8 @@ class Standard extends PrettyPrinterAbstract
     }
 
     protected function pStmt_InlineHTML(Stmt\InlineHTML $node) {
-        return '?>' . $this->pNoIndent("\n" . $node->value) . '<?php ';
+        $newline = $node->getAttribute('hasLeadingNewline', true) ? "\n" : '';
+        return '?>' . $this->pNoIndent($newline . $node->value) . '<?php ';
     }
 
     protected function pStmt_HaltCompiler(Stmt\HaltCompiler $node) {

lib/PhpParser/Serializer.php

@@ -2,6 +2,9 @@
 
 namespace PhpParser;
 
+/**
+ * @deprecated
+ */
 interface Serializer
 {
     /**

lib/PhpParser/Serializer/XML.php

@@ -2,11 +2,14 @@
 
 namespace PhpParser\Serializer;
 
-use XMLWriter;
-use PhpParser\Node;
 use PhpParser\Comment;
+use PhpParser\Node;
 use PhpParser\Serializer;
+use XMLWriter;
 
+/**
+ * @deprecated
+ */
 class XML implements Serializer
 {
     protected $writer;

lib/PhpParser/Unserializer.php

@@ -2,6 +2,9 @@
 
 namespace PhpParser;
 
+/**
+ * @deprecated
+ */
 interface Unserializer
 {
     /**

lib/PhpParser/Unserializer/XML.php

@@ -2,10 +2,13 @@
 
 namespace PhpParser\Unserializer;
 
-use XMLReader;
 use DomainException;
 use PhpParser\Unserializer;
+use XMLReader;
 
+/**
+ * @deprecated
+ */
 class XML implements Unserializer
 {
     protected $reader;

test/PhpParser/AutoloaderTest.php

@@ -2,9 +2,6 @@
 
 namespace PhpParser;
 
-use PhpParser\Node\Expr;
-use PhpParser\Node\Scalar;
-
 /* The autoloader is already active at this point, so we only check effects here. */
 
 class AutoloaderTest extends \PHPUnit_Framework_TestCase {

test/PhpParser/Builder/ClassTest.php

@@ -43,7 +43,7 @@ class ClassTest extends \PHPUnit_Framework_TestCase
 
         $this->assertEquals(
             new Stmt\Class_('Test', array(
-                'type' => Stmt\Class_::MODIFIER_ABSTRACT
+                'flags' => Stmt\Class_::MODIFIER_ABSTRACT
             )),
             $node
         );
@@ -57,7 +57,7 @@ class ClassTest extends \PHPUnit_Framework_TestCase
 
         $this->assertEquals(
             new Stmt\Class_('Test', array(
-                'type' => Stmt\Class_::MODIFIER_FINAL
+                'flags' => Stmt\Class_::MODIFIER_FINAL
             )),
             $node
         );

test/PhpParser/Builder/FunctionTest.php

@@ -4,9 +4,9 @@ namespace PhpParser\Builder;
 
 use PhpParser\Comment;
 use PhpParser\Node;
-use PhpParser\Node\Stmt;
 use PhpParser\Node\Expr\Print_;
 use PhpParser\Node\Scalar\String_;
+use PhpParser\Node\Stmt;
 
 class FunctionTest extends \PHPUnit_Framework_TestCase
 {
@@ -78,14 +78,22 @@ class FunctionTest extends \PHPUnit_Framework_TestCase
 
     public function testReturnType() {
         $node = $this->createFunctionBuilder('test')
-            ->setReturnType('bool')
+            ->setReturnType('void')
             ->getNode();
 
         $this->assertEquals(new Stmt\Function_('test', array(
-            'returnType' => 'bool'
+            'returnType' => 'void'
         ), array()), $node);
     }
 
+    /**
+     * @expectedException \LogicException
+     * @expectedExceptionMessage void type cannot be nullable
+     */
+    public function testInvalidNullableVoidType() {
+        $this->createFunctionBuilder('test')->setReturnType('?void');
+    }
+
     /**
      * @expectedException \LogicException
      * @expectedExceptionMessage Expected parameter node, got "Name"

test/PhpParser/Builder/InterfaceTest.php

@@ -2,10 +2,10 @@
 
 namespace PhpParser\Builder;
 
-use PhpParser\Node;
-use PhpParser\Node\Stmt;
-use PhpParser\Node\Scalar\DNumber;
 use PhpParser\Comment;
+use PhpParser\Node;
+use PhpParser\Node\Scalar\DNumber;
+use PhpParser\Node\Stmt;
 
 class InterfaceTest extends \PHPUnit_Framework_TestCase
 {

test/PhpParser/Builder/MethodTest.php

@@ -2,11 +2,11 @@
 
 namespace PhpParser\Builder;
 
+use PhpParser\Comment;
 use PhpParser\Node;
 use PhpParser\Node\Expr\Print_;
 use PhpParser\Node\Scalar\String_;
 use PhpParser\Node\Stmt;
-use PhpParser\Comment;
 
 class MethodTest extends \PHPUnit_Framework_TestCase
 {
@@ -24,7 +24,7 @@ class MethodTest extends \PHPUnit_Framework_TestCase
 
         $this->assertEquals(
             new Stmt\ClassMethod('test', array(
-                'type' => Stmt\Class_::MODIFIER_PUBLIC
+                'flags' => Stmt\Class_::MODIFIER_PUBLIC
                          | Stmt\Class_::MODIFIER_ABSTRACT
                          | Stmt\Class_::MODIFIER_STATIC,
                 'stmts' => null,
@@ -40,7 +40,7 @@ class MethodTest extends \PHPUnit_Framework_TestCase
 
         $this->assertEquals(
             new Stmt\ClassMethod('test', array(
-                'type' => Stmt\Class_::MODIFIER_PROTECTED
+                'flags' => Stmt\Class_::MODIFIER_PROTECTED
                          | Stmt\Class_::MODIFIER_FINAL
             )),
             $node

test/PhpParser/Builder/ParamTest.php

@@ -78,36 +78,70 @@ class ParamTest extends \PHPUnit_Framework_TestCase
         );
     }
 
-    public function testTypeHints() {
+    /**
+     * @dataProvider provideTestTypeHints
+     */
+    public function testTypeHints($typeHint, $expectedType) {
         $node = $this->createParamBuilder('test')
-            ->setTypeHint('array')
+            ->setTypeHint($typeHint)
             ->getNode()
         ;
+        $type = $node->type;
 
-        $this->assertEquals(
+        /* Manually implement comparison to avoid __toString stupidity */
-            new Node\Param('test', null, 'array'),
+        if ($expectedType instanceof Node\NullableType) {
-            $node
+            $this->assertInstanceOf(get_class($expectedType), $type);
+            $expectedType = $expectedType->type;
+            $type = $type->type;
+        }
+
+        if ($expectedType instanceof Node\Name) {
+            $this->assertInstanceOf(get_class($expectedType), $type);
+            $this->assertEquals($expectedType, $type);
+        } else {
+            $this->assertSame($expectedType, $type);
+        }
+    }
+
+    public function provideTestTypeHints() {
+        return array(
+            array('array', 'array'),
+            array('callable', 'callable'),
+            array('bool', 'bool'),
+            array('int', 'int'),
+            array('float', 'float'),
+            array('string', 'string'),
+            array('iterable', 'iterable'),
+            array('Array', 'array'),
+            array('CALLABLE', 'callable'),
+            array('Some\Class', new Node\Name('Some\Class')),
+            array('\Foo', new Node\Name\FullyQualified('Foo')),
+            array('self', new Node\Name('self')),
+            array('?array', new Node\NullableType('array')),
+            array('?Some\Class', new Node\NullableType(new Node\Name('Some\Class'))),
+            array(new Node\Name('Some\Class'), new Node\Name('Some\Class')),
+            array(new Node\NullableType('int'), new Node\NullableType('int')),
+            array(
+                new Node\NullableType(new Node\Name('Some\Class')),
+                new Node\NullableType(new Node\Name('Some\Class'))
+            ),
         );
+    }
 
-        $node = $this->createParamBuilder('test')
+    /**
-            ->setTypeHint('callable')
+     * @expectedException \LogicException
-            ->getNode()
+     * @expectedExceptionMessage Parameter type cannot be void
-        ;
+     */
+    public function testVoidTypeError() {
+        $this->createParamBuilder('test')->setTypeHint('void');
+    }
 
-        $this->assertEquals(
+    /**
-            new Node\Param('test', null, 'callable'),
+     * @expectedException \LogicException
-            $node
+     * @expectedExceptionMessage Type must be a string, or an instance of Name or NullableType
-        );
+     */
-
+    public function testInvalidTypeError() {
-        $node = $this->createParamBuilder('test')
+        $this->createParamBuilder('test')->setTypeHint(new \stdClass);
-            ->setTypeHint('Some\Class')
-            ->getNode()
-        ;
-
-        $this->assertEquals(
-            new Node\Param('test', null, new Node\Name('Some\Class')),
-            $node
-        );
     }
 
     public function testByRef() {

test/PhpParser/Builder/PropertyTest.php

@@ -2,11 +2,11 @@
 
 namespace PhpParser\Builder;
 
-use PhpParser\Node\Name;
-use PhpParser\Node\Stmt;
-use PhpParser\Node\Expr;
-use PhpParser\Node\Scalar;
 use PhpParser\Comment;
+use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
+use PhpParser\Node\Scalar;
+use PhpParser\Node\Stmt;
 
 class PropertyTest extends \PHPUnit_Framework_TestCase
 {

test/PhpParser/Builder/TraitTest.php

@@ -3,8 +3,6 @@
 namespace PhpParser\Builder;
 
 use PhpParser\Comment;
-use PhpParser\Node;
-use PhpParser\Node\Name;
 use PhpParser\Node\Stmt;
 
 class TraitTest extends \PHPUnit_Framework_TestCase

test/PhpParser/CodeParsingTest.php

@@ -2,8 +2,6 @@
 
 namespace PhpParser;
 
-use PhpParser\Comment;
-
 require_once __DIR__ . '/CodeTestAbstract.php';
 
 class CodeParsingTest extends CodeTestAbstract
@@ -11,24 +9,28 @@ class CodeParsingTest extends CodeTestAbstract
     /**
      * @dataProvider provideTestParse
      */
-    public function testParse($name, $code, $expected, $mode) {
+    public function testParse($name, $code, $expected, $modeLine) {
+        if (null !== $modeLine) {
+            $modes = array_fill_keys(explode(',', $modeLine), true);
+        } else {
+            $modes = [];
+        }
+
+
         $lexer = new Lexer\Emulative(array('usedAttributes' => array(
             'startLine', 'endLine', 'startFilePos', 'endFilePos', 'comments'
         )));
-        $parser5 = new Parser\Php5($lexer, array(
+        $parser5 = new Parser\Php5($lexer);
-            'throwOnError' => false,
+        $parser7 = new Parser\Php7($lexer);
-        ));
-        $parser7 = new Parser\Php7($lexer, array(
-            'throwOnError' => false,
-        ));
 
-        $output5 = $this->getParseOutput($parser5, $code);
+        $dumpPositions = isset($modes['positions']);
-        $output7 = $this->getParseOutput($parser7, $code);
+        $output5 = $this->getParseOutput($parser5, $code, $dumpPositions);
+        $output7 = $this->getParseOutput($parser7, $code, $dumpPositions);
 
-        if ($mode === 'php5') {
+        if (isset($modes['php5'])) {
             $this->assertSame($expected, $output5, $name);
             $this->assertNotSame($expected, $output7, $name);
-        } else if ($mode === 'php7') {
+        } else if (isset($modes['php7'])) {
             $this->assertNotSame($expected, $output5, $name);
             $this->assertSame($expected, $output7, $name);
         } else {
@@ -37,18 +39,18 @@ class CodeParsingTest extends CodeTestAbstract
         }
     }
 
-    private function getParseOutput(Parser $parser, $code) {
+    private function getParseOutput(Parser $parser, $code, $dumpPositions) {
-        $stmts = $parser->parse($code);
+        $errors = new ErrorHandler\Collecting;
-        $errors = $parser->getErrors();
+        $stmts = $parser->parse($code, $errors);
 
         $output = '';
-        foreach ($errors as $error) {
+        foreach ($errors->getErrors() as $error) {
             $output .= $this->formatErrorMessage($error, $code) . "\n";
         }
 
         if (null !== $stmts) {
-            $dumper = new NodeDumper(['dumpComments' => true]);
+            $dumper = new NodeDumper(['dumpComments' => true, 'dumpPositions' => $dumpPositions]);
-            $output .= $dumper->dump($stmts);
+            $output .= $dumper->dump($stmts, $code);
         }
 
         return canonicalize($output);
@@ -60,8 +62,7 @@ class CodeParsingTest extends CodeTestAbstract
 
     private function formatErrorMessage(Error $e, $code) {
         if ($e->hasColumnInfo()) {
-            return $e->getRawMessage() . ' from ' . $e->getStartLine() . ':' . $e->getStartColumn($code)
+            return $e->getMessageWithColumnInfo($code);
-                . ' to ' . $e->getEndLine() . ':' . $e->getEndColumn($code);
         } else {
             return $e->getMessage();
         }

test/PhpParser/ErrorHandler/CollectingTest.php

@@ -0,0 +1,22 @@
+<?php
+
+namespace PhpParser\ErrorHandler;
+
+use PhpParser\Error;
+
+class CollectingTest extends \PHPUnit_Framework_TestCase {
+    public function testHandleError() {
+        $errorHandler = new Collecting();
+        $this->assertFalse($errorHandler->hasErrors());
+        $this->assertEmpty($errorHandler->getErrors());
+
+        $errorHandler->handleError($e1 = new Error('Test 1'));
+        $errorHandler->handleError($e2 = new Error('Test 2'));
+        $this->assertTrue($errorHandler->hasErrors());
+        $this->assertSame([$e1, $e2], $errorHandler->getErrors());
+
+        $errorHandler->clearErrors();
+        $this->assertFalse($errorHandler->hasErrors());
+        $this->assertEmpty($errorHandler->getErrors());
+    }
+}

test/PhpParser/ErrorHandler/ThrowingTest.php

@@ -0,0 +1,16 @@
+<?php
+
+namespace PhpParser\ErrorHandler;
+
+use PhpParser\Error;
+
+class ThrowingTest extends \PHPUnit_Framework_TestCase {
+    /**
+     * @expectedException \PhpParser\Error
+     * @expectedExceptionMessage Test
+     */
+    public function testHandleError() {
+        $errorHandler = new Throwing();
+        $errorHandler->handleError(new Error('Test'));
+    }
+}

test/PhpParser/ErrorTest.php

@@ -15,7 +15,6 @@ class ErrorTest extends \PHPUnit_Framework_TestCase
         $this->assertSame($attributes, $error->getAttributes());
         $this->assertSame(10, $error->getStartLine());
         $this->assertSame(11, $error->getEndLine());
-        $this->assertSame(10, $error->getRawLine());
         $this->assertSame('Some error on line 10', $error->getMessage());
 
         return $error;
@@ -31,10 +30,6 @@ class ErrorTest extends \PHPUnit_Framework_TestCase
         $error->setStartLine(15);
         $this->assertSame(15, $error->getStartLine());
         $this->assertSame('Some other error on line 15', $error->getMessage());
-
-        $error->setRawLine(17);
-        $this->assertSame(17, $error->getRawLine());
-        $this->assertSame('Some other error on line 17', $error->getMessage());
     }
 
     public function testUnknownLine() {
@@ -42,7 +37,6 @@ class ErrorTest extends \PHPUnit_Framework_TestCase
 
         $this->assertSame(-1, $error->getStartLine());
         $this->assertSame(-1, $error->getEndLine());
-        $this->assertSame(-1, $error->getRawLine());
         $this->assertSame('Some error on unknown line', $error->getMessage());
     }
 

test/PhpParser/LexerTest.php

@@ -14,28 +14,36 @@ class LexerTest extends \PHPUnit_Framework_TestCase
     /**
      * @dataProvider provideTestError
      */
-    public function testError($code, $message) {
+    public function testError($code, $messages) {
         if (defined('HHVM_VERSION')) {
             $this->markTestSkipped('HHVM does not throw warnings from token_get_all()');
         }
 
-        $lexer = $this->getLexer();
+        $errorHandler = new ErrorHandler\Collecting();
-        try {
+        $lexer = $this->getLexer(['usedAttributes' => [
-            $lexer->startLexing($code);
+            'comments', 'startLine', 'endLine', 'startFilePos', 'endFilePos'
-        } catch (Error $e) {
+        ]]);
-            $this->assertSame($message, $e->getMessage());
+        $lexer->startLexing($code, $errorHandler);
+        $errors = $errorHandler->getErrors();
 
-            return;
+        $this->assertSame(count($messages), count($errors));
+        for ($i = 0; $i < count($messages); $i++) {
+            $this->assertSame($messages[$i], $errors[$i]->getMessageWithColumnInfo($code));
         }
-
-        $this->fail('Expected PhpParser\Error');
     }
 
     public function provideTestError() {
         return array(
-            array('<?php /*', 'Unterminated comment on line 1'),
+            array("<?php /*", array("Unterminated comment from 1:7 to 1:9")),
-            array('<?php ' . "\1", 'Unexpected character "' . "\1" . '" (ASCII 1) on unknown line'),
+            array("<?php \1", array("Unexpected character \"\1\" (ASCII 1) from 1:7 to 1:7")),
-            array('<?php ' . "\0", 'Unexpected null byte on unknown line'),
+            array("<?php \0", array("Unexpected null byte from 1:7 to 1:7")),
+            // Error with potentially emulated token
+            array("<?php ?? \0", array("Unexpected null byte from 1:10 to 1:10")),
+            array("<?php\n\0\1 foo /* bar", array(
+                "Unexpected null byte from 2:1 to 2:1",
+                "Unexpected character \"\1\" (ASCII 1) from 2:2 to 2:2",
+                "Unterminated comment from 2:8 to 2:14"
+            )),
         );
     }
 
@@ -72,7 +80,8 @@ class LexerTest extends \PHPUnit_Framework_TestCase
                     ),
                     array(
                         Tokens::T_INLINE_HTML, 'plaintext',
-                        array('startLine' => 1), array('endLine' => 1)
+                        array('startLine' => 1, 'hasLeadingNewline' => false),
+                        array('endLine' => 1)
                     ),
                 )
             ),

test/PhpParser/Node/NameTest.php

@@ -10,6 +10,9 @@ class NameTest extends \PHPUnit_Framework_TestCase
 
         $name = new Name('foo\bar');
         $this->assertSame(array('foo', 'bar'), $name->parts);
+
+        $name = new Name($name);
+        $this->assertSame(array('foo', 'bar'), $name->parts);
     }
 
     public function testGet() {
@@ -27,43 +30,22 @@ class NameTest extends \PHPUnit_Framework_TestCase
 
         $this->assertSame('foo\bar', (string) $name);
         $this->assertSame('foo\bar', $name->toString());
-        $this->assertSame('foo_bar', $name->toString('_'));
-    }
-
-    public function testAppend() {
-        $name = new Name('foo');
-
-        $name->append('bar');
-        $this->assertSame('foo\bar', $name->toString());
-
-        $name->append('bar\foo');
-        $this->assertSame('foo\bar\bar\foo', $name->toString());
-    }
-
-    public function testPrepend() {
-        $name = new Name('foo');
-
-        $name->prepend('bar');
-        $this->assertSame('bar\foo', $name->toString());
-
-        $name->prepend('foo\bar');
-        $this->assertSame('foo\bar\bar\foo', $name->toString());
     }
 
     public function testSlice() {
         $name = new Name('foo\bar\baz');
         $this->assertEquals(new Name('foo\bar\baz'), $name->slice(0));
         $this->assertEquals(new Name('bar\baz'), $name->slice(1));
-        $this->assertEquals(new Name([]), $name->slice(3));
+        $this->assertNull($name->slice(3));
         $this->assertEquals(new Name('foo\bar\baz'), $name->slice(-3));
         $this->assertEquals(new Name('bar\baz'), $name->slice(-2));
         $this->assertEquals(new Name('foo\bar'), $name->slice(0, -1));
-        $this->assertEquals(new Name([]), $name->slice(0, -3));
+        $this->assertNull($name->slice(0, -3));
         $this->assertEquals(new Name('bar'), $name->slice(1, -1));
-        $this->assertEquals(new Name([]), $name->slice(1, -2));
+        $this->assertNull($name->slice(1, -2));
         $this->assertEquals(new Name('bar'), $name->slice(-2, 1));
         $this->assertEquals(new Name('bar'), $name->slice(-2, -1));
-        $this->assertEquals(new Name([]), $name->slice(-2, -2));
+        $this->assertNull($name->slice(-2, -2));
     }
 
     /**
@@ -111,8 +93,9 @@ class NameTest extends \PHPUnit_Framework_TestCase
             Name\Relative::concat(new Name\FullyQualified('foo\bar'), 'baz', $attributes)
         );
 
-        $this->assertEquals(new Name('foo'), Name::concat([], 'foo'));
+        $this->assertEquals(new Name('foo'), Name::concat(null, 'foo'));
-        $this->assertEquals(new Name([]), Name::concat([], []));
+        $this->assertEquals(new Name('foo'), Name::concat('foo', null));
+        $this->assertNull(Name::concat(null, null));
     }
 
     public function testIs() {
@@ -143,10 +126,9 @@ class NameTest extends \PHPUnit_Framework_TestCase
 
     /**
      * @expectedException        \InvalidArgumentException
-     * @expectedExceptionMessage When changing a name you need to pass either a string, an array or a Name node
+     * @expectedExceptionMessage Expected string, array of parts or Name instance
      */
     public function testInvalidArg() {
-        $name = new Name('foo');
+        Name::concat('foo', new \stdClass);
-        $name->append(new \stdClass);
     }
 }

test/PhpParser/NodeAbstractTest.php

@@ -70,6 +70,28 @@ class NodeAbstractTest extends \PHPUnit_Framework_TestCase
         $this->assertNull($node->getDocComment());
     }
 
+    public function testSetDocComment() {
+        $node = new DummyNode(null, null, []);
+
+        // Add doc comment to node without comments
+        $docComment = new Comment\Doc('/** doc */');
+        $node->setDocComment($docComment);
+        $this->assertSame($docComment, $node->getDocComment());
+
+        // Replace it
+        $docComment = new Comment\Doc('/** doc 2 */');
+        $node->setDocComment($docComment);
+        $this->assertSame($docComment, $node->getDocComment());
+
+        // Add docmment to node with other comments
+        $c1 = new Comment('/* foo */');
+        $c2 = new Comment('/* bar */');
+        $docComment = new Comment\Doc('/** baz */');
+        $node->setAttribute('comments', [$c1, $c2]);
+        $node->setDocComment($docComment);
+        $this->assertSame([$c1, $c2, $docComment], $node->getAttribute('comments'));
+    }
+
     /**
      * @dataProvider provideNodes
      */
@@ -144,4 +166,109 @@ class NodeAbstractTest extends \PHPUnit_Framework_TestCase
             $node->getAttributes()
         );
     }
+
+    public function testJsonSerialization() {
+        $code = <<<'PHP'
+<?php
+// comment
+/** doc comment */
+function functionName(&$a = 0, $b = 1.0) {
+    echo 'Foo';
+}
+PHP;
+        $expected = <<<'JSON'
+[
+    {
+        "nodeType": "Stmt_Function",
+        "byRef": false,
+        "name": "functionName",
+        "params": [
+            {
+                "nodeType": "Param",
+                "type": null,
+                "byRef": true,
+                "variadic": false,
+                "name": "a",
+                "default": {
+                    "nodeType": "Scalar_LNumber",
+                    "value": 0,
+                    "attributes": {
+                        "startLine": 4,
+                        "endLine": 4,
+                        "kind": 10
+                    }
+                },
+                "attributes": {
+                    "startLine": 4,
+                    "endLine": 4
+                }
+            },
+            {
+                "nodeType": "Param",
+                "type": null,
+                "byRef": false,
+                "variadic": false,
+                "name": "b",
+                "default": {
+                    "nodeType": "Scalar_DNumber",
+                    "value": 1,
+                    "attributes": {
+                        "startLine": 4,
+                        "endLine": 4
+                    }
+                },
+                "attributes": {
+                    "startLine": 4,
+                    "endLine": 4
+                }
+            }
+        ],
+        "returnType": null,
+        "stmts": [
+            {
+                "nodeType": "Stmt_Echo",
+                "exprs": [
+                    {
+                        "nodeType": "Scalar_String",
+                        "value": "Foo",
+                        "attributes": {
+                            "startLine": 5,
+                            "endLine": 5,
+                            "kind": 1
+                        }
+                    }
+                ],
+                "attributes": {
+                    "startLine": 5,
+                    "endLine": 5
+                }
+            }
+        ],
+        "attributes": {
+            "startLine": 4,
+            "comments": [
+                {
+                    "nodeType": "Comment",
+                    "text": "\/\/ comment\n",
+                    "line": 2,
+                    "filePos": 6
+                },
+                {
+                    "nodeType": "Comment_Doc",
+                    "text": "\/** doc comment *\/",
+                    "line": 3,
+                    "filePos": 17
+                }
+            ],
+            "endLine": 6
+        }
+    }
+]
+JSON;
+
+        $parser = new Parser\Php7(new Lexer());
+        $stmts = $parser->parse(canonicalize($code));
+        $json = json_encode($stmts, JSON_PRETTY_PRINT);
+        $this->assertEquals(canonicalize($expected), canonicalize($json));
+    }
 }

test/PhpParser/NodeDumperTest.php

@@ -10,7 +10,6 @@ class NodeDumperTest extends \PHPUnit_Framework_TestCase
 
     /**
      * @dataProvider provideTestDump
-     * @covers PhpParser\NodeDumper::dump
      */
     public function testDump($node, $dump) {
         $dumper = new NodeDumper;
@@ -61,6 +60,40 @@ class NodeDumperTest extends \PHPUnit_Framework_TestCase
         );
     }
 
+    public function testDumpWithPositions() {
+        $parser = (new ParserFactory)->create(
+            ParserFactory::ONLY_PHP7,
+            new Lexer(['usedAttributes' => ['startLine', 'endLine', 'startFilePos', 'endFilePos']])
+        );
+        $dumper = new NodeDumper(['dumpPositions' => true]);
+
+        $code = "<?php\n\$a = 1;\necho \$a;";
+        $expected = <<<'OUT'
+array(
+    0: Expr_Assign[2:1 - 2:6](
+        var: Expr_Variable[2:1 - 2:2](
+            name: a
+        )
+        expr: Scalar_LNumber[2:6 - 2:6](
+            value: 1
+        )
+    )
+    1: Stmt_Echo[3:1 - 3:8](
+        exprs: array(
+            0: Expr_Variable[3:6 - 3:7](
+                name: a
+            )
+        )
+    )
+)
+OUT;
+
+        $stmts = $parser->parse($code);
+        $dump = $dumper->dump($stmts, $code);
+
+        $this->assertSame($this->canonicalize($expected), $this->canonicalize($dump));
+    }
+
     /**
      * @expectedException        \InvalidArgumentException
      * @expectedExceptionMessage Can only dump nodes and arrays.

test/PhpParser/NodeTraverserTest.php

@@ -2,8 +2,8 @@
 
 namespace PhpParser;
 
-use PhpParser\Node\Scalar\String_;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Scalar\String_;
 
 class NodeTraverserTest extends \PHPUnit_Framework_TestCase
 {
@@ -13,7 +13,7 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
         $echoNode = new Node\Stmt\Echo_(array($str1Node, $str2Node));
         $stmts    = array($echoNode);
 
-        $visitor = $this->getMock('PhpParser\NodeVisitor');
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
 
         $visitor->expects($this->at(0))->method('beforeTraverse')->with($stmts);
         $visitor->expects($this->at(1))->method('enterNode')->with($echoNode);
@@ -36,8 +36,8 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
         $printNode = new Expr\Print_($str1Node);
 
         // first visitor changes the node, second verifies the change
-        $visitor1 = $this->getMock('PhpParser\NodeVisitor');
+        $visitor1 = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
-        $visitor2 = $this->getMock('PhpParser\NodeVisitor');
+        $visitor2 = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
 
         // replace empty statements with string1 node
         $visitor1->expects($this->at(0))->method('beforeTraverse')->with(array())
@@ -81,7 +81,7 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
         $str1Node = new String_('Foo');
         $str2Node = new String_('Bar');
 
-        $visitor = $this->getMock('PhpParser\NodeVisitor');
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
 
         // remove the string1 node, leave the string2 node
         $visitor->expects($this->at(2))->method('leaveNode')->with($str1Node)
@@ -100,7 +100,7 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
         $strR1     = new String_('Replacement 1');
         $strR2     = new String_('Replacement 2');
 
-        $visitor = $this->getMock('PhpParser\NodeVisitor');
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
 
         // replace strMiddle with strR1 and strR2 by merge
         $visitor->expects($this->at(4))->method('leaveNode')->with($strMiddle)
@@ -119,7 +119,7 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
         $strNode = new String_('Foo');
         $stmts = array(array(array($strNode)));
 
-        $visitor = $this->getMock('PhpParser\NodeVisitor');
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
         $visitor->expects($this->at(1))->method('enterNode')->with($strNode);
 
         $traverser = new NodeTraverser;
@@ -136,8 +136,8 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
         $negNode = new Expr\UnaryMinus($mulNode);
         $stmts = array($printNode, $negNode);
 
-        $visitor1 = $this->getMock('PhpParser\NodeVisitor');
+        $visitor1 = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
-        $visitor2 = $this->getMock('PhpParser\NodeVisitor');
+        $visitor2 = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
 
         $visitor1->expects($this->at(1))->method('enterNode')->with($printNode)
             ->will($this->returnValue(NodeTraverser::DONT_TRAVERSE_CHILDREN));
@@ -166,10 +166,67 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
         $this->assertEquals($stmts, $traverser->traverse($stmts));
     }
 
+    public function testStopTraversal() {
+        $varNode1 = new Expr\Variable('a');
+        $varNode2 = new Expr\Variable('b');
+        $varNode3 = new Expr\Variable('c');
+        $mulNode = new Expr\BinaryOp\Mul($varNode1, $varNode2);
+        $printNode = new Expr\Print_($varNode3);
+        $stmts = [$mulNode, $printNode];
+
+        // From enterNode() with array parent
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
+        $visitor->expects($this->at(1))->method('enterNode')->with($mulNode)
+            ->will($this->returnValue(NodeTraverser::STOP_TRAVERSAL));
+        $visitor->expects($this->at(2))->method('afterTraversal');
+        $traverser = new NodeTraverser;
+        $traverser->addVisitor($visitor);
+        $this->assertEquals($stmts, $traverser->traverse($stmts));
+
+        // From enterNode with Node parent
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
+        $visitor->expects($this->at(2))->method('enterNode')->with($varNode1)
+            ->will($this->returnValue(NodeTraverser::STOP_TRAVERSAL));
+        $visitor->expects($this->at(3))->method('afterTraversal');
+        $traverser = new NodeTraverser;
+        $traverser->addVisitor($visitor);
+        $this->assertEquals($stmts, $traverser->traverse($stmts));
+
+        // From leaveNode with Node parent
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
+        $visitor->expects($this->at(3))->method('leaveNode')->with($varNode1)
+            ->will($this->returnValue(NodeTraverser::STOP_TRAVERSAL));
+        $visitor->expects($this->at(4))->method('afterTraversal');
+        $traverser = new NodeTraverser;
+        $traverser->addVisitor($visitor);
+        $this->assertEquals($stmts, $traverser->traverse($stmts));
+
+        // From leaveNode with array parent
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
+        $visitor->expects($this->at(6))->method('leaveNode')->with($mulNode)
+            ->will($this->returnValue(NodeTraverser::STOP_TRAVERSAL));
+        $visitor->expects($this->at(7))->method('afterTraversal');
+        $traverser = new NodeTraverser;
+        $traverser->addVisitor($visitor);
+        $this->assertEquals($stmts, $traverser->traverse($stmts));
+
+        // Check that pending array modifications are still carried out
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
+        $visitor->expects($this->at(6))->method('leaveNode')->with($mulNode)
+            ->will($this->returnValue(NodeTraverser::REMOVE_NODE));
+        $visitor->expects($this->at(7))->method('enterNode')->with($printNode)
+            ->will($this->returnValue(NodeTraverser::STOP_TRAVERSAL));
+        $visitor->expects($this->at(8))->method('afterTraversal');
+        $traverser = new NodeTraverser;
+        $traverser->addVisitor($visitor);
+        $this->assertEquals([$printNode], $traverser->traverse($stmts));
+
+    }
+
     public function testRemovingVisitor() {
-        $visitor1 = $this->getMock('PhpParser\NodeVisitor');
+        $visitor1 = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
-        $visitor2 = $this->getMock('PhpParser\NodeVisitor');
+        $visitor2 = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
-        $visitor3 = $this->getMock('PhpParser\NodeVisitor');
+        $visitor3 = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
 
         $traverser = new NodeTraverser;
         $traverser->addVisitor($visitor1);
@@ -185,15 +242,7 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
         $this->assertAttributeSame($postExpected, 'visitors', $traverser, 'The appropriate visitors are not present after removal');
     }
 
-    public function testCloneNodes() {
+    public function testNoCloneNodes() {
-        $stmts = array(new Node\Stmt\Echo_(array(new String_('Foo'), new String_('Bar'))));
-
-        $traverser = new NodeTraverser(true);
-
-        $this->assertNotSame($stmts, $traverser->traverse($stmts));
-    }
-
-    public function testNoCloneNodesByDefault() {
         $stmts = array(new Node\Stmt\Echo_(array(new String_('Foo'), new String_('Bar'))));
 
         $traverser = new NodeTraverser;
@@ -208,7 +257,7 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
     public function testReplaceByArrayOnlyAllowedIfParentIsArray() {
         $stmts = array(new Node\Expr\UnaryMinus(new Node\Scalar\LNumber(42)));
 
-        $visitor = $this->getMock('PhpParser\NodeVisitor');
+        $visitor = $this->getMockBuilder('PhpParser\NodeVisitor')->getMock();
         $visitor->method('leaveNode')->willReturn(array(new Node\Scalar\DNumber(42.0)));
 
         $traverser = new NodeTraverser();

test/PhpParser/NodeVisitor/NameResolverTest.php

@@ -4,9 +4,9 @@ namespace PhpParser\NodeVisitor;
 
 use PhpParser;
 use PhpParser\Node;
+use PhpParser\Node\Expr;
 use PhpParser\Node\Name;
 use PhpParser\Node\Stmt;
-use PhpParser\Node\Expr;
 
 class NameResolverTest extends \PHPUnit_Framework_TestCase
 {
@@ -118,8 +118,8 @@ namespace {
     new \Hallo\Bar();
     new \Bar();
     new \Bar();
-    bar();
+    \bar();
-    hi();
+    \hi();
     \Hallo\bar();
     \foo\bar();
     \bar();
@@ -199,9 +199,12 @@ interface A extends C, D {
     public function a(A $a) : A;
 }
 
-function fn() : A {}
+function fn(A $a) : A {}
-function fn2() : array {}
+function fn2(array $a) : array {}
-function() : A {};
+function(A $a) : A {};
+
+function fn3(?A $a) : ?A {}
+function fn4(?array $a) : ?array {}
 
 A::b();
 A::$b;
@@ -233,14 +236,20 @@ interface A extends \NS\C, \NS\D
 {
     public function a(\NS\A $a) : \NS\A;
 }
-function fn() : \NS\A
+function fn(\NS\A $a) : \NS\A
 {
 }
-function fn2() : array
+function fn2(array $a) : array
 {
 }
-function () : \NS\A {
+function (\NS\A $a) : \NS\A {
 };
+function fn3(?\NS\A $a) : ?\NS\A
+{
+}
+function fn4(?array $a) : ?array
+{
+}
 \NS\A::b();
 \NS\A::$b;
 \NS\A::B;
@@ -278,7 +287,7 @@ EOC;
         $this->assertEquals($stmts, $traverser->traverse($stmts));
     }
 
-    public function testAddNamespacedName() {
+    public function testAddDeclarationNamespacedName() {
         $nsStmts = array(
             new Stmt\Class_('A'),
             new Stmt\Interface_('B'),
@@ -310,6 +319,29 @@ EOC;
         $this->assertObjectNotHasAttribute('namespacedName', $stmts[0]->stmts[5]->class);
     }
 
+    public function testAddRuntimeResolvedNamespacedName() {
+        $stmts = array(
+            new Stmt\Namespace_(new Name('NS'), array(
+                new Expr\FuncCall(new Name('foo')),
+                new Expr\ConstFetch(new Name('FOO')),
+            )),
+            new Stmt\Namespace_(null, array(
+                new Expr\FuncCall(new Name('foo')),
+                new Expr\ConstFetch(new Name('FOO')),
+            )),
+        );
+
+        $traverser = new PhpParser\NodeTraverser;
+        $traverser->addVisitor(new NameResolver);
+        $stmts = $traverser->traverse($stmts);
+        
+        $this->assertSame('NS\\foo', (string) $stmts[0]->stmts[0]->name->getAttribute('namespacedName'));
+        $this->assertSame('NS\\FOO', (string) $stmts[0]->stmts[1]->name->getAttribute('namespacedName'));
+
+        $this->assertFalse($stmts[1]->stmts[0]->name->hasAttribute('namespacedName'));
+        $this->assertFalse($stmts[1]->stmts[1]->name->hasAttribute('namespacedName'));
+    }
+
     /**
      * @dataProvider provideTestError
      */
@@ -414,4 +446,23 @@ EOC;
         $this->assertSame('PARENT', (string)$methodStmt->stmts[1]->class);
         $this->assertSame('STATIC', (string)$methodStmt->stmts[2]->class);
     }
+
+    public function testAddOriginalNames() {
+        $traverser = new PhpParser\NodeTraverser;
+        $traverser->addVisitor(new NameResolver(null, ['preserveOriginalNames' => true]));
+
+        $n1 = new Name('Bar');
+        $n2 = new Name('bar');
+        $origStmts = [
+            new Stmt\Namespace_(new Name('Foo'), [
+                new Expr\ClassConstFetch($n1, 'FOO'),
+                new Expr\FuncCall($n2),
+            ])
+        ];
+
+        $stmts = $traverser->traverse($origStmts);
+
+        $this->assertSame($n1, $stmts[0]->stmts[0]->class->getAttribute('originalName'));
+        $this->assertSame($n2, $stmts[0]->stmts[1]->name->getAttribute('originalName'));
+    }
 }

test/PhpParser/Parser/MultipleTest.php

@@ -4,10 +4,10 @@ namespace PhpParser\Parser;
 
 use PhpParser\Error;
 use PhpParser\Lexer;
-use PhpParser\Node\Scalar\LNumber;
-use PhpParser\ParserTest;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Scalar\LNumber;
 use PhpParser\Node\Stmt;
+use PhpParser\ParserTest;
 
 require_once __DIR__ . '/../ParserTest.php';
 
@@ -30,7 +30,6 @@ class MultipleTest extends ParserTest {
     /** @dataProvider provideTestParse */
     public function testParse($code, Multiple $parser, $expected) {
         $this->assertEquals($expected, $parser->parse($code));
-        $this->assertSame([], $parser->getErrors());
     }
 
     public function provideTestParse() {
@@ -92,22 +91,4 @@ class MultipleTest extends ParserTest {
         $parser = new Multiple([$parserA, $parserB]);
         $parser->parse('dummy');
     }
-
-    public function testGetErrors() {
-        $errorsA = [new Error('A1'), new Error('A2')];
-        $parserA = $this->getMockBuilder('PhpParser\Parser')->getMock();
-        $parserA->expects($this->at(0))->method('parse');
-        $parserA->expects($this->at(1))
-            ->method('getErrors')->will($this->returnValue($errorsA));
-
-        $errorsB = [new Error('B1'), new Error('B2')];
-        $parserB = $this->getMockBuilder('PhpParser\Parser')->getMock();
-        $parserB->expects($this->at(0))->method('parse');
-        $parserB->expects($this->at(1))
-            ->method('getErrors')->will($this->returnValue($errorsB));
-
-        $parser = new Multiple([$parserA, $parserB]);
-        $parser->parse('dummy');
-        $this->assertSame($errorsA, $parser->getErrors());
-    }
 }

test/PhpParser/ParserTest.php

@@ -30,6 +30,15 @@ abstract class ParserTest extends \PHPUnit_Framework_TestCase
         $parser->parse('<?php use foo as self;');
     }
 
+    /**
+     * @expectedException \PhpParser\Error
+     * @expectedExceptionMessage Unterminated comment on line 1
+     */
+    public function testParserThrowsLexerError() {
+        $parser = $this->getParser(new Lexer());
+        $parser->parse('<?php /*');
+    }
+
     public function testAttributeAssignment() {
         $lexer = new Lexer(array(
             'usedAttributes' => array(
@@ -110,9 +119,9 @@ EOC;
     }
 
     /**
-     * @dataProvider provideTestKindAttributes
+     * @dataProvider provideTestExtraAttributes
      */
-    public function testKindAttributes($code, $expectedAttributes) {
+    public function testExtraAttributes($code, $expectedAttributes) {
         $parser = $this->getParser(new Lexer);
         $stmts = $parser->parse("<?php $code;");
         $attributes = $stmts[0]->getAttributes();
@@ -121,7 +130,7 @@ EOC;
         }
     }
 
-    public function provideTestKindAttributes() {
+    public function provideTestExtraAttributes() {
         return array(
             array('0', ['kind' => Scalar\LNumber::KIND_DEC]),
             array('9', ['kind' => Scalar\LNumber::KIND_DEC]),
@@ -158,6 +167,8 @@ EOC;
             array("die('done')", ['kind' => Expr\Exit_::KIND_DIE]),
             array("exit", ['kind' => Expr\Exit_::KIND_EXIT]),
             array("exit(1)", ['kind' => Expr\Exit_::KIND_EXIT]),
+            array("?>Foo", ['hasLeadingNewline' => false]),
+            array("?>\nFoo", ['hasLeadingNewline' => true]),
         );
     }
 }

test/PhpParser/PrettyPrinterTest.php

@@ -4,8 +4,11 @@ namespace PhpParser;
 
 use PhpParser\Comment;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
+use PhpParser\Node\Scalar\DNumber;
 use PhpParser\Node\Scalar\Encapsed;
 use PhpParser\Node\Scalar\EncapsedStringPart;
+use PhpParser\Node\Scalar\LNumber;
 use PhpParser\Node\Scalar\String_;
 use PhpParser\Node\Stmt;
 use PhpParser\PrettyPrinter\Standard;
@@ -161,4 +164,44 @@ class PrettyPrinterTest extends CodeTestAbstract
             [new Encapsed([new EncapsedStringPart("STR")], $heredoc), '"STR"'],
         ];
     }
+
+    /** @dataProvider provideTestUnnaturalLiterals */
+    public function testUnnaturalLiterals($node, $expected) {
+        $prttyPrinter = new PrettyPrinter\Standard;
+        $result = $prttyPrinter->prettyPrintExpr($node);
+        $this->assertSame($expected, $result);
+    }
+
+    public function provideTestUnnaturalLiterals() {
+        return [
+            [new LNumber(-1), '-1'],
+            [new LNumber(-PHP_INT_MAX - 1), '(-' . PHP_INT_MAX . '-1)'],
+            [new LNumber(-1, ['kind' => LNumber::KIND_BIN]), '-0b1'],
+            [new LNumber(-1, ['kind' => LNumber::KIND_OCT]), '-01'],
+            [new LNumber(-1, ['kind' => LNumber::KIND_HEX]), '-0x1'],
+            [new DNumber(\INF), '\INF'],
+            [new DNumber(-\INF), '-\INF'],
+            [new DNumber(-\NAN), '\NAN'],
+        ];
+    }
+
+    /**
+     * @expectedException \LogicException
+     * @expectedExceptionMessage Cannot pretty-print AST with Error nodes
+     */
+    public function testPrettyPrintWithError() {
+        $stmts = [new Expr\PropertyFetch(new Expr\Variable('a'), new Expr\Error())];
+        $prettyPrinter = new PrettyPrinter\Standard;
+        $prettyPrinter->prettyPrint($stmts);
+    }
+
+    /**
+     * @expectedException \LogicException
+     * @expectedExceptionMessage Cannot pretty-print AST with Error nodes
+     */
+    public function testPrettyPrintWithErrorInClassConstFetch() {
+        $stmts = [new Expr\ClassConstFetch(new Name('Foo'), new Expr\Error())];
+        $prettyPrinter = new PrettyPrinter\Standard;
+        $prettyPrinter->prettyPrint($stmts);
+    }
 }

test/PhpParser/Unserializer/XMLTest.php

@@ -2,8 +2,8 @@
 
 namespace PhpParser\Unserializer;
 
-use PhpParser\Node\Scalar;
 use PhpParser\Comment;
+use PhpParser\Node\Scalar;
 
 class XMLTest extends \PHPUnit_Framework_TestCase
 {

test/code/parser/blockComments.test

@@ -0,0 +1,23 @@
+Comments on blocks
+-----
+<?php
+
+// foo
+{
+    // bar
+    {
+        // baz
+        $a;
+    }
+}
+-----
+array(
+    0: Expr_Variable(
+        name: a
+        comments: array(
+            0: // foo
+            1: // bar
+            2: // baz
+        )
+    )
+)

test/code/parser/errorHandling/lexerErrors.test

@@ -0,0 +1,124 @@
+Lexer errors
+-----
+<?php
+
+$a = 42;
+/*
+$b = 24;
+-----
+Unterminated comment from 4:1 to 5:9
+array(
+    0: Expr_Assign(
+        var: Expr_Variable(
+            name: a
+        )
+        expr: Scalar_LNumber(
+            value: 42
+        )
+    )
+    1: Stmt_Nop(
+        comments: array(
+            0: /*
+            $b = 24;
+        )
+    )
+)
+-----
+<?php
+
+$a = 42;
+@@{ "\1" }@@
+$b = 24;
+-----
+Unexpected character "@@{ "\1" }@@" (ASCII 1) from 4:1 to 4:1
+array(
+    0: Expr_Assign(
+        var: Expr_Variable(
+            name: a
+        )
+        expr: Scalar_LNumber(
+            value: 42
+        )
+    )
+    1: Expr_Assign(
+        var: Expr_Variable(
+            name: b
+        )
+        expr: Scalar_LNumber(
+            value: 24
+        )
+    )
+)
+-----
+<?php
+
+$a = 42;
+@@{ "\0" }@@
+$b = 24;
+-----
+Unexpected null byte from 4:1 to 4:1
+array(
+    0: Expr_Assign(
+        var: Expr_Variable(
+            name: a
+        )
+        expr: Scalar_LNumber(
+            value: 42
+        )
+    )
+    1: Expr_Assign(
+        var: Expr_Variable(
+            name: b
+        )
+        expr: Scalar_LNumber(
+            value: 24
+        )
+    )
+)
+-----
+<?php
+
+$a = 1;
+@@{ "\1" }@@
+$b = 2;
+@@{ "\2" }@@
+$c = 3;
+-----
+Unexpected character "@@{ "\1" }@@" (ASCII 1) from 4:1 to 4:1
+Unexpected character "@@{ "\2" }@@" (ASCII 2) from 6:1 to 6:1
+array(
+    0: Expr_Assign(
+        var: Expr_Variable(
+            name: a
+        )
+        expr: Scalar_LNumber(
+            value: 1
+        )
+    )
+    1: Expr_Assign(
+        var: Expr_Variable(
+            name: b
+        )
+        expr: Scalar_LNumber(
+            value: 2
+        )
+    )
+    2: Expr_Assign(
+        var: Expr_Variable(
+            name: c
+        )
+        expr: Scalar_LNumber(
+            value: 3
+        )
+    )
+)
+-----
+<?php
+
+if ($b) {
+    $a = 1;
+    /* unterminated
+}
+-----
+Unterminated comment from 5:5 to 6:2
+Syntax error, unexpected EOF from 6:2 to 6:2

test/code/parser/errorHandling/recovery.test

@@ -240,3 +240,136 @@ while (true) {
     $i = 2;
 -----
 Syntax error, unexpected EOF from 8:12 to 8:12
+-----
+<?php
+$foo->
+;
+-----
+!!positions
+Syntax error, unexpected ';', expecting T_STRING or T_VARIABLE or '{' or '$' from 3:1 to 3:1
+array(
+    0: Expr_PropertyFetch[2:1 - 3:1](
+        var: Expr_Variable[2:1 - 2:4](
+            name: foo
+        )
+        name: Expr_Error[3:1 - 3:1](
+        )
+    )
+)
+-----
+<?php
+function foo() {
+    $bar->
+}
+-----
+!!positions
+Syntax error, unexpected '}', expecting T_STRING or T_VARIABLE or '{' or '$' from 4:1 to 4:1
+array(
+    0: Stmt_Function[2:1 - 4:1](
+        byRef: false
+        name: foo
+        params: array(
+        )
+        returnType: null
+        stmts: array(
+            0: Expr_PropertyFetch[3:5 - 4:1](
+                var: Expr_Variable[3:5 - 3:8](
+                    name: bar
+                )
+                name: Expr_Error[4:1 - 4:1](
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+new T
+-----
+Syntax error, unexpected EOF from 2:6 to 2:6
+array(
+    0: Expr_New(
+        class: Name(
+            parts: array(
+                0: T
+            )
+        )
+        args: array(
+        )
+    )
+)
+-----
+<?php
+new
+-----
+!!php7,positions
+Syntax error, unexpected EOF from 2:4 to 2:4
+array(
+    0: Expr_New[2:1 - 2:4](
+        class: Expr_Error[2:4 - 2:4](
+        )
+        args: array(
+        )
+    )
+)
+-----
+<?php
+$foo instanceof
+-----
+!!php7
+Syntax error, unexpected EOF from 2:16 to 2:16
+array(
+    0: Expr_Instanceof(
+        expr: Expr_Variable(
+            name: foo
+        )
+        class: Expr_Error(
+        )
+    )
+)
+-----
+<?php
+$
+-----
+!!php7
+Syntax error, unexpected EOF, expecting T_VARIABLE or '{' or '$' from 2:2 to 2:2
+array(
+    0: Expr_Variable(
+        name: Expr_Error(
+        )
+    )
+)
+-----
+<?php
+Foo::$
+-----
+!!php7
+Syntax error, unexpected EOF, expecting T_VARIABLE or '{' or '$' from 2:7 to 2:7
+array(
+    0: Expr_StaticPropertyFetch(
+        class: Name(
+            parts: array(
+                0: Foo
+            )
+        )
+        name: Expr_Error(
+        )
+    )
+)
+-----
+<?php
+Foo::
+-----
+!!php7
+Syntax error, unexpected EOF from 2:6 to 2:6
+array(
+    0: Expr_ClassConstFetch(
+        class: Name(
+            parts: array(
+                0: Foo
+            )
+        )
+        name: Expr_Error(
+        )
+    )
+)

test/code/parser/expr/assign.test

@@ -23,7 +23,6 @@ $a = $b *= $c **= $d;
 
 // by ref assign
 $a =& $b;
-$a =& new B;
 
 // list() assign
 list($a) = $b;
@@ -191,21 +190,7 @@ array(
             0: // by ref assign
         )
     )
-    15: Expr_AssignRef(
+    15: Expr_Assign(
-        var: Expr_Variable(
-            name: a
-        )
-        expr: Expr_New(
-            class: Name(
-                parts: array(
-                    0: B
-                )
-            )
-            args: array(
-            )
-        )
-    )
-    16: Expr_Assign(
         var: Expr_List(
             items: array(
                 0: Expr_ArrayItem(
@@ -227,7 +212,7 @@ array(
             0: // list() assign
         )
     )
-    17: Expr_Assign(
+    16: Expr_Assign(
         var: Expr_List(
             items: array(
                 0: Expr_ArrayItem(
@@ -251,7 +236,7 @@ array(
             name: c
         )
     )
-    18: Expr_Assign(
+    17: Expr_Assign(
         var: Expr_List(
             items: array(
                 0: Expr_ArrayItem(
@@ -261,7 +246,9 @@ array(
                     )
                     byRef: false
                 )
-                1: Expr_List(
+                1: Expr_ArrayItem(
+                    key: null
+                    value: Expr_List(
                         items: array(
                             0: null
                             1: Expr_ArrayItem(
@@ -273,6 +260,8 @@ array(
                             )
                         )
                     )
+                    byRef: false
+                )
                 2: Expr_ArrayItem(
                     key: null
                     value: Expr_Variable(
@@ -286,7 +275,7 @@ array(
             name: e
         )
     )
-    19: Expr_PreInc(
+    18: Expr_PreInc(
         var: Expr_Variable(
             name: a
         )
@@ -294,17 +283,17 @@ array(
             0: // inc/dec
         )
     )
-    20: Expr_PostInc(
+    19: Expr_PostInc(
         var: Expr_Variable(
             name: a
         )
     )
-    21: Expr_PreDec(
+    20: Expr_PreDec(
         var: Expr_Variable(
             name: a
         )
     )
-    22: Expr_PostDec(
+    21: Expr_PostDec(
         var: Expr_Variable(
             name: a
         )

test/code/parser/expr/assignNewByRef.test

@@ -0,0 +1,39 @@
+Assigning new by reference (PHP 5 only)
+-----
+<?php
+$a =& new B;
+-----
+!!php5
+array(
+    0: Expr_AssignRef(
+        var: Expr_Variable(
+            name: a
+        )
+        expr: Expr_New(
+            class: Name(
+                parts: array(
+                    0: B
+                )
+            )
+            args: array(
+            )
+        )
+    )
+)
+-----
+<?php
+$a =& new B;
+-----
+!!php7
+Syntax error, unexpected T_NEW from 2:7 to 2:9
+array(
+    0: Expr_New(
+        class: Name(
+            parts: array(
+                0: B
+            )
+        )
+        args: array(
+        )
+    )
+)

test/code/parser/expr/newWithoutClass.test

@@ -3,6 +3,21 @@ New without a class
 <?php
 new;
 -----
+!!php5
 Syntax error, unexpected ';' from 2:4 to 2:4
 array(
 )
+-----
+<?php
+new;
+-----
+!!php7
+Syntax error, unexpected ';' from 2:4 to 2:4
+array(
+    0: Expr_New(
+        class: Expr_Error(
+        )
+        args: array(
+        )
+    )
+)

test/code/parser/scalar/encapsedNegVarOffset.test

@@ -0,0 +1,72 @@
+Encapsed string negative var offsets
+-----
+<?php
+"$a[-0]";
+"$a[-1]";
+"$a[-0x0]";
+"$a[-00]";
+"$a[@@{ -PHP_INT_MAX - 1 }@@]";
+-----
+!!php7
+array(
+    0: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: a
+                )
+                dim: Scalar_String(
+                    value: -0
+                )
+            )
+        )
+    )
+    1: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: a
+                )
+                dim: Scalar_LNumber(
+                    value: -1
+                )
+            )
+        )
+    )
+    2: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: a
+                )
+                dim: Scalar_String(
+                    value: -0x0
+                )
+            )
+        )
+    )
+    3: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: a
+                )
+                dim: Scalar_String(
+                    value: -00
+                )
+            )
+        )
+    )
+    4: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: a
+                )
+                dim: Scalar_LNumber(
+                    value: @@{ -PHP_INT_MAX - 1 }@@
+                )
+            )
+        )
+    )
+)

test/code/parser/scalar/encapsedString.test

@@ -6,7 +6,11 @@ Encapsed strings
 "$A->B";
 "$A[B]";
 "$A[0]";
+"$A[1234]";
+"$A[9223372036854775808]";
+"$A[000]";
 "$A[0x0]";
+"$A[0b0]";
 "$A[$B]";
 "{$A}";
 "{$A['B']}";
@@ -59,13 +63,49 @@ array(
                 var: Expr_Variable(
                     name: A
                 )
-                dim: Scalar_String(
+                dim: Scalar_LNumber(
                     value: 0
                 )
             )
         )
     )
     4: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: A
+                )
+                dim: Scalar_LNumber(
+                    value: 1234
+                )
+            )
+        )
+    )
+    5: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: A
+                )
+                dim: Scalar_String(
+                    value: 9223372036854775808
+                )
+            )
+        )
+    )
+    6: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: A
+                )
+                dim: Scalar_String(
+                    value: 000
+                )
+            )
+        )
+    )
+    7: Scalar_Encapsed(
         parts: array(
             0: Expr_ArrayDimFetch(
                 var: Expr_Variable(
@@ -77,7 +117,19 @@ array(
             )
         )
     )
-    5: Scalar_Encapsed(
+    8: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: A
+                )
+                dim: Scalar_String(
+                    value: 0b0
+                )
+            )
+        )
+    )
+    9: Scalar_Encapsed(
         parts: array(
             0: Expr_ArrayDimFetch(
                 var: Expr_Variable(
@@ -89,45 +141,45 @@ array(
             )
         )
     )
-    6: Scalar_Encapsed(
-        parts: array(
-            0: Expr_Variable(
-                name: A
-            )
-        )
-    )
-    7: Scalar_Encapsed(
-        parts: array(
-            0: Expr_ArrayDimFetch(
-                var: Expr_Variable(
-                    name: A
-                )
-                dim: Scalar_String(
-                    value: B
-                )
-            )
-        )
-    )
-    8: Scalar_Encapsed(
-        parts: array(
-            0: Expr_Variable(
-                name: A
-            )
-        )
-    )
-    9: Scalar_Encapsed(
-        parts: array(
-            0: Expr_ArrayDimFetch(
-                var: Expr_Variable(
-                    name: A
-                )
-                dim: Scalar_String(
-                    value: B
-                )
-            )
-        )
-    )
     10: Scalar_Encapsed(
+        parts: array(
+            0: Expr_Variable(
+                name: A
+            )
+        )
+    )
+    11: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: A
+                )
+                dim: Scalar_String(
+                    value: B
+                )
+            )
+        )
+    )
+    12: Scalar_Encapsed(
+        parts: array(
+            0: Expr_Variable(
+                name: A
+            )
+        )
+    )
+    13: Scalar_Encapsed(
+        parts: array(
+            0: Expr_ArrayDimFetch(
+                var: Expr_Variable(
+                    name: A
+                )
+                dim: Scalar_String(
+                    value: B
+                )
+            )
+        )
+    )
+    14: Scalar_Encapsed(
         parts: array(
             0: Expr_Variable(
                 name: Expr_Variable(
@@ -136,7 +188,7 @@ array(
             )
         )
     )
-    11: Scalar_Encapsed(
+    15: Scalar_Encapsed(
         parts: array(
             0: Scalar_EncapsedStringPart(
                 value: \{
@@ -149,7 +201,7 @@ array(
             )
         )
     )
-    12: Scalar_Encapsed(
+    16: Scalar_Encapsed(
         parts: array(
             0: Scalar_EncapsedStringPart(
                 value: \{
@@ -162,7 +214,7 @@ array(
             )
         )
     )
-    13: Scalar_Encapsed(
+    17: Scalar_Encapsed(
         parts: array(
             0: Scalar_EncapsedStringPart(
                 value: \
@@ -172,7 +224,7 @@ array(
             )
         )
     )
-    14: Scalar_Encapsed(
+    18: Scalar_Encapsed(
         parts: array(
             0: Scalar_EncapsedStringPart(
                 value: \{
@@ -185,7 +237,7 @@ array(
             )
         )
     )
-    15: Scalar_Encapsed(
+    19: Scalar_Encapsed(
         parts: array(
             0: Expr_Variable(
                 name: Expr_Variable(
@@ -197,7 +249,7 @@ array(
             )
         )
     )
-    16: Scalar_Encapsed(
+    20: Scalar_Encapsed(
         parts: array(
             0: Scalar_EncapsedStringPart(
                 value: $
@@ -212,7 +264,7 @@ array(
             )
         )
     )
-    17: Scalar_Encapsed(
+    21: Scalar_Encapsed(
         parts: array(
             0: Scalar_EncapsedStringPart(
                 value: A
@@ -225,14 +277,14 @@ array(
             )
         )
     )
-    18: Scalar_Encapsed(
+    22: Scalar_Encapsed(
         parts: array(
             0: Expr_Variable(
                 name: A
             )
         )
     )
-    19: Scalar_Encapsed(
+    23: Scalar_Encapsed(
         parts: array(
             0: Expr_Variable(
                 name: A

test/code/parser/scalar/invalidOctal.test

@@ -5,6 +5,11 @@ Invalid octal literals
 -----
 !!php7
 Invalid numeric literal from 2:1 to 2:4
+array(
+    0: Scalar_LNumber(
+        value: 0
+    )
+)
 -----
 <?php
 0787;

test/code/parser/stmt/class/constModifierErrors.test

@@ -6,7 +6,29 @@ class A {
 }
 -----
 !!php7
-Cannot use 'static' as constant modifier on line 3
+Cannot use 'static' as constant modifier from 3:5 to 3:10
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassConst(
+                flags: MODIFIER_STATIC (8)
+                consts: array(
+                    0: Const(
+                        name: X
+                        value: Scalar_LNumber(
+                            value: 1
+                        )
+                    )
+                )
+            )
+        )
+    )
+)
 -----
 <?php
 class A {
@@ -14,7 +36,29 @@ class A {
 }
 -----
 !!php7
-Cannot use 'abstract' as constant modifier on line 3
+Cannot use 'abstract' as constant modifier from 3:5 to 3:12
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassConst(
+                flags: MODIFIER_ABSTRACT (16)
+                consts: array(
+                    0: Const(
+                        name: X
+                        value: Scalar_LNumber(
+                            value: 1
+                        )
+                    )
+                )
+            )
+        )
+    )
+)
 -----
 <?php
 class A {
@@ -22,11 +66,56 @@ class A {
 }
 -----
 !!php7
-Cannot use 'final' as constant modifier on line 3
+Cannot use 'final' as constant modifier from 3:5 to 3:9
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassConst(
+                flags: MODIFIER_FINAL (32)
+                consts: array(
+                    0: Const(
+                        name: X
+                        value: Scalar_LNumber(
+                            value: 1
+                        )
+                    )
+                )
+            )
+        )
+    )
+)
 -----
 <?php
 class A {
     public public const X = 1;
 }
 -----
-Multiple access type modifiers are not allowed on line 3
+!!php7
+Multiple access type modifiers are not allowed from 3:12 to 3:17
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassConst(
+                flags: MODIFIER_PUBLIC (1)
+                consts: array(
+                    0: Const(
+                        name: X
+                        value: Scalar_LNumber(
+                            value: 1
+                        )
+                    )
+                )
+            )
+        )
+    )
+)

test/code/parser/stmt/class/modifier.test

@@ -2,27 +2,148 @@ Invalid modifier combination
 -----
 <?php class A { public public $a; }
 -----
-Multiple access type modifiers are not allowed on line 1
+Multiple access type modifiers are not allowed from 1:24 to 1:29
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_Property(
+                flags: MODIFIER_PUBLIC (1)
+                props: array(
+                    0: Stmt_PropertyProperty(
+                        name: a
+                        default: null
+                    )
+                )
+            )
+        )
+    )
+)
 -----
 <?php class A { public protected $a; }
 -----
-Multiple access type modifiers are not allowed on line 1
+Multiple access type modifiers are not allowed from 1:24 to 1:32
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_Property(
+                flags: MODIFIER_PUBLIC | MODIFIER_PROTECTED (3)
+                props: array(
+                    0: Stmt_PropertyProperty(
+                        name: a
+                        default: null
+                    )
+                )
+            )
+        )
+    )
+)
 -----
-<?php class A { abstract abstract a(); }
+<?php class A { abstract abstract function a(); }
 -----
-Multiple abstract modifiers are not allowed on line 1
+Multiple abstract modifiers are not allowed from 1:26 to 1:33
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassMethod(
+                flags: MODIFIER_ABSTRACT (16)
+                byRef: false
+                name: a
+                params: array(
+                )
+                returnType: null
+                stmts: null
+            )
+        )
+    )
+)
 -----
 <?php class A { static static $a; }
 -----
-Multiple static modifiers are not allowed on line 1
+Multiple static modifiers are not allowed from 1:24 to 1:29
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_Property(
+                flags: MODIFIER_STATIC (8)
+                props: array(
+                    0: Stmt_PropertyProperty(
+                        name: a
+                        default: null
+                    )
+                )
+            )
+        )
+    )
+)
 -----
-<?php class A { final final a() {} }
+<?php class A { final final function a() {} }
 -----
-Multiple final modifiers are not allowed on line 1
+Multiple final modifiers are not allowed from 1:23 to 1:27
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassMethod(
+                flags: MODIFIER_FINAL (32)
+                byRef: false
+                name: a
+                params: array(
+                )
+                returnType: null
+                stmts: array(
+                )
+            )
+        )
+    )
+)
 -----
-<?php class A { abstract final a(); }
+<?php class A { abstract final function a(); }
 -----
-Cannot use the final modifier on an abstract class member on line 1
+Cannot use the final modifier on an abstract class member from 1:26 to 1:30
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassMethod(
+                flags: MODIFIER_ABSTRACT | MODIFIER_FINAL (48)
+                byRef: false
+                name: a
+                params: array(
+                )
+                returnType: null
+                stmts: null
+            )
+        )
+    )
+)
 -----
 <?php abstract final class A { }
 // Type in the partial parse could conceivably be any of 0, 16 or 32
@@ -47,8 +168,48 @@ array(
 -----
 <?php class A { abstract $a; }
 -----
-Properties cannot be declared abstract on line 1
+Properties cannot be declared abstract from 1:17 to 1:24
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_Property(
+                flags: MODIFIER_ABSTRACT (16)
+                props: array(
+                    0: Stmt_PropertyProperty(
+                        name: a
+                        default: null
+                    )
+                )
+            )
+        )
+    )
+)
 -----
 <?php class A { final $a; }
 -----
-Properties cannot be declared final on line 1
+Properties cannot be declared final from 1:17 to 1:21
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_Property(
+                flags: MODIFIER_FINAL (32)
+                props: array(
+                    0: Stmt_PropertyProperty(
+                        name: a
+                        default: null
+                    )
+                )
+            )
+        )
+    )
+)

test/code/parser/stmt/class/name.test

@@ -2,11 +2,33 @@ Invalid class name
 -----
 <?php class self {}
 -----
-Cannot use 'self' as class name as it is reserved on line 1
+Cannot use 'self' as class name as it is reserved from 1:13 to 1:16
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: self
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php class PARENT {}
 -----
-Cannot use 'PARENT' as class name as it is reserved on line 1
+Cannot use 'PARENT' as class name as it is reserved from 1:13 to 1:18
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: PARENT
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php class static {}
 -----
@@ -17,10 +39,40 @@ array(
 <?php class A extends self {}
 -----
 Cannot use 'self' as class name as it is reserved from 1:23 to 1:26
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: Name(
+            parts: array(
+                0: self
+            )
+        )
+        implements: array(
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php class A extends PARENT {}
 -----
 Cannot use 'PARENT' as class name as it is reserved from 1:23 to 1:28
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: Name(
+            parts: array(
+                0: PARENT
+            )
+        )
+        implements: array(
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php class A extends static {}
 -----
@@ -31,10 +83,42 @@ array(
 <?php class A implements self {}
 -----
 Cannot use 'self' as interface name as it is reserved from 1:26 to 1:29
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+            0: Name(
+                parts: array(
+                    0: self
+                )
+            )
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php class A implements PARENT {}
 -----
 Cannot use 'PARENT' as interface name as it is reserved from 1:26 to 1:31
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: A
+        extends: null
+        implements: array(
+            0: Name(
+                parts: array(
+                    0: PARENT
+                )
+            )
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php class A implements static {}
 -----
@@ -44,11 +128,29 @@ array(
 -----
 <?php interface self {}
 -----
-Cannot use 'self' as class name as it is reserved on line 1
+Cannot use 'self' as class name as it is reserved from 1:17 to 1:20
+array(
+    0: Stmt_Interface(
+        name: self
+        extends: array(
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php interface PARENT {}
 -----
-Cannot use 'PARENT' as class name as it is reserved on line 1
+Cannot use 'PARENT' as class name as it is reserved from 1:17 to 1:22
+array(
+    0: Stmt_Interface(
+        name: PARENT
+        extends: array(
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php interface static {}
 -----
@@ -59,10 +161,38 @@ array(
 <?php interface A extends self {}
 -----
 Cannot use 'self' as interface name as it is reserved from 1:27 to 1:30
+array(
+    0: Stmt_Interface(
+        name: A
+        extends: array(
+            0: Name(
+                parts: array(
+                    0: self
+                )
+            )
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php interface A extends PARENT {}
 -----
 Cannot use 'PARENT' as interface name as it is reserved from 1:27 to 1:32
+array(
+    0: Stmt_Interface(
+        name: A
+        extends: array(
+            0: Name(
+                parts: array(
+                    0: PARENT
+                )
+            )
+        )
+        stmts: array(
+        )
+    )
+)
 -----
 <?php interface A extends static {}
 -----