Release PHP-Parser 3.0.0 beta 1

Conflicts:
	CHANGELOG.md

.travis.yml

@@ -2,8 +2,11 @@ language: php
 
 sudo: false
 
+cache:
+  directories:
+    - $HOME/.composer/cache
+
 php:
-  - 5.4
   - 5.5
   - 5.6
   - 7.0
@@ -12,7 +15,7 @@ php:
 
 install:
   - if [ $TRAVIS_PHP_VERSION = '5.6' ]; then composer require satooshi/php-coveralls '~1.0'; fi
-  - composer install --prefer-source
+  - composer install --prefer-dist
 
 matrix:
   allow_failures:
@@ -20,7 +23,9 @@ matrix:
   fast_finish: true
 
 script:
-  if [ $TRAVIS_PHP_VERSION = '5.6' ]; then vendor/bin/phpunit --coverage-clover build/logs/clover.xml; else vendor/bin/phpunit; fi
+  - if [ $TRAVIS_PHP_VERSION = '5.6' ]; then vendor/bin/phpunit --coverage-clover build/logs/clover.xml; else vendor/bin/phpunit; fi
+  - if [ $TRAVIS_PHP_VERSION = '7.0' ]; then test_old/run-php-src.sh; fi
 
 after_success:
   if [ $TRAVIS_PHP_VERSION = '5.6' ]; then php vendor/bin/coveralls; fi
+

CHANGELOG.md

@@ -1,8 +1,137 @@
-Version 2.0.2-dev
+Version 3.0.0-dev
 -----------------
 
 Nothing yet.
 
+Version 3.0.0-beta1 (2016-09-16)
+--------------------------------
+### Added
+
+* [7.1] Function/method and parameter builders now support PHP 7.1 type hints (void, iterable and
+  nullable types).
+* Nodes and Comments now implement `JsonSerializable`. The node kind is stored in a `nodeType`
+  property.
+* The `InlineHTML` node now has an `hasLeadingNewline` attribute, that specifies whether the
+  preceding closing tag contained a newline. The pretty printer honors this attribute.
+* Partial parsing of `$obj->` (with missing property name) is now supported in error recovery mode.
+* The error recovery mode is now exposed in the `php-parse` script through the `--with-recovery`
+  or `-r` flags.
+
+The following changes are also part of PHP-Parser 2.1.1:
+
+* The PHP 7 parser will now generate a parse error for `$var =& new Obj` assignments.
+* Comments on free-standing code blocks will no be retained as comments on the first statement in
+  the code block.
+
+Version 3.0.0-alpha1 (2016-07-25)
+---------------------------------
+
+### Added
+
+* [7.1] Added support for `void` and `iterable` types. These will now be represented as strings
+  (instead of `Name` instances) similar to other builtin types.
+* [7.1] Added support for class constant visibility. The `ClassConst` node now has a `flags` subnode
+  holding the visibility modifier, as well as `isPublic()`, `isProtected()` and `isPrivate()`
+  methods. The constructor changed to accept the additional subnode.
+* [7.1] Added support for nullable types. These are represented using a new `NullableType` node
+  with a single `type` subnode.
+* [7.1] Added support for short array destructuring syntax. This means that `Array` nodes may now
+  appear as the left-hand-side of assignments and foreach value targets. Additionally the array
+  items may now contain `null` values if elements are skipped.
+* [7.1] Added support for keys in list() destructuring. The `List` subnode `vars` has been renamed
+  to `items` and now contains `ArrayItem`s instead of plain variables.
+* [7.1] Added support for multi-catch. The `Catch` subnode `type` has been renamed to `types` and
+  is now an array of `Name`s.
+* `Name::slice()` now supports lengths and negative offsets. This brings it in line with
+  `array_slice()` functionality.
+
+### Changed
+
+Due to PHP 7.1 support additions described above, the node structure changed as follows:
+
+* `void` and `iterable` types are now stored as strings if the PHP 7 parser is used.
+* The `ClassConst` constructor changed to accept an additional `flags` subnode.
+* The `Array` subnode `items` may now contain `null` elements (destructuring).
+* The `List` subnode `vars` has been renamed to `items` and now contains `ArrayItem`s instead of
+  plain variables.
+* The `Catch` subnode `type` has been renamed to `types` and is now an array of `Name`s.
+
+Additionally the following changes were made:
+
+* The `type` subnode on `Class`, `ClassMethod` and `Property` has been renamed to `flags`. The
+  `type` subnode has retained for backwards compatibility and is populated to the same value as
+  `flags`. However, writes to `type` will not update `flags`.
+* The `TryCatch` subnode `finallyStmts` has been replaced with a `finally` subnode that holds an
+  explicit `Finally` node. This allows for more accurate attribute assignment.
+* The `Trait` constructor now has the same form as the `Class` and `Interface` constructors: It
+  takes an array of subnodes. Unlike classes/interfaces, traits can only have a `stmts` subnode.
+* The `NodeDumper` now prints class/method/property/constant modifiers, as well as the include and
+  use type in a textual representation, instead of only showing the number.
+* All methods on `PrettyPrinter\Standard` are now protected. Previoulsy most of them were public.
+
+### Removed
+
+* Removed support for running on PHP 5.4. It is however still possible to parse PHP 5.2-5.4 code
+  while running on a newer version.
+* The deprecated `Comment::setLine()` and `Comment::setText()` methods have been removed.
+* The deprecated `Name::set()`, `Name::setFirst()` and `Name::setLast()` methods have been removed.
+
+Version 2.1.1 (2016-09-16)
+--------------------------
+
+### Changed
+
+* The pretty printer will now escape all control characters in the range `\x00-\x1F` inside double
+  quoted strings. If no special escape sequence is available, an octal escape will be used.
+* The quality of the error recovery has been improved. In particular unterminated expressions should
+  be handled more gracefully.
+* The PHP 7 parser will now generate a parse error for `$var =& new Obj` assignments.
+* Comments on free-standing code blocks will no be retained as comments on the first statement in
+  the code block.
+
+Version 2.1.0 (2016-04-19)
+--------------------------
+
+### Fixed
+
+* Properly support `B""` strings (with uppercase `B`) in a number of places.
+* Fixed reformatting of indented parts in a certain non-standard comment style.
+
+### Added
+
+* Added `dumpComments` option to node dumper, to enable dumping of comments associated with nodes.
+* Added `Stmt\Nop` node, that is used to collect comments located at the end of a block or at the
+  end of a file (without a following node with which they could otherwise be associated).
+* Added `kind` attribute to `Expr\Exit` to distinguish between `exit` and `die`.
+* Added `kind` attribute to `Scalar\LNumber` to distinguish between decimal, binary, octal and
+  hexadecimal numbers.
+* Added `kind` attribtue to `Expr\Array` to distinguish between `array()` and `[]`.
+* Added `kind` attribute to `Scalar\String` and `Scalar\Encapsed` to distinguish between
+  single-quoted, double-quoted, heredoc and nowdoc string.
+* Added `docLabel` attribute to `Scalar\String` and `Scalar\Encapsed`, if it is a heredoc or
+  nowdoc string.
+* Added start file offset information to `Comment` nodes.
+* Added `setReturnType()` method to function and method builders.
+* Added `-h` and `--help` options to `php-parse` script.
+
+### Changed
+
+* Invalid octal literals now throw a parse error in PHP 7 mode.
+* The pretty printer takes all the new attributes mentioned in the previous section into account.
+* The protected `AbstractPrettyPrinter::pComments()` method no longer returns a trailing newline.
+* The bundled autoloader supports library files being stored in a different directory than
+  `PhpParser` for easier downstream distribution.
+
+### Deprecated
+
+* The `Comment::setLine()` and `Comment::setText()` methods have been deprecated. Construct new
+  objects instead.
+
+### Removed
+
+* The internal (but public) method `Scalar\LNumber::parse()` has been removed. A non-internal
+  `LNumber::fromString()` method has been added instead.
+
 Version 2.0.1 (2016-02-28)
 --------------------------
 
@@ -56,7 +185,7 @@ A more detailed description of backwards incompatible changes can be found in th
 
 ### Removed
 
-* Removed support for running on PHP 5.4. It is however still possible to parse PHP 5.2 and PHP 5.3
+* Removed support for running on PHP 5.3. It is however still possible to parse PHP 5.2 and PHP 5.3
   code while running on a newer version.
 * Removed legacy class name aliases. This includes the old non-namespaced class names and the old
   names for classes that were renamed for PHP 7 compatibility.
@@ -88,242 +217,8 @@ A more detailed description of backwards incompatible changes can be found in th
 * The constructor for `Scalar` nodes no longer has a default value. E.g. `new LNumber()` should now
   be written as `new LNumber(0)`.
 
-Version 1.4.0 (2015-07-14)
---------------------------
-
-### Added
-
-* Added interface `PhpParser\Node\FunctionLike`, which is implemented by `Stmt\ClassMethod`,
-  `Stmt\Function_` and `Expr\Closure` nodes. This interface provides getters for their common
-  subnodes.
-* Added `Node\Stmt\ClassLike::getMethod()` to look up a specific method on a class/interface/trait.
-
-### Fixed
-
-* Fixed `isPublic()` return value for implicitly public properties and methods that define and
-  additional modifier like `static` or `abstract`.
-* Properties are now accepted by the trait builder.
-* Fixed `__HALT_COMPILER_OFFSET__` support on HHVM.
-
-Version 1.3.0 (2015-05-02)
---------------------------
-
-### Added
-
-* Errors can now store the attributes of the node/token where the error occurred. Previously only the start line was
-  stored.
-* If file positions are enabled in the lexer, errors can now provide column information if it is available. See
-  [documentation](https://github.com/nikic/PHP-Parser/blob/master/doc/component/Error.markdown#column-information).
-* The parser now provides an experimental error recovery mode, which can be enabled by disabling the `throwOnError`
-  parser option. In this mode the parser will try to construct a partial AST even if the code is not valid PHP. See
-  [documentation](https://github.com/nikic/PHP-Parser/blob/master/doc/component/Error.markdown#error-recovery).
-* Added support for PHP 7 `yield from` expression. It is represented by `Expr\YieldFrom`.
-* Added support for PHP 7 anonymous classes. These are represented by ordinary `Stmt\Class_` nodes with the name set to
-  `null`. Furthermore this implies that `Expr\New_` can now contain a `Stmt\Class_` in its `class` subnode.
-
-### Fixed
-
-* Fixed registration of PHP 7 aliases, for the case where the old name was used before the new name.
-* Fixed handling of precedence when pretty-printing `print` expressions.
-* Floating point numbers are now pretty-printed with a higher precision.
-* Checks for special class names like `self` are now case-insensitive.
-
-Version 1.2.2 (2015-04-03)
---------------------------
-
-* The `NameResolver` now resolves parameter type hints when entering the function/method/closure node. As such other
-  visitors running after it will be able to make use of the resolved names at that point already.
-* The autoloader no longer sets the `unserialize_callback_func` ini option on registration - this is not necessary and
-  may cause issues when running PhpUnit tests with process isolation.
-
-Version 1.2.1 (2015-03-24)
---------------------------
-
-* Fixed registration of the aliases introduced in 1.2.0. Previously the old class names could not be used in
-  `instanceof` checks under some circumstances.
-
-Version 1.2.0 (2015-03-22)
---------------------------
-
-### Changed
-
-* To ensure compatibility with PHP 7, the following node classes have been renamed:
-
-        OLD                             => NEW
-        PhpParser\Node\Expr\Cast\Bool   => PhpParser\Node\Expr\Cast\Bool_
-        PhpParser\Node\Expr\Cast\Int    => PhpParser\Node\Expr\Cast\Int_
-        PhpParser\Node\Expr\Cast\Object => PhpParser\Node\Expr\Cast\Object_
-        PhpParser\Node\Expr\Cast\String => PhpParser\Node\Expr\Cast\String_
-        PhpParser\Node\Scalar\String    => PhpParser\Node\Scalar\String_
-
-  **The previous class names are still supported as aliases.** However it is strongly encouraged to use the new names
-  in order to make your code compatible with PHP 7.
-
-* Subnodes are now stored using real properties instead of an array. This improves performance and memory usage of the
-  initial parse and subsequent node tree operations. The `NodeAbstract` class still supports the old way of specifying
-  subnodes, however this is *deprecated*. In any case properties that are assigned to a node after creation will no
-  longer be considered as subnodes.
-
-* Methods and property declarations will no longer set the `Stmt\Class_::MODIFIER_PUBLIC` flag if no visibility is
-  explicitly given. However the `isPublic()` method will continue to return true. This allows you to distinguish whether
-  a method/property is explicitly or implicitly public and control the pretty printer output more precisely.
-
-* The `Stmt\Class_`, `Stmt\Interface_` and `Stmt\Trait_` nodes now inherit from `Stmt\ClassLike`, which provides a
-  `getMethods()` method. Previously this method was only available on `Stmt\Class_`.
-
-* Support including the `bootstrap.php` file multiple times.
-
-* Make documentation and tests part of the release tarball again.
-
-* Improve support for HHVM and PHP 7.
-
-### Added
-
-* Added support for PHP 7 return type declarations. This adds an additional `returnType` subnode to `Stmt\Function_`,
-  `Stmt\ClassMethod` and `Expr\Closure`.
-
-* Added support for the PHP 7 null coalesce operator `??`. The operator is represented by `Expr\BinaryOp\Coalesce`.
-
-* Added support for the PHP 7 spaceship operator `<=>`. The operator is represented by `Expr\BinaryOp\Spaceship`.
-
-* Added use builder.
-
-* Added global namespace support to the namespace builder.
-
-* Added a constructor flag to `NodeTraverser`, which disables cloning of nodes.
-
-Version 1.1.0 (2015-01-18)
---------------------------
-
-* Methods that do not specify an explicit visibility (e.g. `function method()`) will now have the `MODIFIER_PUBLIC`
-  flag set. This also means that their `isPublic()` method will return true.
-
-* Declaring a property as abstract or final is now an error.
-
-* The `Lexer` and `Lexer\Emulative` classes now accept an `$options` array in their constructors. Currently only the
-  `usedAttributes` option is supported, which determines which attributes will be added to AST nodes. In particular
-  it is now possible to add information on the token and file positions corresponding to a node. For more details see
-  the [Lexer component](https://github.com/nikic/PHP-Parser/blob/master/doc/component/Lexer.markdown) documentation.
-
-* Node visitors can now return `NodeTraverser::DONT_TRAVERSE_CHILDREN` from `enterNode()` in order to skip all children
-  of the current node, for all visitors.
-
-* Added builders for traits and namespaces.
-
-* The class, interface, trait, function, method and property builders now support adding doc comments using the
-  `setDocComment()` method.
-
-* Added support for fully-qualified and namespace-relative names in builders. No longer allow use of name component
-  arrays.
-
-* Do not add documentation and tests to distribution archive files.
-
-Version 1.0.2 (2014-11-04)
---------------------------
-
-* The `NameResolver` visitor now also resolves names in trait adaptations (aliases and precedence declarations).
-
-* Remove stray whitespace when pretty-printing trait adaptations that only change visibility.
-
-Version 1.0.1 (2014-10-14)
---------------------------
-
-* Disallow `new` expressions without a class name. Previously `new;` was accidentally considered to be valid code.
-
-* Support T_ONUMBER token used by HHVM.
-
-* Add ability to directly pass code to the `php-parse.php` script.
-
-* Prevent truncation of `var_dump()` output in the `php-parse.php` script if XDebug is used.
-
-Version 1.0.0 (2014-09-12)
---------------------------
-
-* [BC] Removed deprecated `Template` and `TemplateLoader` classes.
-
-* Fixed XML unserializer to properly work with new namespaced node names.
-
-Version 1.0.0-beta2 (2014-08-31)
---------------------------------
-
-* [PHP 5.6] Updated support for constant scalar expressions to comply with latest changes. This means that arrays
-  and array dimension fetches are now supported as well.
-
-* [PHP 5.6] Direct array dereferencing of constants is supported now, i.e. both `FOO[0]` and `Foo::BAR[0]` are valid
-  now.
-
-* Fixed handling of special class names (`self`, `parent` and `static`) in the name resolver to be case insensitive.
-  Additionally the name resolver now enforces that special class names are only used as unqualified names, e.g. `\self`
-  is considered invalid.
-
-* The case of references to the `static` class name is now preserved. Previously `static` was always lowercased,
-  regardless of the case used in the source code.
-
-* The autoloader now only requires a file if it exists. This allows usages like
-  `class_exists('PhpParser\NotExistingClass')`.
-
-* Added experimental `bin/php-parse.php` script, which is intended to help exploring and debugging the node tree.
-
-* Separated the parser implemention (in `lib/PhpParser/ParserAbstract.php`) and the generated data (in
-  `lib/PhpParser/Parser.php`). Furthermore the parser now uses meaningful variable names and contains comments
-  explaining their usage.
-
-Version 1.0.0-beta1 (2014-03-27)
---------------------------------
-
-* [BC] PHP-Parser now requires PHP 5.3 or newer to run. It is however still possible to *parse* PHP 5.2 source code,
-  while running on a newer version.
-
-* [BC] The library has been moved to use namespaces with the `PhpParser` vendor prefix. However, the old names using
-  underscores are still available as aliases, as such most code should continue running on the new version without
-  further changes.
-
-  However, code performing dispatch operations on `Node::getType()` may be affected by some of the name changes. For
-  example a `+` node will now return type `Expr_BinaryOp_Plus` instead of `Expr_Plus`. In particular this may affect
-  custom pretty printers.
-
-  Due to conflicts with reserved keywords, some class names now end with an underscore, e.g. `PHPParser_Node_Stmt_Class`
-  is now `PhpParser\Node\Stmt\Class_`. (But as usual, the old name is still available)
-
-* [PHP 5.6] Added support for the power operator `**` (node `Expr\BinaryOp\Pow`) and the compound power assignment
-  operator `**=` (node `Expr\AssignOp\Pow`).
-
-* [PHP 5.6] Added support for variadic functions: `Param` nodes now have `variadic` as a boolean subnode.
-
-* [PHP 5.6] Added support for argument unpacking: `Arg` nodes now have `unpack` as a boolean subnode.
-
-* [PHP 5.6] Added support for aliasing of functions and constants. `Stmt\Use_` nodes now have an integral `type`
-  subnode, which is one of `Stmt\Use_::TYPE_NORMAL` (`use`), `Stmt\Use_::TYPE_FUNCTION` (`use function`) or
-  `Stmt\Use_::TYPE_CONSTANT` (`use const`).
-
-  The `NameResolver` now also supports resolution of such aliases.
-
-* [PHP 5.6] Added support for constant scalar expressions. This means that certain expressions are now allowed as the
-  initializer for constants, properties, parameters, static variables, etc.
-
-* [BC] Improved pretty printing of empty statements lists, which are now printed as `{\n}` instead of `{\n    \n}`.
-  This changes the behavior of the protected `PrettyPrinterAbstract::pStmts()` method, so custom pretty printing code
-  making use it of may need to be adjusted.
-
-* Changed the order of some subnodes to be consistent with their order in the sour code. For example `Stmt\If->cond`
-  will now appear before `Stmt\If->stmts` etc.
-
-* Added `Scalar\MagicConstant->getName()`, which returns the name of the magic constant (e.g. `__CLASS__`).
-
-**The following changes are also included in 0.9.5**:
-
-* [BC] Deprecated `PHPParser_Template` and `PHPParser_TemplateLoader`. This functionality does not belong in the main project
-  and - as far as I know - nobody is using it.
-
-* Add `NodeTraverser::removeVisitor()` method, which removes a visitor from the node traverser. This also modifies the
-  corresponding `NodeTraverserInterface`.
-
-* Fix alias resolution in `NameResolver`: Class names are now correctly handled as case-insensitive.
-
-* The undefined variable error, which is used to the lexer to reset the error state, will no longer interfere with
-  custom error handlers.
-
 ---
 
-**This changelog only includes changes from the 1.0 and 2.0 series. For older changes see the
+**This changelog only includes changes from the 2.0 series. For older changes see the
-[0.9 series changelog][https://github.com/nikic/PHP-Parser/blob/0.9/CHANGELOG.md].**
+[1.x series changelog](https://github.com/nikic/PHP-Parser/blob/1.x/CHANGELOG.md) and the
+[0.9 series changelog](https://github.com/nikic/PHP-Parser/blob/0.9/CHANGELOG.md).**

README.md

@@ -3,12 +3,12 @@ PHP Parser
 
 [![Build Status](https://travis-ci.org/nikic/PHP-Parser.svg?branch=master)](https://travis-ci.org/nikic/PHP-Parser) [![Coverage Status](https://coveralls.io/repos/github/nikic/PHP-Parser/badge.svg?branch=master)](https://coveralls.io/github/nikic/PHP-Parser?branch=master)
 
-This is a PHP 5.2 to PHP 7.0 parser written in PHP. Its purpose is to simplify static code analysis and
+This is a PHP 5.2 to PHP 7.1 parser written in PHP. Its purpose is to simplify static code analysis and
 manipulation.
 
 [**Documentation for version 2.x**][doc_master] (stable; for running on PHP >= 5.4; for parsing PHP 5.2 to PHP 7.0).
 
-[Documentation for version 1.x][doc_1_x] (stable; for running on PHP >= 5.3; for parsing PHP 5.2 to PHP 5.6).
+[Documentation for version 1.x][doc_1_x] (unsupported; for running on PHP >= 5.3; for parsing PHP 5.2 to PHP 5.6).
 
 In a Nutshell
 -------------
@@ -93,4 +93,5 @@ Component documentation:
  2. [Lexer](doc/component/Lexer.markdown)
 
  [doc_1_x]: https://github.com/nikic/PHP-Parser/tree/1.x/doc
+ [doc_2_x]: https://github.com/nikic/PHP-Parser/tree/2.x/doc
  [doc_master]: https://github.com/nikic/PHP-Parser/tree/master/doc

UPGRADE-3.0.md

@@ -0,0 +1,48 @@
+Upgrading from PHP-Parser 2.x to 3.0
+====================================
+
+This version does not include any major API changes. Only specific details of the node
+representation have changed in some cases.
+
+### PHP version requirements
+
+PHP-Parser now requires PHP 5.5 or newer to run. It is however still possible to *parse* PHP 5.2,
+5.3 and 5.4 source code, while running on a newer version.
+
+### Changes to the node structure
+
+The following changes are likely to require code changes if the respective nodes are used:
+
+ * The `List` subnode `vars` has been renamed to `items` and now contains `ArrayItem`s instead of
+   plain variables.
+ * The `Catch` subnode `type` has been renamed to `types` and is now an array of `Name`s.
+ * The `TryCatch` subnode `finallyStmts` has been replaced with a `finally` subnode that holds an
+   explicit `Finally` node.
+ * The `type` subnode on `Class`, `ClassMethod` and `Property` has been renamed to `flags`. The
+   `type` subnode has retained for backwards compatibility and is populated to the same value as
+   `flags`. However, writes to `type` will not update `flags` and use of `type` is discouraged.
+
+The following changes are unlikely to require code changes:
+
+ * The `ClassConst` constructor changed to accept an additional `flags` subnode.
+ * The `Trait` constructor now has the same form as the `Class` and `Interface` constructors: It
+   takes an array of subnodes. Unlike classes/interfaces, traits can only have a `stmts` subnode.
+ * The `Array` subnode `items` may now contain `null` elements (due to destructuring).
+ * `void` and `iterable` types are now stored as strings if the PHP 7 parser is used. Previously
+   these would have been represented as `Name` instances.
+
+### Removed methods
+
+The following methods have been removed:
+
+ * `Comment::setLine()`, `Comment::setText()`: Create new `Comment` instances instead.
+ * `Name::set()`, `Name::setFirst()`, `Name::setLast()`: Create new `Name` instances instead. For
+   the latter two a combination of `Name::concat()` and `Name::slice()` can be used.
+
+### Miscellaneous
+
+ * All methods on `PrettyPrinter\Standard` are now protected. Previoulsy most of them were public.
+   The pretty printer should only be invoked using the `prettyPrint()`, `prettyPrintFile()` and
+   `prettyPrintExpr()` methods.
+ * The node dumper now prints numeric values that act as enums/flags in a string representation.
+   If node dumper results are used in tests, updates may be needed to account for this.

bin/php-parse

@@ -27,10 +27,14 @@ if (empty($files)) {
 }
 
 $lexer = new PhpParser\Lexer\Emulative(array('usedAttributes' => array(
-    'startLine', 'endLine', 'startFilePos', 'endFilePos'
+    'startLine', 'endLine', 'startFilePos', 'endFilePos', 'comments'
 )));
-$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7, $lexer);
+$parser = (new PhpParser\ParserFactory)->create(
-$dumper = new PhpParser\NodeDumper;
+    PhpParser\ParserFactory::PREFER_PHP7,
+    $lexer,
+    array('throwOnError' => !$attributes['with-recovery'])
+);
+$dumper = new PhpParser\NodeDumper(['dumpComments' => true]);
 $prettyPrinter = new PhpParser\PrettyPrinter\Standard;
 $serializer = new PhpParser\Serializer\XML;
 
@@ -52,17 +56,15 @@ foreach ($files as $file) {
 
     try {
         $stmts = $parser->parse($code);
-    } catch (PhpParser\Error $e) {
+        foreach ($parser->getErrors() as $error) {
-        if ($attributes['with-column-info'] && $e->hasColumnInfo()) {
+            $message = formatErrorMessage($error, $code, $attributes['with-column-info']);
-            $startLine = $e->getStartLine();
+            echo $message . "\n";
-            $endLine = $e->getEndLine();
-            $startColumn = $e->getStartColumn($code);
-            $endColumn   = $e->getEndColumn($code);
-            $message .= $e->getRawMessage() . " from $startLine:$startColumn to $endLine:$endColumn";
-        } else {
-            $message = $e->getMessage();
         }
-
+        if (null === $stmts) {
+            continue;
+        }
+    } catch (PhpParser\Error $error) {
+        $message = formatErrorMessage($error, $code, $attributes['with-column-info']);
         die($message . "\n");
     }
 
@@ -86,9 +88,23 @@ foreach ($files as $file) {
     }
 }
 
-function showHelp($error) {
+function formatErrorMessage(PhpParser\Error $e, $code, $withColumnInfo) {
-    die($error . "\n\n" .
+    if ($withColumnInfo && $e->hasColumnInfo()) {
-        <<<OUTPUT
+        $startLine = $e->getStartLine();
+        $endLine = $e->getEndLine();
+        $startColumn = $e->getStartColumn($code);
+        $endColumn   = $e->getEndColumn($code);
+        return $e->getRawMessage() . " from $startLine:$startColumn to $endLine:$endColumn";
+    } else {
+        return $e->getMessage();
+    }
+}
+
+function showHelp($error = '') {
+    if ($error) {
+        echo $error . "\n\n";
+    }
+    die(<<<OUTPUT
 Usage: php-parse [operations] file1.php [file2.php ...]
    or: php-parse [operations] "<?php code"
 Turn PHP source code into an abstract syntax tree.
@@ -101,6 +117,8 @@ Operations is a list of the following options (--dump by default):
         --var-dump          var_dump() nodes (for exact structure)
     -N, --resolve-names     Resolve names using NodeVisitor\NameResolver
     -c, --with-column-info  Show column-numbers for errors (if available)
+    -r, --with-recovery     Use parsing with error recovery
+    -h, --help              Display this page
 
 Example:
     php-parse -d -p -N -d file.php
@@ -116,7 +134,8 @@ function parseArgs($args) {
     $operations = array();
     $files = array();
     $attributes = array(
-        'with-column-info' => false,
+        'with-column-info'    => false,
+        'with-recovery' => false,
     );
 
     array_shift($args);
@@ -150,6 +169,14 @@ function parseArgs($args) {
             case '-c';
                 $attributes['with-column-info'] = true;
                 break;
+            case '--with-recovery':
+            case '-r':
+                $attributes['with-recovery'] = true;
+                break;
+            case '--help':
+            case '-h';
+                showHelp();
+                break;
             case '--':
                 $parseOptions = false;
                 break;

composer.json

@@ -10,7 +10,7 @@
         }
     ],
     "require": {
-        "php": ">=5.4",
+        "php": ">=5.5",
         "ext-tokenizer": "*"
     },
     "require-dev": {
@@ -24,7 +24,7 @@
     "bin": ["bin/php-parse"],
     "extra": {
         "branch-alias": {
-            "dev-master": "2.0-dev"
+            "dev-master": "3.0-dev"
         }
     }
 }

doc/0_Introduction.markdown

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-This project is a PHP 5.2 to PHP 7.0 parser **written in PHP itself**.
+This project is a PHP 5.2 to PHP 7.1 parser **written in PHP itself**.
 
 What is this for?
 -----------------
@@ -29,9 +29,9 @@ What can it parse?
 The parser supports parsing PHP 5.2-5.6 and PHP 7.
 
 As the parser is based on the tokens returned by `token_get_all` (which is only able to lex the PHP
-version it runs on), additionally a wrapper for emulating new tokens from 5.5, 5.6 and 7.0 is
+version it runs on), additionally a wrapper for emulating tokens from newer versions is provided.
-provided. This allows to parse PHP 7.0 source code running on PHP 5.4, for example. This emulation
+This allows to parse PHP 7.1 source code running on PHP 5.5, for example. This emulation is somewhat
-is somewhat hacky and not perfect, but it should work well on any sane code.
+hacky and not perfect, but it should work well on any sane code.
 
 What output does it produce?
 ----------------------------

doc/2_Usage_of_basic_components.markdown

@@ -18,7 +18,9 @@ Additionally you may want to set the `xdebug.max_nesting_level` ini option to a
 ini_set('xdebug.max_nesting_level', 3000);
 ```
 
-This ensures that there will be no errors when traversing highly nested node trees.
+This ensures that there will be no errors when traversing highly nested node trees. However, it is
+preferable to disable XDebug completely, as it can easily make this library more than five times
+slower.
 
 Parsing
 -------

doc/4_Code_generation.markdown

@@ -29,6 +29,7 @@ $node = $factory->namespace('Name\Space')
         ->addStmt($factory->method('someMethod')
             ->makePublic()
             ->makeAbstract() // ->makeFinal()
+            ->setReturnType('bool')
             ->addParam($factory->param('someParam')->setTypeHint('SomeClass'))
             ->setDocComment('/**
                               * This method does something.
@@ -74,7 +75,7 @@ abstract class SomeClass extends SomeOtherClass implements A\Few, \Interfaces
      *
      * @param SomeClass And takes a parameter
      */
-    public abstract function someMethod(SomeClass $someParam);
+    public abstract function someMethod(SomeClass $someParam) : bool;
     protected function anotherMethod($someParam = 'test')
     {
         print $someParam;

grammar/README.md

@@ -1,13 +1,12 @@
 What do all those files mean?
 =============================
 
- * `php5.y`:            PHP 5 grammar written in a pseudo language
+ * `php5.y`:             PHP 5 grammar written in a pseudo language
- * `php7.y`:            PHP 7 grammar written in a pseudo language
+ * `php7.y`:             PHP 7 grammar written in a pseudo language
- * `tokens.y`:          Tokens definition shared between PHP 5 and PHP 7 grammars
+ * `tokens.y`:           Tokens definition shared between PHP 5 and PHP 7 grammars
- * `parser.template`:   A `kmyacc` parser prototype file for PHP
+ * `parser.template`:    A `kmyacc` parser prototype file for PHP
- * `tokens.template`:   A `kmyacc` prototype file for the `Tokens` class
+ * `tokens.template`:    A `kmyacc` prototype file for the `Tokens` class
- * `analyze.php`:       Analyzes the grammer and outputs some info about it
+ * `rebuildParsers.php`: Preprocesses the grammar and builds the parser using `kmyacc`
- * `rebuildParser.php`: Preprocesses the grammar and builds the parser using `kmyacc`
 
 .phpy pseudo language
 =====================
@@ -17,13 +16,13 @@ applied to it:
 
  * Nodes are created using the syntax `Name[..., ...]`. This is transformed into
    `new Name(..., ..., attributes())`
- * Some function-like constructs are resolved (see `rebuildParser.php` for a list)
+ * Some function-like constructs are resolved (see `rebuildParsers.php` for a list)
 
 Building the parser
 ===================
 
 In order to rebuild the parser, you need [moriyoshi's fork of kmyacc](https://github.com/moriyoshi/kmyacc-forked).
-After you compiled/installed it, run the `rebuildParser.php` script.
+After you compiled/installed it, run the `rebuildParsers.php` script.
 
 By default only the `Parser.php` is built. If you want to additionally emit debug symbols and create `y.output`, run the
 script with `--debug`. If you want to retain the preprocessed grammar pass `--keep-tmp-grammar`.

grammar/analyze.php

@@ -1,96 +0,0 @@
-<?php
-
-const GRAMMAR_FILE = './php5.y';
-
-const LIB = '(?(DEFINE)
-    (?<singleQuotedString>\'[^\\\\\']*+(?:\\\\.[^\\\\\']*+)*+\')
-    (?<doubleQuotedString>"[^\\\\"]*+(?:\\\\.[^\\\\"]*+)*+")
-    (?<string>(?&singleQuotedString)|(?&doubleQuotedString))
-    (?<comment>/\*[^*]*+(?:\*(?!/)[^*]*+)*+\*/)
-    (?<code>\{[^\'"/{}]*+(?:(?:(?&string)|(?&comment)|(?&code)|/)[^\'"/{}]*+)*+})
-)';
-
-const RULE_BLOCK = '(?<name>[a-z_]++):(?<rules>[^\'"/{};]*+(?:(?:(?&string)|(?&comment)|(?&code)|/|})[^\'"/{};]*+)*+);';
-
-$usedTerminals = array_flip(array(
-    'T_VARIABLE', 'T_STRING', 'T_INLINE_HTML', 'T_ENCAPSED_AND_WHITESPACE',
-    'T_LNUMBER', 'T_DNUMBER', 'T_CONSTANT_ENCAPSED_STRING', 'T_STRING_VARNAME', 'T_NUM_STRING'
-));
-$unusedNonterminals = array_flip(array(
-    'case_separator', 'optional_comma'
-));
-
-function regex($regex) {
-    return '~' . LIB . '(?:' . str_replace('~', '\~', $regex) . ')~';
-}
-
-function magicSplit($regex, $string) {
-    $pieces = preg_split(regex('(?:(?&string)|(?&comment)|(?&code))(*SKIP)(*FAIL)|' . $regex), $string);
-
-    foreach ($pieces as &$piece) {
-        $piece = trim($piece);
-    }
-
-    return array_filter($pieces);
-}
-
-echo '<pre>';
-
-////////////////////
-////////////////////
-////////////////////
-
-list($defs, $ruleBlocks) = magicSplit('%%', file_get_contents(GRAMMAR_FILE));
-
-if ('' !== trim(preg_replace(regex(RULE_BLOCK), '', $ruleBlocks))) {
-    die('Not all rule blocks were properly recognized!');
-}
-
-preg_match_all(regex(RULE_BLOCK), $ruleBlocks, $ruleBlocksMatches, PREG_SET_ORDER);
-foreach ($ruleBlocksMatches as $match) {
-    $ruleBlockName = $match['name'];
-    $rules = magicSplit('\|', $match['rules']);
-
-    foreach ($rules as &$rule) {
-        $parts = magicSplit('\s+', $rule);
-        $usedParts = array();
-
-        foreach ($parts as $part) {
-            if ('{' === $part[0]) {
-                preg_match_all('~\$([0-9]+)~', $part, $backReferencesMatches, PREG_SET_ORDER);
-                foreach ($backReferencesMatches as $match) {
-                    $usedParts[$match[1]] = true;
-                }
-            }
-        }
-
-        $i = 1;
-        foreach ($parts as &$part) {
-            if ('/' === $part[0]) {
-                continue;
-            }
-
-            if (isset($usedParts[$i])) {
-                if ('\'' === $part[0] || '{' === $part[0]
-                    || (ctype_upper($part[0]) && !isset($usedTerminals[$part]))
-                    || (ctype_lower($part[0]) && isset($unusedNonterminals[$part]))
-                ) {
-                    $part = '<span style="background-color: red; color: white;">' . $part . '</span>';
-                } else {
-                    $part = '<strong><em>' . $part . '</em></strong>';
-                }
-            } elseif ((ctype_upper($part[0]) && isset($usedTerminals[$part]))
-                      || (ctype_lower($part[0]) && !isset($unusedNonterminals[$part]))
-
-            ) {
-                $part = '<span style="background-color: blue; color: white;">' . $part . '</span>';
-            }
-
-            ++$i;
-        }
-
-        $rule = implode(' ', $parts);
-    }
-
-    echo $ruleBlockName, ':', "\n", '      ', implode("\n" . '    | ', $rules), "\n", ';', "\n\n";
-}

grammar/php5.y

@@ -9,11 +9,17 @@ start:
     top_statement_list                                      { $$ = $this->handleNamespaces($1); }
 ;
 
-top_statement_list:
+top_statement_list_ex:
-      top_statement_list top_statement                      { pushNormalizing($1, $2); }
+      top_statement_list_ex top_statement                   { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+top_statement_list:
+      top_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
 reserved_non_modifiers:
       T_INCLUDE | T_INCLUDE_ONCE | T_EVAL | T_REQUIRE | T_REQUIRE_ONCE | T_LOGICAL_OR | T_LOGICAL_XOR | T_LOGICAL_AND
     | T_INSTANCEOF | T_NEW | T_CLONE | T_EXIT | T_IF | T_ELSEIF | T_ELSE | T_ENDIF | T_ECHO | T_DO | T_WHILE
@@ -124,11 +130,17 @@ class_const:
     identifier '=' static_scalar                            { $$ = Node\Const_[$1, $3]; }
 ;
 
-inner_statement_list:
+inner_statement_list_ex:
-      inner_statement_list inner_statement                  { pushNormalizing($1, $2); }
+      inner_statement_list_ex inner_statement               { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+inner_statement_list:
+      inner_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
 inner_statement:
       statement                                             { $$ = $1; }
     | function_declaration_statement                        { $$ = $1; }
@@ -138,7 +150,7 @@ inner_statement:
 ;
 
 non_empty_statement:
-      '{' inner_statement_list '}'                          { $$ = $2; }
+      '{' inner_statement_list '}'                          { $$ = $2; prependLeadingComments($$); }
     | T_IF parentheses_expr statement elseif_list else_single
           { $$ = Stmt\If_[$2, ['stmts' => toArray($3), 'elseifs' => $4, 'else' => $5]]; }
     | T_IF parentheses_expr ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
@@ -171,12 +183,15 @@ non_empty_statement:
     | T_THROW expr ';'                                      { $$ = Stmt\Throw_[$2]; }
     | T_GOTO T_STRING ';'                                   { $$ = Stmt\Goto_[$2]; }
     | T_STRING ':'                                          { $$ = Stmt\Label[$1]; }
+    | expr error                                            { $$ = $1; }
     | error                                                 { $$ = array(); /* means: no statement */ }
 ;
 
 statement:
       non_empty_statement                                   { $$ = $1; }
-    | ';'                                                   { $$ = array(); /* means: no statement */ }
+    | ';'
+          { makeNop($$, $this->startAttributeStack[#1]);
+            if ($$ === null) $$ = array(); /* means: no statement */ }
 ;
 
 catches:
@@ -186,12 +201,12 @@ catches:
 
 catch:
     T_CATCH '(' name T_VARIABLE ')' '{' inner_statement_list '}'
-        { $$ = Stmt\Catch_[$3, parseVar($4), $7]; }
+        { $$ = Stmt\Catch_[array($3), parseVar($4), $7]; }
 ;
 
 optional_finally:
       /* empty */                                           { $$ = null; }
-    | T_FINALLY '{' inner_statement_list '}'                { $$ = $3; }
+    | T_FINALLY '{' inner_statement_list '}'                { $$ = Stmt\Finally_[$3]; }
 ;
 
 variables_list:
@@ -220,7 +235,7 @@ class_declaration_statement:
     | T_INTERFACE T_STRING interface_extends_list '{' class_statement_list '}'
           { $$ = Stmt\Interface_[$2, ['extends' => $3, 'stmts' => $5]]; }
     | T_TRAIT T_STRING '{' class_statement_list '}'
-          { $$ = Stmt\Trait_[$2, $4]; }
+          { $$ = Stmt\Trait_[$2, ['stmts' => $4]]; }
 ;
 
 class_entry_type:
@@ -413,7 +428,7 @@ class_statement_list:
 
 class_statement:
       variable_modifiers property_declaration_list ';'      { $$ = Stmt\Property[$1, $2]; }
-    | T_CONST class_const_list ';'                          { $$ = Stmt\ClassConst[$2]; }
+    | T_CONST class_const_list ';'                          { $$ = Stmt\ClassConst[$2, 0]; }
     | method_modifiers T_FUNCTION optional_ref identifier '(' parameter_list ')' optional_return_type method_body
           { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]]; }
     | T_USE name_list trait_adaptations                     { $$ = Stmt\TraitUse[$2, $3]; }
@@ -574,7 +589,10 @@ expr:
     | T_OBJECT_CAST expr                                    { $$ = Expr\Cast\Object_ [$2]; }
     | T_BOOL_CAST expr                                      { $$ = Expr\Cast\Bool_   [$2]; }
     | T_UNSET_CAST expr                                     { $$ = Expr\Cast\Unset_  [$2]; }
-    | T_EXIT exit_expr                                      { $$ = Expr\Exit_        [$2]; }
+    | T_EXIT exit_expr
+          { $attrs = attributes();
+            $attrs['kind'] = strtolower($1) === 'exit' ? Expr\Exit_::KIND_EXIT : Expr\Exit_::KIND_DIE;
+            $$ = new Expr\Exit_($2, $attrs); }
     | '@' expr                                              { $$ = Expr\ErrorSuppress[$2]; }
     | scalar                                                { $$ = $1; }
     | array_expr                                            { $$ = $1; }
@@ -602,14 +620,19 @@ yield_expr:
 ;
 
 array_expr:
-      T_ARRAY '(' array_pair_list ')'                       { $$ = Expr\Array_[$3]; }
+      T_ARRAY '(' array_pair_list ')'
-    | '[' array_pair_list ']'                               { $$ = Expr\Array_[$2]; }
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_LONG;
+            $$ = new Expr\Array_($3, $attrs); }
+    | '[' array_pair_list ']'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_SHORT;
+            $$ = new Expr\Array_($2, $attrs); }
 ;
 
 scalar_dereference:
       array_expr '[' dim_offset ']'                         { $$ = Expr\ArrayDimFetch[$1, $3]; }
     | T_CONSTANT_ENCAPSED_STRING '[' dim_offset ']'
-          { $$ = Expr\ArrayDimFetch[Scalar\String_[Scalar\String_::parse($1, false)], $3]; }
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = Expr\ArrayDimFetch[new Scalar\String_(Scalar\String_::parse($1), $attrs), $3]; }
     | constant '[' dim_offset ']'                           { $$ = Expr\ArrayDimFetch[$1, $3]; }
     | scalar_dereference '[' dim_offset ']'                 { $$ = Expr\ArrayDimFetch[$1, $3]; }
     /* alternative array syntax missing intentionally */
@@ -720,9 +743,11 @@ ctor_arguments:
 ;
 
 common_scalar:
-      T_LNUMBER                                             { $$ = Scalar\LNumber[Scalar\LNumber::parse($1)]; }
+      T_LNUMBER                                             { $$ = Scalar\LNumber::fromString($1, attributes(), true); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
-    | T_CONSTANT_ENCAPSED_STRING                            { $$ = Scalar\String_[Scalar\String_::parse($1, false)]; }
+    | T_CONSTANT_ENCAPSED_STRING
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = new Scalar\String_(Scalar\String_::parse($1, false), $attrs); }
     | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
     | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
     | T_DIR                                                 { $$ = Scalar\MagicConst\Dir[]; }
@@ -732,9 +757,11 @@ common_scalar:
     | T_FUNC_C                                              { $$ = Scalar\MagicConst\Function_[]; }
     | T_NS_C                                                { $$ = Scalar\MagicConst\Namespace_[]; }
     | T_START_HEREDOC T_ENCAPSED_AND_WHITESPACE T_END_HEREDOC
-          { $$ = Scalar\String_[Scalar\String_::parseDocString($1, $2, false)]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_(Scalar\String_::parseDocString($1, $2, false), $attrs); }
     | T_START_HEREDOC T_END_HEREDOC
-          { $$ = Scalar\String_['']; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_('', $attrs); }
 ;
 
 static_scalar:
@@ -792,9 +819,11 @@ scalar:
       common_scalar                                         { $$ = $1; }
     | constant                                              { $$ = $1; }
     | '"' encaps_list '"'
-          { parseEncapsed($2, '"', false); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
+            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
     | T_START_HEREDOC encaps_list T_END_HEREDOC
-          { parseEncapsedDoc($2, false); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            parseEncapsedDoc($2, true); $$ = new Scalar\Encapsed($2, $attrs); }
 ;
 
 static_array_pair_list:
@@ -886,6 +915,7 @@ object_property:
       T_STRING                                              { $$ = $1; }
     | '{' expr '}'                                          { $$ = $2; }
     | variable_without_objects                              { $$ = $1; }
+    | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 list_expr:
@@ -898,7 +928,7 @@ list_expr_elements:
 ;
 
 list_expr_element:
-      variable                                              { $$ = $1; }
+      variable                                              { $$ = Expr\ArrayItem[$1, null, false]; }
     | list_expr                                             { $$ = $1; }
     | /* empty */                                           { $$ = null; }
 ;

grammar/php7.y

@@ -9,11 +9,17 @@ start:
     top_statement_list                                      { $$ = $this->handleNamespaces($1); }
 ;
 
-top_statement_list:
+top_statement_list_ex:
-      top_statement_list top_statement                      { pushNormalizing($1, $2); }
+      top_statement_list_ex top_statement                   { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+top_statement_list:
+      top_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
 reserved_non_modifiers:
       T_INCLUDE | T_INCLUDE_ONCE | T_EVAL | T_REQUIRE | T_REQUIRE_ONCE | T_LOGICAL_OR | T_LOGICAL_XOR | T_LOGICAL_AND
     | T_INSTANCEOF | T_NEW | T_CLONE | T_EXIT | T_IF | T_ELSEIF | T_ELSE | T_ENDIF | T_ECHO | T_DO | T_WHILE
@@ -124,11 +130,17 @@ class_const:
     identifier '=' expr                                     { $$ = Node\Const_[$1, $3]; }
 ;
 
-inner_statement_list:
+inner_statement_list_ex:
-      inner_statement_list inner_statement                  { pushNormalizing($1, $2); }
+      inner_statement_list_ex inner_statement               { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+inner_statement_list:
+      inner_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
 inner_statement:
       statement                                             { $$ = $1; }
     | function_declaration_statement                        { $$ = $1; }
@@ -138,7 +150,7 @@ inner_statement:
 ;
 
 non_empty_statement:
-      '{' inner_statement_list '}'                          { $$ = $2; }
+      '{' inner_statement_list '}'                          { $$ = $2; prependLeadingComments($$); }
     | T_IF '(' expr ')' statement elseif_list else_single
           { $$ = Stmt\If_[$3, ['stmts' => toArray($5), 'elseifs' => $6, 'else' => $7]]; }
     | T_IF '(' expr ')' ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
@@ -167,12 +179,15 @@ non_empty_statement:
     | T_THROW expr ';'                                      { $$ = Stmt\Throw_[$2]; }
     | T_GOTO T_STRING ';'                                   { $$ = Stmt\Goto_[$2]; }
     | T_STRING ':'                                          { $$ = Stmt\Label[$1]; }
+    | expr error                                            { $$ = $1; }
     | error                                                 { $$ = array(); /* means: no statement */ }
 ;
 
 statement:
       non_empty_statement                                   { $$ = $1; }
-    | ';'                                                   { $$ = array(); /* means: no statement */ }
+    | ';'
+          { makeNop($$, $this->startAttributeStack[#1]);
+            if ($$ === null) $$ = array(); /* means: no statement */ }
 ;
 
 catches:
@@ -180,14 +195,19 @@ catches:
     | catches catch                                         { push($1, $2); }
 ;
 
+name_union:
+      name                                                  { init($1); }
+    | name_union '|' name                                   { push($1, $3); }
+;
+
 catch:
-    T_CATCH '(' name T_VARIABLE ')' '{' inner_statement_list '}'
+    T_CATCH '(' name_union T_VARIABLE ')' '{' inner_statement_list '}'
         { $$ = Stmt\Catch_[$3, parseVar($4), $7]; }
 ;
 
 optional_finally:
       /* empty */                                           { $$ = null; }
-    | T_FINALLY '{' inner_statement_list '}'                { $$ = $3; }
+    | T_FINALLY '{' inner_statement_list '}'                { $$ = Stmt\Finally_[$3]; }
 ;
 
 variables_list:
@@ -216,7 +236,7 @@ class_declaration_statement:
     | T_INTERFACE T_STRING interface_extends_list '{' class_statement_list '}'
           { $$ = Stmt\Interface_[$2, ['extends' => $3, 'stmts' => $5]]; }
     | T_TRAIT T_STRING '{' class_statement_list '}'
-          { $$ = Stmt\Trait_[$2, $4]; }
+          { $$ = Stmt\Trait_[$2, ['stmts' => $4]]; }
 ;
 
 class_entry_type:
@@ -329,6 +349,7 @@ foreach_variable:
       variable                                              { $$ = array($1, false); }
     | '&' variable                                          { $$ = array($2, true); }
     | list_expr                                             { $$ = array($1, false); }
+    | array_short_syntax                                    { $$ = array($1, false); }
 ;
 
 parameter_list:
@@ -348,20 +369,25 @@ parameter:
           { $$ = Node\Param[parseVar($4), $6, $1, $2, $3]; }
 ;
 
+type_expr:
+      type                                                  { $$ = $1; }
+    | '?' type                                              { $$ = Node\NullableType[$2]; }
+;
+
 type:
-      name                                                  { $$ = $this->handleScalarTypes($1); }
+      name                                                  { $$ = $this->handleBuiltinTypes($1); }
     | T_ARRAY                                               { $$ = 'array'; }
     | T_CALLABLE                                            { $$ = 'callable'; }
 ;
 
 optional_param_type:
       /* empty */                                           { $$ = null; }
-    | type                                                  { $$ = $1; }
+    | type_expr                                             { $$ = $1; }
 ;
 
 optional_return_type:
       /* empty */                                           { $$ = null; }
-    | ':' type                                              { $$ = $2; }
+    | ':' type_expr                                         { $$ = $2; }
 ;
 
 argument_list:
@@ -406,7 +432,7 @@ class_statement_list:
 
 class_statement:
       variable_modifiers property_declaration_list ';'      { $$ = Stmt\Property[$1, $2]; }
-    | T_CONST class_const_list ';'                          { $$ = Stmt\ClassConst[$2]; }
+    | method_modifiers T_CONST class_const_list ';'         { $$ = Stmt\ClassConst[$3, $1]; }
     | method_modifiers T_FUNCTION optional_ref identifier '(' parameter_list ')' optional_return_type method_body
           { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]]; }
     | T_USE name_list trait_adaptations                     { $$ = Stmt\TraitUse[$2, $3]; }
@@ -495,9 +521,9 @@ for_expr:
 expr:
       variable                                              { $$ = $1; }
     | list_expr '=' expr                                    { $$ = Expr\Assign[$1, $3]; }
+    | array_short_syntax '=' expr                           { $$ = Expr\Assign[$1, $3]; }
     | variable '=' expr                                     { $$ = Expr\Assign[$1, $3]; }
     | variable '=' '&' variable                             { $$ = Expr\AssignRef[$1, $4]; }
-    | variable '=' '&' new_expr                             { $$ = Expr\AssignRef[$1, $4]; }
     | new_expr                                              { $$ = $1; }
     | T_CLONE expr                                          { $$ = Expr\Clone_[$2]; }
     | variable T_PLUS_EQUAL expr                            { $$ = Expr\AssignOp\Plus      [$1, $3]; }
@@ -565,7 +591,10 @@ expr:
     | T_OBJECT_CAST expr                                    { $$ = Expr\Cast\Object_ [$2]; }
     | T_BOOL_CAST expr                                      { $$ = Expr\Cast\Bool_   [$2]; }
     | T_UNSET_CAST expr                                     { $$ = Expr\Cast\Unset_  [$2]; }
-    | T_EXIT exit_expr                                      { $$ = Expr\Exit_        [$2]; }
+    | T_EXIT exit_expr
+          { $attrs = attributes();
+            $attrs['kind'] = strtolower($1) === 'exit' ? Expr\Exit_::KIND_EXIT : Expr\Exit_::KIND_DIE;
+            $$ = new Expr\Exit_($2, $attrs); }
     | '@' expr                                              { $$ = Expr\ErrorSuppress[$2]; }
     | scalar                                                { $$ = $1; }
     | '`' backticks_expr '`'                                { $$ = Expr\ShellExec[$2]; }
@@ -657,14 +686,24 @@ constant:
           { $$ = Expr\ClassConstFetch[$1, $3]; }
 ;
 
+array_short_syntax:
+      '[' array_pair_list ']'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_SHORT;
+            $$ = new Expr\Array_($2, $attrs); }
+;
+
 dereferencable_scalar:
-      T_ARRAY '(' array_pair_list ')'                       { $$ = Expr\Array_[$3]; }
+      T_ARRAY '(' array_pair_list ')'
-    | '[' array_pair_list ']'                               { $$ = Expr\Array_[$2]; }
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_LONG;
-    | T_CONSTANT_ENCAPSED_STRING                            { $$ = Scalar\String_[Scalar\String_::parse($1)]; }
+            $$ = new Expr\Array_($3, $attrs); }
+    | array_short_syntax                                    { $$ = $1; }
+    | T_CONSTANT_ENCAPSED_STRING
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = new Scalar\String_(Scalar\String_::parse($1), $attrs); }
 ;
 
 scalar:
-      T_LNUMBER                                             { $$ = Scalar\LNumber[Scalar\LNumber::parse($1)]; }
+      T_LNUMBER                                             { $$ = Scalar\LNumber::fromString($1, attributes()); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
     | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
     | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
@@ -677,18 +716,17 @@ scalar:
     | dereferencable_scalar                                 { $$ = $1; }
     | constant                                              { $$ = $1; }
     | T_START_HEREDOC T_ENCAPSED_AND_WHITESPACE T_END_HEREDOC
-          { $$ = Scalar\String_[Scalar\String_::parseDocString($1, $2)]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_(Scalar\String_::parseDocString($1, $2), $attrs); }
     | T_START_HEREDOC T_END_HEREDOC
-          { $$ = Scalar\String_['']; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_('', $attrs); }
     | '"' encaps_list '"'
-          { parseEncapsed($2, '"', true); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
+            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
     | T_START_HEREDOC encaps_list T_END_HEREDOC
-          { parseEncapsedDoc($2, true); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
-;
+            parseEncapsedDoc($2, true); $$ = new Scalar\Encapsed($2, $attrs); }
-
-optional_comma:
-      /* empty */
-    | ','
 ;
 
 optional_expr:
@@ -754,6 +792,7 @@ property_name:
       T_STRING                                              { $$ = $1; }
     | '{' expr '}'	                                        { $$ = $2; }
     | simple_variable	                                    { $$ = Expr\Variable[$1]; }
+    | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 list_expr:
@@ -766,18 +805,20 @@ list_expr_elements:
 ;
 
 list_expr_element:
-      variable                                              { $$ = $1; }
+      variable                                              { $$ = Expr\ArrayItem[$1, null, false]; }
     | list_expr                                             { $$ = $1; }
+    | expr T_DOUBLE_ARROW variable                          { $$ = Expr\ArrayItem[$3, $1, false]; }
+    | expr T_DOUBLE_ARROW list_expr                         { $$ = Expr\ArrayItem[$3, $1, false]; }
     | /* empty */                                           { $$ = null; }
 ;
 
 array_pair_list:
-      /* empty */                                           { $$ = array(); }
+      inner_array_pair_list
-    | non_empty_array_pair_list optional_comma              { $$ = $1; }
+          { $$ = $1; $end = count($$)-1; if ($$[$end] === null) unset($$[$end]); }
 ;
 
-non_empty_array_pair_list:
+inner_array_pair_list:
-      non_empty_array_pair_list ',' array_pair              { push($1, $3); }
+      inner_array_pair_list ',' array_pair                  { push($1, $3); }
     | array_pair                                            { init($1); }
 ;
 
@@ -786,6 +827,7 @@ array_pair:
     | expr                                                  { $$ = Expr\ArrayItem[$1, null, false]; }
     | expr T_DOUBLE_ARROW '&' variable                      { $$ = Expr\ArrayItem[$4, $1,   true]; }
     | '&' variable                                          { $$ = Expr\ArrayItem[$2, null, true]; }
+    | /* empty */                                           { $$ = null; }
 ;
 
 encaps_list:

grammar/rebuildParsers.php

@@ -85,7 +85,7 @@ foreach ($grammarFileToName as $grammarFile => $name) {
 
 function resolveNodes($code) {
     return preg_replace_callback(
-        '~(?<name>[A-Z][a-zA-Z_\\\\]++)\s*' . PARAMS . '~',
+        '~\b(?<name>[A-Z][a-zA-Z_\\\\]++)\s*' . PARAMS . '~',
         function($matches) {
             // recurse
             $matches['params'] = resolveNodes($matches['params']);
@@ -169,6 +169,41 @@ function resolveMacros($code) {
                      . ' if (\'\' === $s->value) array_pop(' . $args[0] . ');';
             }
 
+            if ('makeNop' == $name) {
+                assertArgs(2, $args, $name);
+
+                return '$startAttributes = ' . $args[1] . ';'
+                . ' if (isset($startAttributes[\'comments\']))'
+                . ' { ' . $args[0] . ' = new Stmt\Nop([\'comments\' => $startAttributes[\'comments\']]); }'
+                . ' else { ' . $args[0] . ' = null; }';
+            }
+
+            if ('strKind' == $name) {
+                assertArgs(1, $args, $name);
+
+                return '(' . $args[0] . '[0] === "\'" || (' . $args[0] . '[1] === "\'" && '
+                     . '(' . $args[0] . '[0] === \'b\' || ' . $args[0] . '[0] === \'B\')) '
+                     . '? Scalar\String_::KIND_SINGLE_QUOTED : Scalar\String_::KIND_DOUBLE_QUOTED)';
+            }
+
+            if ('setDocStringAttrs' == $name) {
+                assertArgs(2, $args, $name);
+
+                return $args[0] . '[\'kind\'] = strpos(' . $args[1] . ', "\'") === false '
+                     . '? Scalar\String_::KIND_HEREDOC : Scalar\String_::KIND_NOWDOC; '
+                     . 'preg_match(\'/\A[bB]?<<<[ \t]*[\\\'"]?([a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*)[\\\'"]?(?:\r\n|\n|\r)\z/\', ' . $args[1] . ', $matches); '
+                     . $args[0] . '[\'docLabel\'] = $matches[1];';
+            }
+
+            if ('prependLeadingComments' == $name) {
+                assertArgs(1, $args, $name);
+
+                return '$attrs = $this->startAttributeStack[#1]; $stmts = ' . $args[0] . '; '
+                . 'if (!empty($attrs[\'comments\']) && isset($stmts[0])) {'
+                . '$stmts[0]->setAttribute(\'comments\', '
+                . 'array_merge($attrs[\'comments\'], $stmts[0]->getAttribute(\'comments\', []))); }';
+            }
+
             return $matches[0];
         },
         $code

lib/PhpParser/Autoloader.php

@@ -31,7 +31,7 @@ class Autoloader
      */
     static public function autoload($class) {
         if (0 === strpos($class, 'PhpParser\\')) {
-            $fileName = dirname(__DIR__) . '/' . strtr($class, '\\', '/') . '.php';
+            $fileName = __DIR__ . strtr(substr($class, 9), '\\', '/') . '.php';
             if (file_exists($fileName)) {
                 require $fileName;
             }

lib/PhpParser/Builder/Class_.php

@@ -12,7 +12,7 @@ class Class_ extends Declaration
 
     protected $extends = null;
     protected $implements = array();
-    protected $type = 0;
+    protected $flags = 0;
 
     protected $uses = array();
     protected $constants = array();
@@ -112,7 +112,7 @@ class Class_ extends Declaration
      */
     public function getNode() {
         return new Stmt\Class_($this->name, array(
-            'type' => $this->type,
+            'flags' => $this->flags,
             'extends' => $this->extends,
             'implements' => $this->implements,
             'stmts' => array_merge($this->uses, $this->constants, $this->properties, $this->methods),

lib/PhpParser/Builder/FunctionLike.php

@@ -10,6 +10,7 @@ abstract class FunctionLike extends Declaration
 {
     protected $returnByRef = false;
     protected $params = array();
+    protected $returnType = null;
 
     /**
      * Make the function return by reference.
@@ -55,4 +56,19 @@ abstract class FunctionLike extends Declaration
 
         return $this;
     }
-}
+
+    /**
+     * Sets the return type for PHP 7.
+     *
+     * @param string|Node\Name $type One of array, callable, string, int, float, bool, iterable,
+     *                               or a class/interface name.
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function setReturnType($type)
+    {
+        $this->returnType = $this->normalizeType($type);
+
+        return $this;
+    }
+}

lib/PhpParser/Builder/Function_.php

@@ -40,9 +40,10 @@ class Function_ extends FunctionLike
      */
     public function getNode() {
         return new Stmt\Function_($this->name, array(
-            'byRef'  => $this->returnByRef,
+            'byRef'      => $this->returnByRef,
-            'params' => $this->params,
+            'params'     => $this->params,
-            'stmts'  => $this->stmts,
+            'returnType' => $this->returnType,
+            'stmts'      => $this->stmts,
         ), $this->attributes);
     }
-}
+}

lib/PhpParser/Builder/Method.php

@@ -9,7 +9,7 @@ use PhpParser\Node\Stmt;
 class Method extends FunctionLike
 {
     protected $name;
-    protected $type = 0;
+    protected $flags = 0;
     protected $stmts = array();
 
     /**
@@ -116,10 +116,11 @@ class Method extends FunctionLike
      */
     public function getNode() {
         return new Stmt\ClassMethod($this->name, array(
-            'type'   => $this->type,
+            'flags'      => $this->flags,
-            'byRef'  => $this->returnByRef,
+            'byRef'      => $this->returnByRef,
-            'params' => $this->params,
+            'params'     => $this->params,
-            'stmts'  => $this->stmts,
+            'returnType' => $this->returnType,
+            'stmts'      => $this->stmts,
         ), $this->attributes);
     }
 }

lib/PhpParser/Builder/Param.php

@@ -43,10 +43,9 @@ class Param extends PhpParser\BuilderAbstract
      * @return $this The builder instance (for fluid interface)
      */
     public function setTypeHint($type) {
-        if ($type === 'array' || $type === 'callable') {
+        $this->type = $this->normalizeType($type);
-            $this->type = $type;
+        if ($this->type === 'void') {
-        } else {
+            throw new \LogicException('Parameter type cannot be void');
-            $this->type = $this->normalizeName($type);
         }
 
         return $this;
@@ -73,4 +72,4 @@ class Param extends PhpParser\BuilderAbstract
             $this->name, $this->default, $this->type, $this->byRef
         );
     }
-}
+}

lib/PhpParser/Builder/Property.php

@@ -9,7 +9,7 @@ class Property extends PhpParser\BuilderAbstract
 {
     protected $name;
 
-    protected $type = 0;
+    protected $flags = 0;
     protected $default = null;
     protected $attributes = array();
 
@@ -101,7 +101,7 @@ class Property extends PhpParser\BuilderAbstract
      */
     public function getNode() {
         return new Stmt\Property(
-            $this->type !== 0 ? $this->type : Stmt\Class_::MODIFIER_PUBLIC,
+            $this->flags !== 0 ? $this->flags : Stmt\Class_::MODIFIER_PUBLIC,
             array(
                 new Stmt\PropertyProperty($this->name, $this->default)
             ),

lib/PhpParser/Builder/Trait_.php

@@ -49,7 +49,9 @@ class Trait_ extends Declaration
      */
     public function getNode() {
         return new Stmt\Trait_(
-            $this->name, array_merge($this->properties, $this->methods), $this->attributes
+            $this->name, array(
+                'stmts' => array_merge($this->properties, $this->methods)
+            ), $this->attributes
         );
     }
 }

lib/PhpParser/BuilderAbstract.php

@@ -4,6 +4,7 @@ namespace PhpParser;
 
 use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\NullableType;
 use PhpParser\Node\Stmt;
 use PhpParser\Node\Scalar;
 use PhpParser\Comment;
@@ -53,6 +54,49 @@ abstract class BuilderAbstract implements Builder {
         throw new \LogicException('Name must be a string or an instance of PhpParser\Node\Name');
     }
 
+    /**
+     * Normalizes a type: Converts plain-text type names into proper AST representation.
+     *
+     * In particular, builtin types are left as strings, custom types become Names and nullables
+     * are wrapped in NullableType nodes.
+     *
+     * @param Name|string $type The type to normalize
+     *
+     * @return Name|string The normalized type
+     */
+    protected function normalizeType($type) {
+        if (!is_string($type)) {
+            if (!$type instanceof Name && !$type instanceof NullableType) {
+                throw new \LogicException(
+                    'Type must be a string, or an instance of Name or NullableType');
+            }
+            return $type;
+        }
+
+        $nullable = false;
+        if (strlen($type) > 0 && $type[0] === '?') {
+            $nullable = true;
+            $type = substr($type, 1);
+        }
+
+        $builtinTypes = array(
+            'array', 'callable', 'string', 'int', 'float', 'bool', 'iterable', 'void'
+        );
+
+        $lowerType = strtolower($type);
+        if (in_array($lowerType, $builtinTypes)) {
+            $type = $lowerType;
+        } else {
+            $type = $this->normalizeName($type);
+        }
+
+        if ($nullable && $type === 'void') {
+            throw new \LogicException('void type cannot be nullable');
+        }
+
+        return $nullable ? new Node\NullableType($type) : $type;
+    }
+
     /**
      * Normalizes a value: Converts nulls, booleans, integers,
      * floats, strings and arrays into their respective nodes
@@ -125,7 +169,7 @@ abstract class BuilderAbstract implements Builder {
      * @param int $modifier Modifier to set
      */
     protected function setModifier($modifier) {
-        Stmt\Class_::verifyModifier($this->type, $modifier);
+        Stmt\Class_::verifyModifier($this->flags, $modifier);
-        $this->type |= $modifier;
+        $this->flags |= $modifier;
     }
 }

lib/PhpParser/BuilderFactory.php

@@ -20,7 +20,7 @@ class BuilderFactory
 {
     /**
      * Creates a namespace builder.
-     * 
+     *
      * @param null|string|Node\Name $name Name of the namespace
      *
      * @return Builder\Namespace_ The created namespace builder

lib/PhpParser/Comment.php

@@ -2,20 +2,23 @@
 
 namespace PhpParser;
 
-class Comment
+class Comment implements \JsonSerializable
 {
     protected $text;
     protected $line;
+    protected $filePos;
 
     /**
      * Constructs a comment node.
      *
-     * @param string $text Comment text (including comment delimiters like /*)
+     * @param string $text         Comment text (including comment delimiters like /*)
-     * @param int    $line Line number the comment started on
+     * @param int    $startLine    Line number the comment started on
+     * @param int    $startFilePos File offset the comment started on
      */
-    public function __construct($text, $line = -1) {
+    public function __construct($text, $startLine = -1, $startFilePos = -1) {
         $this->text = $text;
-        $this->line = $line;
+        $this->line = $startLine;
+        $this->filePos = $startFilePos;
     }
 
     /**
@@ -27,15 +30,6 @@ class Comment
         return $this->text;
     }
 
-    /**
-     * Sets the comment text.
-     *
-     * @param string $text The comment text (including comment delimiters like /*)
-     */
-    public function setText($text) {
-        $this->text = $text;
-    }
-
     /**
      * Gets the line number the comment started on.
      *
@@ -46,12 +40,12 @@ class Comment
     }
 
     /**
-     * Sets the line number the comment started on.
+     * Gets the file offset the comment started on.
      *
-     * @param int $line Line number
+     * @return int File offset
      */
-    public function setLine($line) {
+    public function getFilePos() {
-        $this->line = $line;
+        return $this->filePos;
     }
 
     /**
@@ -75,7 +69,8 @@ class Comment
      */
     public function getReformattedText() {
         $text = trim($this->text);
-        if (false === strpos($text, "\n")) {
+        $newlinePos = strpos($text, "\n");
+        if (false === $newlinePos) {
             // Single line comments don't need further processing
             return $text;
         } elseif (preg_match('((*BSR_ANYCRLF)(*ANYCRLF)^.*(?:\R\s+\*.*)+$)', $text)) {
@@ -105,15 +100,41 @@ class Comment
             //
             //     /* Some text.
             //        Some more text.
+            //          Indented text.
             //        Even more text. */
             //
-            // is handled by taking the length of the "/* " segment and leaving only that
+            // is handled by removing the difference between the shortest whitespace prefix on all
-            // many space characters before the lines. Thus in the above example only three
+            // lines and the length of the "/* " opening sequence.
-            // space characters are left at the start of every line.
+            $prefixLen = $this->getShortestWhitespacePrefixLen(substr($text, $newlinePos + 1));
-            return preg_replace('(^\s*(?= {' . strlen($matches[0]) . '}(?!\s)))m', '', $text);
+            $removeLen = $prefixLen - strlen($matches[0]);
+            return preg_replace('(^\s{' . $removeLen . '})m', '', $text);
         }
 
         // No idea how to format this comment, so simply return as is
         return $text;
     }
+
+    private function getShortestWhitespacePrefixLen($str) {
+        $lines = explode("\n", $str);
+        $shortestPrefixLen = INF;
+        foreach ($lines as $line) {
+            preg_match('(^\s*)', $line, $matches);
+            $prefixLen = strlen($matches[0]);
+            if ($prefixLen < $shortestPrefixLen) {
+                $shortestPrefixLen = $prefixLen;
+            }
+        }
+        return $shortestPrefixLen;
+    }
+
+    public function jsonSerialize() {
+        // Technically not a node, but we make it look like one anyway
+        $type = $this instanceof Comment\Doc ? 'Comment_Doc' : 'Comment';
+        return [
+            'nodeType' => $type,
+            'text' => $this->text,
+            'line' => $this->line,
+            'filePos' => $this->filePos,
+        ];
+    }
 }

lib/PhpParser/Lexer.php

@@ -2,6 +2,7 @@
 
 namespace PhpParser;
 
+use PhpParser\Node\Scalar\LNumber;
 use PhpParser\Parser\Tokens;
 
 class Lexer
@@ -11,6 +12,7 @@ class Lexer
     protected $pos;
     protected $line;
     protected $filePos;
+    protected $prevCloseTagHasNewline;
 
     protected $tokenMap;
     protected $dropTokens;
@@ -21,10 +23,10 @@ class Lexer
      * Creates a Lexer.
      *
      * @param array $options Options array. Currently only the 'usedAttributes' option is supported,
-     *                       which is an array of attributes to add to the AST nodes. Possible attributes
+     *                       which is an array of attributes to add to the AST nodes. Possible
-     *                       are: 'comments', 'startLine', 'endLine', 'startTokenPos', 'endTokenPos',
+     *                       attributes are: 'comments', 'startLine', 'endLine', 'startTokenPos',
-     *                       'startFilePos', 'endFilePos'. The option defaults to the first three.
+     *                       'endTokenPos', 'startFilePos', 'endFilePos'. The option defaults to the
-     *                       For more info see getNextToken() docs.
+     *                       first three. For more info see getNextToken() docs.
      */
     public function __construct(array $options = array()) {
         // map from internal tokens to PhpParser tokens
@@ -32,7 +34,9 @@ class Lexer
 
         // map of tokens to drop while lexing (the map is only used for isset lookup,
         // that's why the value is simply set to 1; the value is never actually used.)
-        $this->dropTokens = array_fill_keys(array(T_WHITESPACE, T_OPEN_TAG), 1);
+        $this->dropTokens = array_fill_keys(
+            array(T_WHITESPACE, T_OPEN_TAG, T_COMMENT, T_DOC_COMMENT), 1
+        );
 
         // the usedAttributes member is a map of the used attribute names to a dummy
         // value (here "true")
@@ -64,6 +68,10 @@ class Lexer
         $this->pos  = -1;
         $this->line =  1;
         $this->filePos = 0;
+
+        // If inline HTML occurs without preceding code, treat it as if it had a leading newline.
+        // This ensures proper composability, because having a newline is the "safe" assumption.
+        $this->prevCloseTagHasNewline = true;
     }
 
     protected function resetErrors() {
@@ -120,7 +128,7 @@ class Lexer
      *  * 'startTokenPos' => Offset into the token array of the first token in the node.
      *  * 'endTokenPos'   => Offset into the token array of the last token in the node.
      *  * 'startFilePos'  => Offset into the code string of the first character that is part of the node.
-     *  * 'endFilePos'    => Offset into the code string of the last character that is part of the node
+     *  * 'endFilePos'    => Offset into the code string of the last character that is part of the node.
      *
      * @param mixed $value           Variable to store token content in
      * @param mixed $startAttributes Variable to store start attributes in
@@ -140,6 +148,9 @@ class Lexer
                 $token = "\0";
             }
 
+            if (isset($this->usedAttributes['startLine'])) {
+                $startAttributes['startLine'] = $this->line;
+            }
             if (isset($this->usedAttributes['startTokenPos'])) {
                 $startAttributes['startTokenPos'] = $this->pos;
             }
@@ -147,63 +158,53 @@ class Lexer
                 $startAttributes['startFilePos'] = $this->filePos;
             }
 
-            if (is_string($token)) {
+            if (\is_string($token)) {
-                // bug in token_get_all
+                $value = $token;
-                if ('b"' === $token) {
+                if (isset($token[1])) {
-                    $value = 'b"';
+                    // bug in token_get_all
                     $this->filePos += 2;
                     $id = ord('"');
                 } else {
-                    $value = $token;
                     $this->filePos += 1;
                     $id = ord($token);
                 }
-
+            } elseif (!isset($this->dropTokens[$token[0]])) {
-                if (isset($this->usedAttributes['startLine'])) {
+                $value = $token[1];
-                    $startAttributes['startLine'] = $this->line;
+                $id = $this->tokenMap[$token[0]];
-                }
+                if (T_CLOSE_TAG === $token[0]) {
-                if (isset($this->usedAttributes['endLine'])) {
+                    $this->prevCloseTagHasNewline = false !== strpos($token[1], "\n");
-                    $endAttributes['endLine'] = $this->line;
+                } else if (T_INLINE_HTML === $token[0]) {
-                }
+                    $startAttributes['hasLeadingNewline'] = $this->prevCloseTagHasNewline;
-                if (isset($this->usedAttributes['endTokenPos'])) {
-                    $endAttributes['endTokenPos'] = $this->pos;
-                }
-                if (isset($this->usedAttributes['endFilePos'])) {
-                    $endAttributes['endFilePos'] = $this->filePos - 1;
                 }
 
-                return $id;
+                $this->line += substr_count($value, "\n");
+                $this->filePos += \strlen($value);
             } else {
-                $this->line += substr_count($token[1], "\n");
+                if (T_COMMENT === $token[0] || T_DOC_COMMENT === $token[0]) {
-                $this->filePos += strlen($token[1]);
-
-                if (T_COMMENT === $token[0]) {
                     if (isset($this->usedAttributes['comments'])) {
-                        $startAttributes['comments'][] = new Comment($token[1], $token[2]);
+                        $comment = T_DOC_COMMENT === $token[0]
+                            ? new Comment\Doc($token[1], $this->line, $this->filePos)
+                            : new Comment($token[1], $this->line, $this->filePos);
+                        $startAttributes['comments'][] = $comment;
                     }
-                } elseif (T_DOC_COMMENT === $token[0]) {
-                    if (isset($this->usedAttributes['comments'])) {
-                        $startAttributes['comments'][] = new Comment\Doc($token[1], $token[2]);
-                    }
-                } elseif (!isset($this->dropTokens[$token[0]])) {
-                    $value = $token[1];
-
-                    if (isset($this->usedAttributes['startLine'])) {
-                        $startAttributes['startLine'] = $token[2];
-                    }
-                    if (isset($this->usedAttributes['endLine'])) {
-                        $endAttributes['endLine'] = $this->line;
-                    }
-                    if (isset($this->usedAttributes['endTokenPos'])) {
-                        $endAttributes['endTokenPos'] = $this->pos;
-                    }
-                    if (isset($this->usedAttributes['endFilePos'])) {
-                        $endAttributes['endFilePos'] = $this->filePos - 1;
-                    }
-
-                    return $this->tokenMap[$token[0]];
                 }
+
+                $this->line += substr_count($token[1], "\n");
+                $this->filePos += \strlen($token[1]);
+                continue;
             }
+
+            if (isset($this->usedAttributes['endLine'])) {
+                $endAttributes['endLine'] = $this->line;
+            }
+            if (isset($this->usedAttributes['endTokenPos'])) {
+                $endAttributes['endTokenPos'] = $this->pos;
+            }
+            if (isset($this->usedAttributes['endFilePos'])) {
+                $endAttributes['endFilePos'] = $this->filePos - 1;
+            }
+
+            return $id;
         }
 
         throw new \RuntimeException('Reached end of lexer loop');

lib/PhpParser/Lexer/Emulative.php

@@ -21,16 +21,12 @@ class Emulative extends \PhpParser\Lexer
 
     const PHP_7_0 = '7.0.0dev';
     const PHP_5_6 = '5.6.0rc1';
-    const PHP_5_5 = '5.5.0beta1';
 
     public function __construct(array $options = array()) {
         parent::__construct($options);
 
         $newKeywordsPerVersion = array(
-            self::PHP_5_5 => array(
+            // No new keywords since PHP 5.5
-                'finally'       => Tokens::T_FINALLY,
-                'yield'         => Tokens::T_YIELD,
-            ),
         );
 
         $this->newKeywords = array();

lib/PhpParser/Node/Expr/Array_.php

@@ -6,6 +6,10 @@ use PhpParser\Node\Expr;
 
 class Array_ extends Expr
 {
+    // For use in "kind" attribute
+    const KIND_LONG = 1;  // array() syntax
+    const KIND_SHORT = 2; // [] syntax
+
     /** @var ArrayItem[] Items */
     public $items;
 

lib/PhpParser/Node/Expr/Error.php

@@ -0,0 +1,27 @@
+<?php
+
+namespace PhpParser\Node\Expr;
+
+use PhpParser\Node\Expr;
+
+/**
+ * Error node used during parsing with error recovery.
+ *
+ * An error node may be placed at a position where an expression is required, but an error occurred.
+ * Error nodes will not be present if the parser is run in throwOnError mode (the default).
+ */
+class Error extends Expr
+{
+    /**
+     * Constructs an error node.
+     *
+     * @param array $attributes Additional attributes
+     */
+    public function __construct(array $attributes = array()) {
+        parent::__construct($attributes);
+    }
+
+    public function getSubNodeNames() {
+        return array();
+    }
+}

lib/PhpParser/Node/Expr/Exit_.php

@@ -6,6 +6,10 @@ use PhpParser\Node\Expr;
 
 class Exit_ extends Expr
 {
+    /* For use in "kind" attribute */
+    const KIND_EXIT = 1;
+    const KIND_DIE = 2;
+
     /** @var null|Expr Expression */
     public $expr;
 

lib/PhpParser/Node/Expr/List_.php

@@ -6,21 +6,21 @@ use PhpParser\Node\Expr;
 
 class List_ extends Expr
 {
-    /** @var Expr[] List of variables to assign to */
+    /** @var ArrayItem[] List of items to assign to */
-    public $vars;
+    public $items;
 
     /**
      * Constructs a list() destructuring node.
      *
-     * @param Expr[] $vars       List of variables to assign to
+     * @param ArrayItem[] $items      List of items to assign to
-     * @param array  $attributes Additional attributes
+     * @param array       $attributes Additional attributes
      */
-    public function __construct(array $vars, array $attributes = array()) {
+    public function __construct(array $items, array $attributes = array()) {
         parent::__construct($attributes);
-        $this->vars = $vars;
+        $this->items = $items;
     }
 
     public function getSubNodeNames() {
-        return array('vars');
+        return array('items');
     }
 }

lib/PhpParser/Node/Name.php

@@ -103,17 +103,6 @@ class Name extends NodeAbstract
         return implode('\\', $this->parts);
     }
 
-    /**
-     * Sets the whole name.
-     *
-     * @deprecated Create a new Name instead, or manually modify the $parts property
-     *
-     * @param string|array|self $name The name to set the whole name to
-     */
-    public function set($name) {
-        $this->parts = self::prepareName($name);
-    }
-
     /**
      * Prepends a name to this name.
      *
@@ -136,26 +125,6 @@ class Name extends NodeAbstract
         $this->parts = array_merge($this->parts, self::prepareName($name));
     }
 
-    /**
-     * Sets the first part of the name.
-     *
-     * @deprecated Use concat($first, $name->slice(1)) instead
-     *
-     * @param string|array|self $name The name to set the first part to
-     */
-    public function setFirst($name) {
-        array_splice($this->parts, 0, 1, self::prepareName($name));
-    }
-
-    /**
-     * Sets the last part of the name.
-     *
-     * @param string|array|self $name The name to set the last part to
-     */
-    public function setLast($name) {
-        array_splice($this->parts, -1, 1, self::prepareName($name));
-    }
-
     /**
      * Gets a slice of a name (similar to array_slice).
      *
@@ -165,17 +134,31 @@ class Name extends NodeAbstract
      * If the slice is empty, a Name with an empty parts array is returned. While this is
      * meaningless in itself, it works correctly in conjunction with concat().
      *
-     * @param int $offset Offset to start the slice at
+     * Offset and length have the same meaning as in array_slice().
+     *
+     * @param int      $offset Offset to start the slice at (may be negative)
+     * @param int|null $length Length of the slice (may be negative)
      *
      * @return static Sliced name
      */
-    public function slice($offset) {
+    public function slice($offset, $length = null) {
-        // TODO negative offset and length
+        $numParts = count($this->parts);
-        if ($offset < 0 || $offset > count($this->parts)) {
+
+        $realOffset = $offset < 0 ? $offset + $numParts : $offset;
+        if ($realOffset < 0 || $realOffset > $numParts) {
             throw new \OutOfBoundsException(sprintf('Offset %d is out of bounds', $offset));
         }
 
-        return new static(array_slice($this->parts, $offset), $this->attributes);
+        if (null === $length) {
+            $realLength = $numParts - $realOffset;
+        } else {
+            $realLength = $length < 0 ? $length + $numParts - $realOffset : $length;
+            if ($realLength < 0 || $realLength > $numParts) {
+                throw new \OutOfBoundsException(sprintf('Length %d is out of bounds', $length));
+            }
+        }
+
+        return new static(array_slice($this->parts, $realOffset, $realLength), $this->attributes);
     }
 
     /**

lib/PhpParser/Node/NullableType.php

@@ -0,0 +1,26 @@
+<?php
+
+namespace PhpParser\Node;
+
+use PhpParser\NodeAbstract;
+
+class NullableType extends NodeAbstract
+{
+    /** @var string|Name Type */
+    public $type;
+
+    /**
+     * Constructs a nullable type (wrapping another type).
+     *
+     * @param string|Name $type       Type
+     * @param array       $attributes Additional attributes
+     */
+    public function __construct($type, array $attributes = array()) {
+        parent::__construct($attributes);
+        $this->type = $type;
+    }
+
+    public function getSubNodeNames() {
+        return array('type');
+    }
+}

lib/PhpParser/Node/Scalar/LNumber.php

@@ -2,10 +2,17 @@
 
 namespace PhpParser\Node\Scalar;
 
+use PhpParser\Error;
 use PhpParser\Node\Scalar;
 
 class LNumber extends Scalar
 {
+    /* For use in "kind" attribute */
+    const KIND_BIN = 2;
+    const KIND_OCT = 8;
+    const KIND_DEC = 10;
+    const KIND_HEX = 16;
+
     /** @var int Number value */
     public $value;
 
@@ -25,37 +32,36 @@ class LNumber extends Scalar
     }
 
     /**
-     * @internal
+     * Constructs an LNumber node from a string number literal.
      *
-     * Parses an LNUMBER token (dec, hex, oct and bin notations) like PHP would.
+     * @param string $str               String number literal (decimal, octal, hex or binary)
+     * @param array  $attributes        Additional attributes
+     * @param bool   $allowInvalidOctal Whether to allow invalid octal numbers (PHP 5)
      *
-     * @param string $str A string number
+     * @return LNumber The constructed LNumber, including kind attribute
-     *
-     * @return int The parsed number
      */
-    public static function parse($str) {
+    public static function fromString($str, array $attributes = array(), $allowInvalidOctal = false) {
-        // handle plain 0 specially
+        if ('0' !== $str[0] || '0' === $str) {
-        if ('0' === $str) {
+            $attributes['kind'] = LNumber::KIND_DEC;
-            return 0;
+            return new LNumber((int) $str, $attributes);
         }
 
-        // if first char is 0 (and number isn't 0) it's a special syntax
+        if ('x' === $str[1] || 'X' === $str[1]) {
-        if ('0' === $str[0]) {
+            $attributes['kind'] = LNumber::KIND_HEX;
-            // hex
+            return new LNumber(hexdec($str), $attributes);
-            if ('x' === $str[1] || 'X' === $str[1]) {
-                return hexdec($str);
-            }
-
-            // bin
-            if ('b' === $str[1] || 'B' === $str[1]) {
-                return bindec($str);
-            }
-
-            // oct (intval instead of octdec to get proper cutting behavior with malformed numbers)
-            return intval($str, 8);
         }
 
-        // dec
+        if ('b' === $str[1] || 'B' === $str[1]) {
-        return (int) $str;
+            $attributes['kind'] = LNumber::KIND_BIN;
+            return new LNumber(bindec($str), $attributes);
+        }
+
+        if (!$allowInvalidOctal && strpbrk($str, '89')) {
+            throw new Error('Invalid numeric literal', $attributes);
+        }
+
+        // use intval instead of octdec to get proper cutting behavior with malformed numbers
+        $attributes['kind'] = LNumber::KIND_OCT;
+        return new LNumber(intval($str, 8), $attributes);
     }
 }

lib/PhpParser/Node/Scalar/String_.php

@@ -7,6 +7,12 @@ use PhpParser\Node\Scalar;
 
 class String_ extends Scalar
 {
+    /* For use in "kind" attribute */
+    const KIND_SINGLE_QUOTED = 1;
+    const KIND_DOUBLE_QUOTED = 2;
+    const KIND_HEREDOC = 3;
+    const KIND_NOWDOC = 4;
+
     /** @var string String value */
     public $value;
 
@@ -48,7 +54,7 @@ class String_ extends Scalar
      */
     public static function parse($str, $parseUnicodeEscape = true) {
         $bLength = 0;
-        if ('b' === $str[0]) {
+        if ('b' === $str[0] || 'B' === $str[0]) {
             $bLength = 1;
         }
 

lib/PhpParser/Node/Stmt/Catch_.php

@@ -6,8 +6,8 @@ use PhpParser\Node;
 
 class Catch_ extends Node\Stmt
 {
-    /** @var Node\Name Class of exception */
+    /** @var Node\Name[] Types of exceptions to catch */
-    public $type;
+    public $types;
     /** @var string Variable for exception */
     public $var;
     /** @var Node[] Statements */
@@ -16,19 +16,19 @@ class Catch_ extends Node\Stmt
     /**
      * Constructs a catch node.
      *
-     * @param Node\Name $type       Class of exception
+     * @param Node\Name[] $types      Types of exceptions to catch
-     * @param string    $var        Variable for exception
+     * @param string      $var        Variable for exception
-     * @param Node[]    $stmts      Statements
+     * @param Node[]      $stmts      Statements
-     * @param array     $attributes Additional attributes
+     * @param array       $attributes Additional attributes
      */
-    public function __construct(Node\Name $type, $var, array $stmts = array(), array $attributes = array()) {
+    public function __construct(array $types, $var, array $stmts = array(), array $attributes = array()) {
         parent::__construct($attributes);
-        $this->type = $type;
+        $this->types = $types;
         $this->var = $var;
         $this->stmts = $stmts;
     }
 
     public function getSubNodeNames() {
-        return array('type', 'var', 'stmts');
+        return array('types', 'var', 'stmts');
     }
 }

lib/PhpParser/Node/Stmt/ClassConst.php

@@ -3,9 +3,12 @@
 namespace PhpParser\Node\Stmt;
 
 use PhpParser\Node;
+use PhpParser\Error;
 
 class ClassConst extends Node\Stmt
 {
+    /** @var int Modifiers */
+    public $flags;
     /** @var Node\Const_[] Constant declarations */
     public $consts;
 
@@ -13,14 +16,43 @@ class ClassConst extends Node\Stmt
      * Constructs a class const list node.
      *
      * @param Node\Const_[] $consts     Constant declarations
+     * @param int           $flags      Modifiers
      * @param array         $attributes Additional attributes
      */
-    public function __construct(array $consts, array $attributes = array()) {
+    public function __construct(array $consts, $flags = 0, array $attributes = array()) {
+        if ($flags & Class_::MODIFIER_STATIC) {
+            throw new Error("Cannot use 'static' as constant modifier");
+        }
+        if ($flags & Class_::MODIFIER_ABSTRACT) {
+            throw new Error("Cannot use 'abstract' as constant modifier");
+        }
+        if ($flags & Class_::MODIFIER_FINAL) {
+            throw new Error("Cannot use 'final' as constant modifier");
+        }
+
         parent::__construct($attributes);
+        $this->flags = $flags;
         $this->consts = $consts;
     }
 
     public function getSubNodeNames() {
-        return array('consts');
+        return array('flags', 'consts');
+    }
+
+    public function isPublic() {
+        return ($this->flags & Class_::MODIFIER_PUBLIC) !== 0
+            || ($this->flags & Class_::VISIBILITY_MODIFER_MASK) === 0;
+    }
+
+    public function isProtected() {
+        return (bool) ($this->flags & Class_::MODIFIER_PROTECTED);
+    }
+
+    public function isPrivate() {
+        return (bool) ($this->flags & Class_::MODIFIER_PRIVATE);
+    }
+
+    public function isStatic() {
+        return (bool) ($this->flags & Class_::MODIFIER_STATIC);
     }
 }

lib/PhpParser/Node/Stmt/ClassMethod.php

@@ -8,8 +8,8 @@ use PhpParser\Error;
 
 class ClassMethod extends Node\Stmt implements FunctionLike
 {
-    /** @var int Type */
+    /** @var int Flags */
-    public $type;
+    public $flags;
     /** @var bool Whether to return by reference */
     public $byRef;
     /** @var string Name */
@@ -21,12 +21,15 @@ class ClassMethod extends Node\Stmt implements FunctionLike
     /** @var Node[] Statements */
     public $stmts;
 
+    /** @deprecated Use $flags instead */
+    public $type;
+
     /**
      * Constructs a class method node.
      *
      * @param string      $name       Name
      * @param array       $subNodes   Array of the following optional subnodes:
-     *                                'type'       => MODIFIER_PUBLIC: Type
+     *                                'flags       => MODIFIER_PUBLIC: Flags
      *                                'byRef'      => false          : Whether to return by reference
      *                                'params'     => array()        : Parameters
      *                                'returnType' => null           : Return type
@@ -35,14 +38,16 @@ class ClassMethod extends Node\Stmt implements FunctionLike
      */
     public function __construct($name, array $subNodes = array(), array $attributes = array()) {
         parent::__construct($attributes);
-        $this->type = isset($subNodes['type']) ? $subNodes['type'] : 0;
+        $this->flags = isset($subNodes['flags']) ? $subNodes['flags']
+            : (isset($subNodes['type']) ? $subNodes['type'] : 0);
+        $this->type = $this->flags;
         $this->byRef = isset($subNodes['byRef'])  ? $subNodes['byRef']  : false;
         $this->name = $name;
         $this->params = isset($subNodes['params']) ? $subNodes['params'] : array();
         $this->returnType = isset($subNodes['returnType']) ? $subNodes['returnType'] : null;
         $this->stmts = array_key_exists('stmts', $subNodes) ? $subNodes['stmts'] : array();
 
-        if ($this->type & Class_::MODIFIER_STATIC) {
+        if ($this->flags & Class_::MODIFIER_STATIC) {
             switch (strtolower($this->name)) {
                 case '__construct':
                     throw new Error(sprintf('Constructor %s() cannot be static', $this->name));
@@ -55,7 +60,7 @@ class ClassMethod extends Node\Stmt implements FunctionLike
     }
 
     public function getSubNodeNames() {
-        return array('type', 'byRef', 'name', 'params', 'returnType', 'stmts');
+        return array('flags', 'byRef', 'name', 'params', 'returnType', 'stmts');
     }
 
     public function returnsByRef() {
@@ -75,27 +80,27 @@ class ClassMethod extends Node\Stmt implements FunctionLike
     }
 
     public function isPublic() {
-        return ($this->type & Class_::MODIFIER_PUBLIC) !== 0
+        return ($this->flags & Class_::MODIFIER_PUBLIC) !== 0
-            || ($this->type & Class_::VISIBILITY_MODIFER_MASK) === 0;
+            || ($this->flags & Class_::VISIBILITY_MODIFER_MASK) === 0;
     }
 
     public function isProtected() {
-        return (bool) ($this->type & Class_::MODIFIER_PROTECTED);
+        return (bool) ($this->flags & Class_::MODIFIER_PROTECTED);
     }
 
     public function isPrivate() {
-        return (bool) ($this->type & Class_::MODIFIER_PRIVATE);
+        return (bool) ($this->flags & Class_::MODIFIER_PRIVATE);
     }
 
     public function isAbstract() {
-        return (bool) ($this->type & Class_::MODIFIER_ABSTRACT);
+        return (bool) ($this->flags & Class_::MODIFIER_ABSTRACT);
     }
 
     public function isFinal() {
-        return (bool) ($this->type & Class_::MODIFIER_FINAL);
+        return (bool) ($this->flags & Class_::MODIFIER_FINAL);
     }
 
     public function isStatic() {
-        return (bool) ($this->type & Class_::MODIFIER_STATIC);
+        return (bool) ($this->flags & Class_::MODIFIER_STATIC);
     }
 }

lib/PhpParser/Node/Stmt/Class_.php

@@ -17,12 +17,15 @@ class Class_ extends ClassLike
     const VISIBILITY_MODIFER_MASK = 7; // 1 | 2 | 4
 
     /** @var int Type */
-    public $type;
+    public $flags;
     /** @var null|Node\Name Name of extended class */
     public $extends;
     /** @var Node\Name[] Names of implemented interfaces */
     public $implements;
 
+    /** @deprecated Use $flags instead */
+    public $type;
+
     protected static $specialNames = array(
         'self'   => true,
         'parent' => true,
@@ -34,7 +37,7 @@ class Class_ extends ClassLike
      *
      * @param string|null $name       Name
      * @param array       $subNodes   Array of the following optional subnodes:
-     *                                'type'       => 0      : Type
+     *                                'flags'      => 0      : Flags
      *                                'extends'    => null   : Name of extended class
      *                                'implements' => array(): Names of implemented interfaces
      *                                'stmts'      => array(): Statements
@@ -42,7 +45,9 @@ class Class_ extends ClassLike
      */
     public function __construct($name, array $subNodes = array(), array $attributes = array()) {
         parent::__construct($attributes);
-        $this->type = isset($subNodes['type']) ? $subNodes['type'] : 0;
+        $this->flags = isset($subNodes['flags']) ? $subNodes['flags']
+            : (isset($subNodes['type']) ? $subNodes['type'] : 0);
+        $this->type = $this->flags;
         $this->name = $name;
         $this->extends = isset($subNodes['extends']) ? $subNodes['extends'] : null;
         $this->implements = isset($subNodes['implements']) ? $subNodes['implements'] : array();
@@ -70,15 +75,15 @@ class Class_ extends ClassLike
     }
 
     public function getSubNodeNames() {
-        return array('type', 'name', 'extends', 'implements', 'stmts');
+        return array('flags', 'name', 'extends', 'implements', 'stmts');
     }
 
     public function isAbstract() {
-        return (bool) ($this->type & self::MODIFIER_ABSTRACT);
+        return (bool) ($this->flags & self::MODIFIER_ABSTRACT);
     }
 
     public function isFinal() {
-        return (bool) ($this->type & self::MODIFIER_FINAL);
+        return (bool) ($this->flags & self::MODIFIER_FINAL);
     }
 
     public function isAnonymous() {

lib/PhpParser/Node/Stmt/Finally_.php

@@ -0,0 +1,26 @@
+<?php
+
+namespace PhpParser\Node\Stmt;
+
+use PhpParser\Node;
+
+class Finally_ extends Node\Stmt
+{
+    /** @var Node[] Statements */
+    public $stmts;
+
+    /**
+     * Constructs a finally node.
+     *
+     * @param Node[] $stmts      Statements
+     * @param array  $attributes Additional attributes
+     */
+    public function __construct(array $stmts = array(), array $attributes = array()) {
+        parent::__construct($attributes);
+        $this->stmts = $stmts;
+    }
+
+    public function getSubNodeNames() {
+        return array('stmts');
+    }
+}

lib/PhpParser/Node/Stmt/Nop.php

@@ -0,0 +1,13 @@
+<?php
+
+namespace PhpParser\Node\Stmt;
+
+use PhpParser\Node;
+
+/** Nop/empty statement (;). */
+class Nop extends Node\Stmt
+{
+    public function getSubNodeNames() {
+        return array();
+    }
+}

lib/PhpParser/Node/Stmt/Property.php

@@ -8,49 +8,53 @@ use PhpParser\Error;
 class Property extends Node\Stmt
 {
     /** @var int Modifiers */
-    public $type;
+    public $flags;
     /** @var PropertyProperty[] Properties */
     public $props;
 
+    /** @deprecated Use $flags instead */
+    public $type;
+
     /**
      * Constructs a class property list node.
      *
-     * @param int                $type       Modifiers
+     * @param int                $flags      Modifiers
      * @param PropertyProperty[] $props      Properties
      * @param array              $attributes Additional attributes
      */
-    public function __construct($type, array $props, array $attributes = array()) {
+    public function __construct($flags, array $props, array $attributes = array()) {
-        if ($type & Class_::MODIFIER_ABSTRACT) {
+        if ($flags & Class_::MODIFIER_ABSTRACT) {
             throw new Error('Properties cannot be declared abstract');
         }
 
-        if ($type & Class_::MODIFIER_FINAL) {
+        if ($flags & Class_::MODIFIER_FINAL) {
             throw new Error('Properties cannot be declared final');
         }
 
         parent::__construct($attributes);
-        $this->type = $type;
+        $this->flags = $flags;
+        $this->type = $flags;
         $this->props = $props;
     }
 
     public function getSubNodeNames() {
-        return array('type', 'props');
+        return array('flags', 'props');
     }
 
     public function isPublic() {
-        return ($this->type & Class_::MODIFIER_PUBLIC) !== 0
+        return ($this->flags & Class_::MODIFIER_PUBLIC) !== 0
-            || ($this->type & Class_::VISIBILITY_MODIFER_MASK) === 0;
+            || ($this->flags & Class_::VISIBILITY_MODIFER_MASK) === 0;
     }
 
     public function isProtected() {
-        return (bool) ($this->type & Class_::MODIFIER_PROTECTED);
+        return (bool) ($this->flags & Class_::MODIFIER_PROTECTED);
     }
 
     public function isPrivate() {
-        return (bool) ($this->type & Class_::MODIFIER_PRIVATE);
+        return (bool) ($this->flags & Class_::MODIFIER_PRIVATE);
     }
 
     public function isStatic() {
-        return (bool) ($this->type & Class_::MODIFIER_STATIC);
+        return (bool) ($this->flags & Class_::MODIFIER_STATIC);
     }
 }

lib/PhpParser/Node/Stmt/Trait_.php

@@ -10,13 +10,14 @@ class Trait_ extends ClassLike
      * Constructs a trait node.
      *
      * @param string $name       Name
-     * @param Node[] $stmts      Statements
+     * @param array  $subNodes   Array of the following optional subnodes:
+     *                           'stmts' => array(): Statements
      * @param array  $attributes Additional attributes
      */
-    public function __construct($name, array $stmts = array(), array $attributes = array()) {
+    public function __construct($name, array $subNodes = array(), array $attributes = array()) {
         parent::__construct($attributes);
         $this->name = $name;
-        $this->stmts = $stmts;
+        $this->stmts = isset($subNodes['stmts']) ? $subNodes['stmts'] : array();
     }
 
     public function getSubNodeNames() {

lib/PhpParser/Node/Stmt/TryCatch.php

@@ -11,29 +11,29 @@ class TryCatch extends Node\Stmt
     public $stmts;
     /** @var Catch_[] Catches */
     public $catches;
-    /** @var null|Node[] Finally statements */
+    /** @var null|Finally_ Optional finally node */
-    public $finallyStmts;
+    public $finally;
 
     /**
      * Constructs a try catch node.
      *
-     * @param Node[]      $stmts        Statements
+     * @param Node[]        $stmts      Statements
-     * @param Catch_[]    $catches      Catches
+     * @param Catch_[]      $catches    Catches
-     * @param null|Node[] $finallyStmts Finally statements (null means no finally clause)
+     * @param null|Finally_ $finally    Optionaly finally node
-     * @param array|null  $attributes   Additional attributes
+     * @param array|null    $attributes Additional attributes
      */
-    public function __construct(array $stmts, array $catches, array $finallyStmts = null, array $attributes = array()) {
+    public function __construct(array $stmts, array $catches, Finally_ $finally = null, array $attributes = array()) {
-        if (empty($catches) && null === $finallyStmts) {
+        if (empty($catches) && null === $finally) {
             throw new Error('Cannot use try without catch or finally');
         }
 
         parent::__construct($attributes);
         $this->stmts = $stmts;
         $this->catches = $catches;
-        $this->finallyStmts = $finallyStmts;
+        $this->finally = $finally;
     }
 
     public function getSubNodeNames() {
-        return array('stmts', 'catches', 'finallyStmts');
+        return array('stmts', 'catches', 'finally');
     }
 }

lib/PhpParser/NodeAbstract.php

@@ -2,7 +2,7 @@
 
 namespace PhpParser;
 
-abstract class NodeAbstract implements Node
+abstract class NodeAbstract implements Node, \JsonSerializable
 {
     protected $attributes;
 
@@ -82,4 +82,8 @@ abstract class NodeAbstract implements Node
     public function getAttributes() {
         return $this->attributes;
     }
+
+    public function jsonSerialize() {
+        return ['nodeType' => $this->getType()] + get_object_vars($this);
+    }
 }

lib/PhpParser/NodeDumper.php

@@ -2,8 +2,26 @@
 
 namespace PhpParser;
 
+use PhpParser\Node\Expr\Include_;
+use PhpParser\Node\Stmt\Class_;
+use PhpParser\Node\Stmt\GroupUse;
+use PhpParser\Node\Stmt\Use_;
+use PhpParser\Node\Stmt\UseUse;
+
 class NodeDumper
 {
+    private $dumpComments;
+
+    /**
+     * Constructs a NodeDumper.
+     *
+     * @param array $options Boolean option 'dumpComments' controls whether comments should be
+     *                       dumped
+     */
+    public function __construct(array $options = []) {
+        $this->dumpComments = !empty($options['dumpComments']);
+    }
+
     /**
      * Dumps a node or array.
      *
@@ -26,11 +44,24 @@ class NodeDumper
                 } elseif (true === $value) {
                     $r .= 'true';
                 } elseif (is_scalar($value)) {
-                    $r .= $value;
+                    if ('flags' === $key || 'newModifier' === $key) {
+                        $r .= $this->dumpFlags($value);
+                    } else if ('type' === $key && $node instanceof Include_) {
+                        $r .= $this->dumpIncludeType($value);
+                    } else if ('type' === $key
+                            && ($node instanceof Use_ || $node instanceof UseUse || $node instanceof GroupUse)) {
+                        $r .= $this->dumpUseType($value);
+                    } else {
+                        $r .= $value;
+                    }
                 } else {
                     $r .= str_replace("\n", "\n    ", $this->dump($value));
                 }
             }
+
+            if ($this->dumpComments && $comments = $node->getAttribute('comments')) {
+                $r .= "\n    comments: " . str_replace("\n", "\n    ", $this->dump($comments));
+            }
         } elseif (is_array($node)) {
             $r = 'array(';
 
@@ -49,10 +80,68 @@ class NodeDumper
                     $r .= str_replace("\n", "\n    ", $this->dump($value));
                 }
             }
+        } elseif ($node instanceof Comment) {
+            return $node->getReformattedText();
         } else {
             throw new \InvalidArgumentException('Can only dump nodes and arrays.');
         }
 
         return $r . "\n)";
     }
+
+    protected function dumpFlags($flags) {
+        $strs = [];
+        if ($flags & Class_::MODIFIER_PUBLIC) {
+            $strs[] = 'MODIFIER_PUBLIC';
+        }
+        if ($flags & Class_::MODIFIER_PROTECTED) {
+            $strs[] = 'MODIFIER_PROTECTED';
+        }
+        if ($flags & Class_::MODIFIER_PRIVATE) {
+            $strs[] = 'MODIFIER_PRIVATE';
+        }
+        if ($flags & Class_::MODIFIER_ABSTRACT) {
+            $strs[] = 'MODIFIER_ABSTRACT';
+        }
+        if ($flags & Class_::MODIFIER_STATIC) {
+            $strs[] = 'MODIFIER_STATIC';
+        }
+        if ($flags & Class_::MODIFIER_FINAL) {
+            $strs[] = 'MODIFIER_FINAL';
+        }
+
+        if ($strs) {
+            return implode(' | ', $strs) . ' (' . $flags . ')';
+        } else {
+            return $flags;
+        }
+    }
+
+    protected function dumpIncludeType($type) {
+        $map = [
+            Include_::TYPE_INCLUDE      => 'TYPE_INCLUDE',
+            Include_::TYPE_INCLUDE_ONCE => 'TYPE_INCLUDE_ONCE',
+            Include_::TYPE_REQUIRE      => 'TYPE_REQUIRE',
+            Include_::TYPE_REQUIRE_ONCE => 'TYPE_REQURE_ONCE',
+        ];
+
+        if (!isset($map[$type])) {
+            return $type;
+        }
+        return $map[$type] . ' (' . $type . ')';
+    }
+
+    protected function dumpUseType($type) {
+        $map = [
+            Use_::TYPE_UNKNOWN  => 'TYPE_UNKNOWN',
+            Use_::TYPE_NORMAL   => 'TYPE_NORMAL',
+            Use_::TYPE_FUNCTION => 'TYPE_FUNCTION',
+            Use_::TYPE_CONSTANT => 'TYPE_CONSTANT',
+        ];
+
+        if (!isset($map[$type])) {
+            return $type;
+        }
+        return $map[$type] . ' (' . $type . ')';
+    }
 }

lib/PhpParser/NodeTraverser.php

@@ -99,6 +99,12 @@ class NodeTraverser implements NodeTraverserInterface
 
                 foreach ($this->visitors as $visitor) {
                     if (null !== $return = $visitor->leaveNode($subNode)) {
+                        if (is_array($return)) {
+                            throw new \LogicException(
+                                'leaveNode() may only return an array ' .
+                                'if the parent structure is an array'
+                            );
+                        }
                         $subNode = $return;
                     }
                 }

lib/PhpParser/NodeVisitor/NameResolver.php

@@ -74,7 +74,9 @@ class NameResolver extends NodeVisitorAbstract
                 $node->class = $this->resolveClassName($node->class);
             }
         } elseif ($node instanceof Stmt\Catch_) {
-            $node->type = $this->resolveClassName($node->type);
+            foreach ($node->types as &$type) {
+                $type = $this->resolveClassName($type);
+            }
         } elseif ($node instanceof Expr\FuncCall) {
             if ($node->name instanceof Name) {
                 $node->name = $this->resolveOtherName($node->name, Stmt\Use_::TYPE_FUNCTION);

lib/PhpParser/ParserAbstract.php

@@ -88,11 +88,15 @@ abstract class ParserAbstract implements Parser
     protected $startAttributeStack;
     /** @var array End attributes of last *shifted* token */
     protected $endAttributes;
+    /** @var array Start attributes of last *read* token */
+    protected $lookaheadStartAttributes;
 
     /** @var bool Whether to throw on first error */
     protected $throwOnError;
     /** @var Error[] Errors collected during last parse */
     protected $errors;
+    /** @var int Error state, used to avoid error floods */
+    protected $errorState;
 
     /**
      * Creates a parser instance.
@@ -155,7 +159,7 @@ abstract class ParserAbstract implements Parser
         // Current position in the stack(s)
         $this->stackPos = 0;
 
-        $errorState = 0;
+        $this->errorState = 0;
 
         for (;;) {
             //$this->traceNewState($state, $symbol);
@@ -185,6 +189,7 @@ abstract class ParserAbstract implements Parser
                     // This is necessary to assign some meaningful attributes to /* empty */ productions. They'll get
                     // the attributes of the next token, even though they don't contain it themselves.
                     $this->startAttributeStack[$this->stackPos+1] = $startAttributes;
+                    $this->lookaheadStartAttributes = $startAttributes;
 
                     //$this->traceRead($symbol);
                 }
@@ -213,8 +218,8 @@ abstract class ParserAbstract implements Parser
                         $this->endAttributes = $endAttributes;
                         $symbol = self::SYMBOL_NONE;
 
-                        if ($errorState) {
+                        if ($this->errorState) {
-                            --$errorState;
+                            --$this->errorState;
                         }
 
                         if ($action < $this->YYNLSTATES) {
@@ -271,7 +276,7 @@ abstract class ParserAbstract implements Parser
                     $this->semStack[$this->stackPos] = $this->semValue;
                 } else {
                     /* error */
-                    switch ($errorState) {
+                    switch ($this->errorState) {
                         case 0:
                             $msg = $this->getErrorMessage($symbol, $state);
                             $error = new Error($msg, $startAttributes + $endAttributes);
@@ -282,7 +287,7 @@ abstract class ParserAbstract implements Parser
                             // Break missing intentionally
                         case 1:
                         case 2:
-                            $errorState = 3;
+                            $this->errorState = 3;
 
                             // Pop until error-expecting state uncovered
                             while (!(
@@ -348,7 +353,9 @@ abstract class ParserAbstract implements Parser
                 && ($idx = $this->actionBase[$state + $this->YYNLSTATES] + $symbol) >= 0
                 && $idx < $this->actionTableSize && $this->actionCheck[$idx] === $symbol
             ) {
-                if ($this->action[$idx] != $this->unexpectedTokenRule) {
+                if ($this->action[$idx] != $this->unexpectedTokenRule
+                    && $this->action[$idx] != $this->defaultAction
+                ) {
                     if (count($expected) == 4) {
                         /* Too many expected tokens */
                         return array();
@@ -460,8 +467,10 @@ abstract class ParserAbstract implements Parser
                 continue;
             }
 
-            /* declare() and __halt_compiler() can be used before a namespace declaration */
+            /* declare(), __halt_compiler() and nops can be used before a namespace declaration */
-            if ($stmt instanceof Node\Stmt\Declare_ || $stmt instanceof Node\Stmt\HaltCompiler) {
+            if ($stmt instanceof Node\Stmt\Declare_
+                || $stmt instanceof Node\Stmt\HaltCompiler
+                || $stmt instanceof Node\Stmt\Nop) {
                 continue;
             }
 
@@ -476,12 +485,14 @@ abstract class ParserAbstract implements Parser
         return $style;
     }
 
-    protected function handleScalarTypes(Name $name) {
+    protected function handleBuiltinTypes(Name $name) {
         $scalarTypes = [
-            'bool'   => true,
+            'bool'     => true,
-            'int'    => true,
+            'int'      => true,
-            'float'  => true,
+            'float'    => true,
-            'string' => true,
+            'string'   => true,
+            'iterable' => true,
+            'void'     => true,
         ];
 
         if (!$name->isUnqualified()) {

lib/PhpParser/PrettyPrinter/Standard.php

@@ -17,7 +17,7 @@ class Standard extends PrettyPrinterAbstract
 {
     // Special nodes
 
-    public function pParam(Node\Param $node) {
+    protected function pParam(Node\Param $node) {
         return ($node->type ? $this->pType($node->type) . ' ' : '')
              . ($node->byRef ? '&' : '')
              . ($node->variadic ? '...' : '')
@@ -25,77 +25,137 @@ class Standard extends PrettyPrinterAbstract
              . ($node->default ? ' = ' . $this->p($node->default) : '');
     }
 
-    public function pArg(Node\Arg $node) {
+    protected function pArg(Node\Arg $node) {
         return ($node->byRef ? '&' : '') . ($node->unpack ? '...' : '') . $this->p($node->value);
     }
 
-    public function pConst(Node\Const_ $node) {
+    protected function pConst(Node\Const_ $node) {
         return $node->name . ' = ' . $this->p($node->value);
     }
 
+    protected function pNullableType(Node\NullableType $node) {
+        return '?' . $node->type;
+    }
+
     // Names
 
-    public function pName(Name $node) {
+    protected function pName(Name $node) {
         return implode('\\', $node->parts);
     }
 
-    public function pName_FullyQualified(Name\FullyQualified $node) {
+    protected function pName_FullyQualified(Name\FullyQualified $node) {
         return '\\' . implode('\\', $node->parts);
     }
 
-    public function pName_Relative(Name\Relative $node) {
+    protected function pName_Relative(Name\Relative $node) {
         return 'namespace\\' . implode('\\', $node->parts);
     }
 
     // Magic Constants
 
-    public function pScalar_MagicConst_Class(MagicConst\Class_ $node) {
+    protected function pScalar_MagicConst_Class(MagicConst\Class_ $node) {
         return '__CLASS__';
     }
 
-    public function pScalar_MagicConst_Dir(MagicConst\Dir $node) {
+    protected function pScalar_MagicConst_Dir(MagicConst\Dir $node) {
         return '__DIR__';
     }
 
-    public function pScalar_MagicConst_File(MagicConst\File $node) {
+    protected function pScalar_MagicConst_File(MagicConst\File $node) {
         return '__FILE__';
     }
 
-    public function pScalar_MagicConst_Function(MagicConst\Function_ $node) {
+    protected function pScalar_MagicConst_Function(MagicConst\Function_ $node) {
         return '__FUNCTION__';
     }
 
-    public function pScalar_MagicConst_Line(MagicConst\Line $node) {
+    protected function pScalar_MagicConst_Line(MagicConst\Line $node) {
         return '__LINE__';
     }
 
-    public function pScalar_MagicConst_Method(MagicConst\Method $node) {
+    protected function pScalar_MagicConst_Method(MagicConst\Method $node) {
         return '__METHOD__';
     }
 
-    public function pScalar_MagicConst_Namespace(MagicConst\Namespace_ $node) {
+    protected function pScalar_MagicConst_Namespace(MagicConst\Namespace_ $node) {
         return '__NAMESPACE__';
     }
 
-    public function pScalar_MagicConst_Trait(MagicConst\Trait_ $node) {
+    protected function pScalar_MagicConst_Trait(MagicConst\Trait_ $node) {
         return '__TRAIT__';
     }
 
     // Scalars
 
-    public function pScalar_String(Scalar\String_ $node) {
+    protected function pScalar_String(Scalar\String_ $node) {
-        return '\'' . $this->pNoIndent(addcslashes($node->value, '\'\\')) . '\'';
+        $kind = $node->getAttribute('kind', Scalar\String_::KIND_SINGLE_QUOTED);
+        switch ($kind) {
+            case Scalar\String_::KIND_NOWDOC:
+                $label = $node->getAttribute('docLabel');
+                if ($label && !$this->containsEndLabel($node->value, $label)) {
+                    if ($node->value === '') {
+                        return $this->pNoIndent("<<<'$label'\n$label") . $this->docStringEndToken;
+                    }
+
+                    return $this->pNoIndent("<<<'$label'\n$node->value\n$label")
+                         . $this->docStringEndToken;
+                }
+                /* break missing intentionally */
+            case Scalar\String_::KIND_SINGLE_QUOTED:
+                return '\'' . $this->pNoIndent(addcslashes($node->value, '\'\\')) . '\'';
+            case Scalar\String_::KIND_HEREDOC:
+                $label = $node->getAttribute('docLabel');
+                if ($label && !$this->containsEndLabel($node->value, $label)) {
+                    if ($node->value === '') {
+                        return $this->pNoIndent("<<<$label\n$label") . $this->docStringEndToken;
+                    }
+
+                    $escaped = $this->escapeString($node->value, null);
+                    return $this->pNoIndent("<<<$label\n" . $escaped ."\n$label")
+                         . $this->docStringEndToken;
+                }
+            /* break missing intentionally */
+            case Scalar\String_::KIND_DOUBLE_QUOTED:
+                return '"' . $this->escapeString($node->value, '"') . '"';
+        }
+        throw new \Exception('Invalid string kind');
     }
 
-    public function pScalar_Encapsed(Scalar\Encapsed $node) {
+    protected function pScalar_Encapsed(Scalar\Encapsed $node) {
+        if ($node->getAttribute('kind') === Scalar\String_::KIND_HEREDOC) {
+            $label = $node->getAttribute('docLabel');
+            if ($label && !$this->encapsedContainsEndLabel($node->parts, $label)) {
+                if (count($node->parts) === 1
+                    && $node->parts[0] instanceof Scalar\EncapsedStringPart
+                    && $node->parts[0]->value === ''
+                ) {
+                    return $this->pNoIndent("<<<$label\n$label") . $this->docStringEndToken;
+                }
+
+                return $this->pNoIndent(
+                    "<<<$label\n" . $this->pEncapsList($node->parts, null) . "\n$label"
+                ) . $this->docStringEndToken;
+            }
+        }
         return '"' . $this->pEncapsList($node->parts, '"') . '"';
     }
 
-    public function pScalar_LNumber(Scalar\LNumber $node) {
+    protected function pScalar_LNumber(Scalar\LNumber $node) {
-        return (string) $node->value;
+        $str = (string) $node->value;
+        switch ($node->getAttribute('kind', Scalar\LNumber::KIND_DEC)) {
+            case Scalar\LNumber::KIND_BIN:
+                return '0b' . base_convert($str, 10, 2);
+            case Scalar\LNumber::KIND_OCT:
+                return '0' . base_convert($str, 10, 8);
+            case Scalar\LNumber::KIND_DEC:
+                return $str;
+            case Scalar\LNumber::KIND_HEX:
+                return '0x' . base_convert($str, 10, 16);
+        }
+        throw new \Exception('Invalid number kind');
     }
 
-    public function pScalar_DNumber(Scalar\DNumber $node) {
+    protected function pScalar_DNumber(Scalar\DNumber $node) {
         $stringValue = sprintf('%.16G', $node->value);
         if ($node->value !== (double) $stringValue) {
             $stringValue = sprintf('%.17G', $node->value);
@@ -107,265 +167,265 @@ class Standard extends PrettyPrinterAbstract
 
     // Assignments
 
-    public function pExpr_Assign(Expr\Assign $node) {
+    protected function pExpr_Assign(Expr\Assign $node) {
         return $this->pInfixOp('Expr_Assign', $node->var, ' = ', $node->expr);
     }
 
-    public function pExpr_AssignRef(Expr\AssignRef $node) {
+    protected function pExpr_AssignRef(Expr\AssignRef $node) {
         return $this->pInfixOp('Expr_AssignRef', $node->var, ' =& ', $node->expr);
     }
 
-    public function pExpr_AssignOp_Plus(AssignOp\Plus $node) {
+    protected function pExpr_AssignOp_Plus(AssignOp\Plus $node) {
         return $this->pInfixOp('Expr_AssignOp_Plus', $node->var, ' += ', $node->expr);
     }
 
-    public function pExpr_AssignOp_Minus(AssignOp\Minus $node) {
+    protected function pExpr_AssignOp_Minus(AssignOp\Minus $node) {
         return $this->pInfixOp('Expr_AssignOp_Minus', $node->var, ' -= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_Mul(AssignOp\Mul $node) {
+    protected function pExpr_AssignOp_Mul(AssignOp\Mul $node) {
         return $this->pInfixOp('Expr_AssignOp_Mul', $node->var, ' *= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_Div(AssignOp\Div $node) {
+    protected function pExpr_AssignOp_Div(AssignOp\Div $node) {
         return $this->pInfixOp('Expr_AssignOp_Div', $node->var, ' /= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_Concat(AssignOp\Concat $node) {
+    protected function pExpr_AssignOp_Concat(AssignOp\Concat $node) {
         return $this->pInfixOp('Expr_AssignOp_Concat', $node->var, ' .= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_Mod(AssignOp\Mod $node) {
+    protected function pExpr_AssignOp_Mod(AssignOp\Mod $node) {
         return $this->pInfixOp('Expr_AssignOp_Mod', $node->var, ' %= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_BitwiseAnd(AssignOp\BitwiseAnd $node) {
+    protected function pExpr_AssignOp_BitwiseAnd(AssignOp\BitwiseAnd $node) {
         return $this->pInfixOp('Expr_AssignOp_BitwiseAnd', $node->var, ' &= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_BitwiseOr(AssignOp\BitwiseOr $node) {
+    protected function pExpr_AssignOp_BitwiseOr(AssignOp\BitwiseOr $node) {
         return $this->pInfixOp('Expr_AssignOp_BitwiseOr', $node->var, ' |= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_BitwiseXor(AssignOp\BitwiseXor $node) {
+    protected function pExpr_AssignOp_BitwiseXor(AssignOp\BitwiseXor $node) {
         return $this->pInfixOp('Expr_AssignOp_BitwiseXor', $node->var, ' ^= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_ShiftLeft(AssignOp\ShiftLeft $node) {
+    protected function pExpr_AssignOp_ShiftLeft(AssignOp\ShiftLeft $node) {
         return $this->pInfixOp('Expr_AssignOp_ShiftLeft', $node->var, ' <<= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_ShiftRight(AssignOp\ShiftRight $node) {
+    protected function pExpr_AssignOp_ShiftRight(AssignOp\ShiftRight $node) {
         return $this->pInfixOp('Expr_AssignOp_ShiftRight', $node->var, ' >>= ', $node->expr);
     }
 
-    public function pExpr_AssignOp_Pow(AssignOp\Pow $node) {
+    protected function pExpr_AssignOp_Pow(AssignOp\Pow $node) {
         return $this->pInfixOp('Expr_AssignOp_Pow', $node->var, ' **= ', $node->expr);
     }
 
     // Binary expressions
 
-    public function pExpr_BinaryOp_Plus(BinaryOp\Plus $node) {
+    protected function pExpr_BinaryOp_Plus(BinaryOp\Plus $node) {
         return $this->pInfixOp('Expr_BinaryOp_Plus', $node->left, ' + ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Minus(BinaryOp\Minus $node) {
+    protected function pExpr_BinaryOp_Minus(BinaryOp\Minus $node) {
         return $this->pInfixOp('Expr_BinaryOp_Minus', $node->left, ' - ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Mul(BinaryOp\Mul $node) {
+    protected function pExpr_BinaryOp_Mul(BinaryOp\Mul $node) {
         return $this->pInfixOp('Expr_BinaryOp_Mul', $node->left, ' * ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Div(BinaryOp\Div $node) {
+    protected function pExpr_BinaryOp_Div(BinaryOp\Div $node) {
         return $this->pInfixOp('Expr_BinaryOp_Div', $node->left, ' / ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Concat(BinaryOp\Concat $node) {
+    protected function pExpr_BinaryOp_Concat(BinaryOp\Concat $node) {
         return $this->pInfixOp('Expr_BinaryOp_Concat', $node->left, ' . ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Mod(BinaryOp\Mod $node) {
+    protected function pExpr_BinaryOp_Mod(BinaryOp\Mod $node) {
         return $this->pInfixOp('Expr_BinaryOp_Mod', $node->left, ' % ', $node->right);
     }
 
-    public function pExpr_BinaryOp_BooleanAnd(BinaryOp\BooleanAnd $node) {
+    protected function pExpr_BinaryOp_BooleanAnd(BinaryOp\BooleanAnd $node) {
         return $this->pInfixOp('Expr_BinaryOp_BooleanAnd', $node->left, ' && ', $node->right);
     }
 
-    public function pExpr_BinaryOp_BooleanOr(BinaryOp\BooleanOr $node) {
+    protected function pExpr_BinaryOp_BooleanOr(BinaryOp\BooleanOr $node) {
         return $this->pInfixOp('Expr_BinaryOp_BooleanOr', $node->left, ' || ', $node->right);
     }
 
-    public function pExpr_BinaryOp_BitwiseAnd(BinaryOp\BitwiseAnd $node) {
+    protected function pExpr_BinaryOp_BitwiseAnd(BinaryOp\BitwiseAnd $node) {
         return $this->pInfixOp('Expr_BinaryOp_BitwiseAnd', $node->left, ' & ', $node->right);
     }
 
-    public function pExpr_BinaryOp_BitwiseOr(BinaryOp\BitwiseOr $node) {
+    protected function pExpr_BinaryOp_BitwiseOr(BinaryOp\BitwiseOr $node) {
         return $this->pInfixOp('Expr_BinaryOp_BitwiseOr', $node->left, ' | ', $node->right);
     }
 
-    public function pExpr_BinaryOp_BitwiseXor(BinaryOp\BitwiseXor $node) {
+    protected function pExpr_BinaryOp_BitwiseXor(BinaryOp\BitwiseXor $node) {
         return $this->pInfixOp('Expr_BinaryOp_BitwiseXor', $node->left, ' ^ ', $node->right);
     }
 
-    public function pExpr_BinaryOp_ShiftLeft(BinaryOp\ShiftLeft $node) {
+    protected function pExpr_BinaryOp_ShiftLeft(BinaryOp\ShiftLeft $node) {
         return $this->pInfixOp('Expr_BinaryOp_ShiftLeft', $node->left, ' << ', $node->right);
     }
 
-    public function pExpr_BinaryOp_ShiftRight(BinaryOp\ShiftRight $node) {
+    protected function pExpr_BinaryOp_ShiftRight(BinaryOp\ShiftRight $node) {
         return $this->pInfixOp('Expr_BinaryOp_ShiftRight', $node->left, ' >> ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Pow(BinaryOp\Pow $node) {
+    protected function pExpr_BinaryOp_Pow(BinaryOp\Pow $node) {
         return $this->pInfixOp('Expr_BinaryOp_Pow', $node->left, ' ** ', $node->right);
     }
 
-    public function pExpr_BinaryOp_LogicalAnd(BinaryOp\LogicalAnd $node) {
+    protected function pExpr_BinaryOp_LogicalAnd(BinaryOp\LogicalAnd $node) {
         return $this->pInfixOp('Expr_BinaryOp_LogicalAnd', $node->left, ' and ', $node->right);
     }
 
-    public function pExpr_BinaryOp_LogicalOr(BinaryOp\LogicalOr $node) {
+    protected function pExpr_BinaryOp_LogicalOr(BinaryOp\LogicalOr $node) {
         return $this->pInfixOp('Expr_BinaryOp_LogicalOr', $node->left, ' or ', $node->right);
     }
 
-    public function pExpr_BinaryOp_LogicalXor(BinaryOp\LogicalXor $node) {
+    protected function pExpr_BinaryOp_LogicalXor(BinaryOp\LogicalXor $node) {
         return $this->pInfixOp('Expr_BinaryOp_LogicalXor', $node->left, ' xor ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Equal(BinaryOp\Equal $node) {
+    protected function pExpr_BinaryOp_Equal(BinaryOp\Equal $node) {
         return $this->pInfixOp('Expr_BinaryOp_Equal', $node->left, ' == ', $node->right);
     }
 
-    public function pExpr_BinaryOp_NotEqual(BinaryOp\NotEqual $node) {
+    protected function pExpr_BinaryOp_NotEqual(BinaryOp\NotEqual $node) {
         return $this->pInfixOp('Expr_BinaryOp_NotEqual', $node->left, ' != ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Identical(BinaryOp\Identical $node) {
+    protected function pExpr_BinaryOp_Identical(BinaryOp\Identical $node) {
         return $this->pInfixOp('Expr_BinaryOp_Identical', $node->left, ' === ', $node->right);
     }
 
-    public function pExpr_BinaryOp_NotIdentical(BinaryOp\NotIdentical $node) {
+    protected function pExpr_BinaryOp_NotIdentical(BinaryOp\NotIdentical $node) {
         return $this->pInfixOp('Expr_BinaryOp_NotIdentical', $node->left, ' !== ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Spaceship(BinaryOp\Spaceship $node) {
+    protected function pExpr_BinaryOp_Spaceship(BinaryOp\Spaceship $node) {
         return $this->pInfixOp('Expr_BinaryOp_Spaceship', $node->left, ' <=> ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Greater(BinaryOp\Greater $node) {
+    protected function pExpr_BinaryOp_Greater(BinaryOp\Greater $node) {
         return $this->pInfixOp('Expr_BinaryOp_Greater', $node->left, ' > ', $node->right);
     }
 
-    public function pExpr_BinaryOp_GreaterOrEqual(BinaryOp\GreaterOrEqual $node) {
+    protected function pExpr_BinaryOp_GreaterOrEqual(BinaryOp\GreaterOrEqual $node) {
         return $this->pInfixOp('Expr_BinaryOp_GreaterOrEqual', $node->left, ' >= ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Smaller(BinaryOp\Smaller $node) {
+    protected function pExpr_BinaryOp_Smaller(BinaryOp\Smaller $node) {
         return $this->pInfixOp('Expr_BinaryOp_Smaller', $node->left, ' < ', $node->right);
     }
 
-    public function pExpr_BinaryOp_SmallerOrEqual(BinaryOp\SmallerOrEqual $node) {
+    protected function pExpr_BinaryOp_SmallerOrEqual(BinaryOp\SmallerOrEqual $node) {
         return $this->pInfixOp('Expr_BinaryOp_SmallerOrEqual', $node->left, ' <= ', $node->right);
     }
 
-    public function pExpr_BinaryOp_Coalesce(BinaryOp\Coalesce $node) {
+    protected function pExpr_BinaryOp_Coalesce(BinaryOp\Coalesce $node) {
         return $this->pInfixOp('Expr_BinaryOp_Coalesce', $node->left, ' ?? ', $node->right);
     }
 
-    public function pExpr_Instanceof(Expr\Instanceof_ $node) {
+    protected function pExpr_Instanceof(Expr\Instanceof_ $node) {
         return $this->pInfixOp('Expr_Instanceof', $node->expr, ' instanceof ', $node->class);
     }
 
     // Unary expressions
 
-    public function pExpr_BooleanNot(Expr\BooleanNot $node) {
+    protected function pExpr_BooleanNot(Expr\BooleanNot $node) {
         return $this->pPrefixOp('Expr_BooleanNot', '!', $node->expr);
     }
 
-    public function pExpr_BitwiseNot(Expr\BitwiseNot $node) {
+    protected function pExpr_BitwiseNot(Expr\BitwiseNot $node) {
         return $this->pPrefixOp('Expr_BitwiseNot', '~', $node->expr);
     }
 
-    public function pExpr_UnaryMinus(Expr\UnaryMinus $node) {
+    protected function pExpr_UnaryMinus(Expr\UnaryMinus $node) {
         return $this->pPrefixOp('Expr_UnaryMinus', '-', $node->expr);
     }
 
-    public function pExpr_UnaryPlus(Expr\UnaryPlus $node) {
+    protected function pExpr_UnaryPlus(Expr\UnaryPlus $node) {
         return $this->pPrefixOp('Expr_UnaryPlus', '+', $node->expr);
     }
 
-    public function pExpr_PreInc(Expr\PreInc $node) {
+    protected function pExpr_PreInc(Expr\PreInc $node) {
         return $this->pPrefixOp('Expr_PreInc', '++', $node->var);
     }
 
-    public function pExpr_PreDec(Expr\PreDec $node) {
+    protected function pExpr_PreDec(Expr\PreDec $node) {
         return $this->pPrefixOp('Expr_PreDec', '--', $node->var);
     }
 
-    public function pExpr_PostInc(Expr\PostInc $node) {
+    protected function pExpr_PostInc(Expr\PostInc $node) {
         return $this->pPostfixOp('Expr_PostInc', $node->var, '++');
     }
 
-    public function pExpr_PostDec(Expr\PostDec $node) {
+    protected function pExpr_PostDec(Expr\PostDec $node) {
         return $this->pPostfixOp('Expr_PostDec', $node->var, '--');
     }
 
-    public function pExpr_ErrorSuppress(Expr\ErrorSuppress $node) {
+    protected function pExpr_ErrorSuppress(Expr\ErrorSuppress $node) {
         return $this->pPrefixOp('Expr_ErrorSuppress', '@', $node->expr);
     }
 
-    public function pExpr_YieldFrom(Expr\YieldFrom $node) {
+    protected function pExpr_YieldFrom(Expr\YieldFrom $node) {
         return $this->pPrefixOp('Expr_YieldFrom', 'yield from ', $node->expr);
     }
 
-    public function pExpr_Print(Expr\Print_ $node) {
+    protected function pExpr_Print(Expr\Print_ $node) {
         return $this->pPrefixOp('Expr_Print', 'print ', $node->expr);
     }
 
     // Casts
 
-    public function pExpr_Cast_Int(Cast\Int_ $node) {
+    protected function pExpr_Cast_Int(Cast\Int_ $node) {
         return $this->pPrefixOp('Expr_Cast_Int', '(int) ', $node->expr);
     }
 
-    public function pExpr_Cast_Double(Cast\Double $node) {
+    protected function pExpr_Cast_Double(Cast\Double $node) {
         return $this->pPrefixOp('Expr_Cast_Double', '(double) ', $node->expr);
     }
 
-    public function pExpr_Cast_String(Cast\String_ $node) {
+    protected function pExpr_Cast_String(Cast\String_ $node) {
         return $this->pPrefixOp('Expr_Cast_String', '(string) ', $node->expr);
     }
 
-    public function pExpr_Cast_Array(Cast\Array_ $node) {
+    protected function pExpr_Cast_Array(Cast\Array_ $node) {
         return $this->pPrefixOp('Expr_Cast_Array', '(array) ', $node->expr);
     }
 
-    public function pExpr_Cast_Object(Cast\Object_ $node) {
+    protected function pExpr_Cast_Object(Cast\Object_ $node) {
         return $this->pPrefixOp('Expr_Cast_Object', '(object) ', $node->expr);
     }
 
-    public function pExpr_Cast_Bool(Cast\Bool_ $node) {
+    protected function pExpr_Cast_Bool(Cast\Bool_ $node) {
         return $this->pPrefixOp('Expr_Cast_Bool', '(bool) ', $node->expr);
     }
 
-    public function pExpr_Cast_Unset(Cast\Unset_ $node) {
+    protected function pExpr_Cast_Unset(Cast\Unset_ $node) {
         return $this->pPrefixOp('Expr_Cast_Unset', '(unset) ', $node->expr);
     }
 
     // Function calls and similar constructs
 
-    public function pExpr_FuncCall(Expr\FuncCall $node) {
+    protected function pExpr_FuncCall(Expr\FuncCall $node) {
         return $this->pCallLhs($node->name)
              . '(' . $this->pCommaSeparated($node->args) . ')';
     }
 
-    public function pExpr_MethodCall(Expr\MethodCall $node) {
+    protected function pExpr_MethodCall(Expr\MethodCall $node) {
         return $this->pDereferenceLhs($node->var) . '->' . $this->pObjectProperty($node->name)
              . '(' . $this->pCommaSeparated($node->args) . ')';
     }
 
-    public function pExpr_StaticCall(Expr\StaticCall $node) {
+    protected function pExpr_StaticCall(Expr\StaticCall $node) {
         return $this->pDereferenceLhs($node->class) . '::'
              . ($node->name instanceof Expr
                 ? ($node->name instanceof Expr\Variable
@@ -375,19 +435,19 @@ class Standard extends PrettyPrinterAbstract
              . '(' . $this->pCommaSeparated($node->args) . ')';
     }
 
-    public function pExpr_Empty(Expr\Empty_ $node) {
+    protected function pExpr_Empty(Expr\Empty_ $node) {
         return 'empty(' . $this->p($node->expr) . ')';
     }
 
-    public function pExpr_Isset(Expr\Isset_ $node) {
+    protected function pExpr_Isset(Expr\Isset_ $node) {
         return 'isset(' . $this->pCommaSeparated($node->vars) . ')';
     }
 
-    public function pExpr_Eval(Expr\Eval_ $node) {
+    protected function pExpr_Eval(Expr\Eval_ $node) {
         return 'eval(' . $this->p($node->expr) . ')';
     }
 
-    public function pExpr_Include(Expr\Include_ $node) {
+    protected function pExpr_Include(Expr\Include_ $node) {
         static $map = array(
             Expr\Include_::TYPE_INCLUDE      => 'include',
             Expr\Include_::TYPE_INCLUDE_ONCE => 'include_once',
@@ -398,22 +458,13 @@ class Standard extends PrettyPrinterAbstract
         return $map[$node->type] . ' ' . $this->p($node->expr);
     }
 
-    public function pExpr_List(Expr\List_ $node) {
+    protected function pExpr_List(Expr\List_ $node) {
-        $pList = array();
+        return 'list(' . $this->pCommaSeparated($node->items) . ')';
-        foreach ($node->vars as $var) {
-            if (null === $var) {
-                $pList[] = '';
-            } else {
-                $pList[] = $this->p($var);
-            }
-        }
-
-        return 'list(' . implode(', ', $pList) . ')';
     }
 
     // Other
 
-    public function pExpr_Variable(Expr\Variable $node) {
+    protected function pExpr_Variable(Expr\Variable $node) {
         if ($node->name instanceof Expr) {
             return '${' . $this->p($node->name) . '}';
         } else {
@@ -421,45 +472,47 @@ class Standard extends PrettyPrinterAbstract
         }
     }
 
-    public function pExpr_Array(Expr\Array_ $node) {
+    protected function pExpr_Array(Expr\Array_ $node) {
-        if ($this->options['shortArraySyntax']) {
+        $syntax = $node->getAttribute('kind',
+            $this->options['shortArraySyntax'] ? Expr\Array_::KIND_SHORT : Expr\Array_::KIND_LONG);
+        if ($syntax === Expr\Array_::KIND_SHORT) {
             return '[' . $this->pCommaSeparated($node->items) . ']';
         } else {
             return 'array(' . $this->pCommaSeparated($node->items) . ')';
         }
     }
 
-    public function pExpr_ArrayItem(Expr\ArrayItem $node) {
+    protected function pExpr_ArrayItem(Expr\ArrayItem $node) {
         return (null !== $node->key ? $this->p($node->key) . ' => ' : '')
              . ($node->byRef ? '&' : '') . $this->p($node->value);
     }
 
-    public function pExpr_ArrayDimFetch(Expr\ArrayDimFetch $node) {
+    protected function pExpr_ArrayDimFetch(Expr\ArrayDimFetch $node) {
         return $this->pDereferenceLhs($node->var)
              . '[' . (null !== $node->dim ? $this->p($node->dim) : '') . ']';
     }
 
-    public function pExpr_ConstFetch(Expr\ConstFetch $node) {
+    protected function pExpr_ConstFetch(Expr\ConstFetch $node) {
         return $this->p($node->name);
     }
 
-    public function pExpr_ClassConstFetch(Expr\ClassConstFetch $node) {
+    protected function pExpr_ClassConstFetch(Expr\ClassConstFetch $node) {
         return $this->p($node->class) . '::' . $node->name;
     }
 
-    public function pExpr_PropertyFetch(Expr\PropertyFetch $node) {
+    protected function pExpr_PropertyFetch(Expr\PropertyFetch $node) {
         return $this->pDereferenceLhs($node->var) . '->' . $this->pObjectProperty($node->name);
     }
 
-    public function pExpr_StaticPropertyFetch(Expr\StaticPropertyFetch $node) {
+    protected function pExpr_StaticPropertyFetch(Expr\StaticPropertyFetch $node) {
         return $this->pDereferenceLhs($node->class) . '::$' . $this->pObjectProperty($node->name);
     }
 
-    public function pExpr_ShellExec(Expr\ShellExec $node) {
+    protected function pExpr_ShellExec(Expr\ShellExec $node) {
         return '`' . $this->pEncapsList($node->parts, '`') . '`';
     }
 
-    public function pExpr_Closure(Expr\Closure $node) {
+    protected function pExpr_Closure(Expr\Closure $node) {
         return ($node->static ? 'static ' : '')
              . 'function ' . ($node->byRef ? '&' : '')
              . '(' . $this->pCommaSeparated($node->params) . ')'
@@ -468,11 +521,11 @@ class Standard extends PrettyPrinterAbstract
              . ' {' . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pExpr_ClosureUse(Expr\ClosureUse $node) {
+    protected function pExpr_ClosureUse(Expr\ClosureUse $node) {
         return ($node->byRef ? '&' : '') . '$' . $node->var;
     }
 
-    public function pExpr_New(Expr\New_ $node) {
+    protected function pExpr_New(Expr\New_ $node) {
         if ($node->class instanceof Stmt\Class_) {
             $args = $node->args ? '(' . $this->pCommaSeparated($node->args) . ')' : '';
             return 'new ' . $this->pClassCommon($node->class, $args);
@@ -480,11 +533,11 @@ class Standard extends PrettyPrinterAbstract
         return 'new ' . $this->p($node->class) . '(' . $this->pCommaSeparated($node->args) . ')';
     }
 
-    public function pExpr_Clone(Expr\Clone_ $node) {
+    protected function pExpr_Clone(Expr\Clone_ $node) {
         return 'clone ' . $this->p($node->expr);
     }
 
-    public function pExpr_Ternary(Expr\Ternary $node) {
+    protected function pExpr_Ternary(Expr\Ternary $node) {
         // a bit of cheating: we treat the ternary as a binary op where the ?...: part is the operator.
         // this is okay because the part between ? and : never needs parentheses.
         return $this->pInfixOp('Expr_Ternary',
@@ -492,11 +545,13 @@ class Standard extends PrettyPrinterAbstract
         );
     }
 
-    public function pExpr_Exit(Expr\Exit_ $node) {
+    protected function pExpr_Exit(Expr\Exit_ $node) {
-        return 'die' . (null !== $node->expr ? '(' . $this->p($node->expr) . ')' : '');
+        $kind = $node->getAttribute('kind', Expr\Exit_::KIND_DIE);
+        return ($kind === Expr\Exit_::KIND_EXIT ? 'exit' : 'die')
+             . (null !== $node->expr ? '(' . $this->p($node->expr) . ')' : '');
     }
 
-    public function pExpr_Yield(Expr\Yield_ $node) {
+    protected function pExpr_Yield(Expr\Yield_ $node) {
         if ($node->value === null) {
             return 'yield';
         } else {
@@ -510,7 +565,7 @@ class Standard extends PrettyPrinterAbstract
 
     // Declarations
 
-    public function pStmt_Namespace(Stmt\Namespace_ $node) {
+    protected function pStmt_Namespace(Stmt\Namespace_ $node) {
         if ($this->canUseSemicolonNamespaces) {
             return 'namespace ' . $this->p($node->name) . ';' . "\n" . $this->pStmts($node->stmts, false);
         } else {
@@ -519,17 +574,17 @@ class Standard extends PrettyPrinterAbstract
         }
     }
 
-    public function pStmt_Use(Stmt\Use_ $node) {
+    protected function pStmt_Use(Stmt\Use_ $node) {
         return 'use ' . $this->pUseType($node->type)
              . $this->pCommaSeparated($node->uses) . ';';
     }
 
-    public function pStmt_GroupUse(Stmt\GroupUse $node) {
+    protected function pStmt_GroupUse(Stmt\GroupUse $node) {
         return 'use ' . $this->pUseType($node->type) . $this->pName($node->prefix)
              . '\{' . $this->pCommaSeparated($node->uses) . '};';
     }
 
-    public function pStmt_UseUse(Stmt\UseUse $node) {
+    protected function pStmt_UseUse(Stmt\UseUse $node) {
         return $this->pUseType($node->type) . $this->p($node->name)
              . ($node->name->getLast() !== $node->alias ? ' as ' . $node->alias : '');
     }
@@ -539,34 +594,34 @@ class Standard extends PrettyPrinterAbstract
             : ($type === Stmt\Use_::TYPE_CONSTANT ? 'const ' : '');
     }
 
-    public function pStmt_Interface(Stmt\Interface_ $node) {
+    protected function pStmt_Interface(Stmt\Interface_ $node) {
         return 'interface ' . $node->name
              . (!empty($node->extends) ? ' extends ' . $this->pCommaSeparated($node->extends) : '')
              . "\n" . '{' . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_Class(Stmt\Class_ $node) {
+    protected function pStmt_Class(Stmt\Class_ $node) {
         return $this->pClassCommon($node, ' ' . $node->name);
     }
 
-    public function pStmt_Trait(Stmt\Trait_ $node) {
+    protected function pStmt_Trait(Stmt\Trait_ $node) {
         return 'trait ' . $node->name
              . "\n" . '{' . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_TraitUse(Stmt\TraitUse $node) {
+    protected function pStmt_TraitUse(Stmt\TraitUse $node) {
         return 'use ' . $this->pCommaSeparated($node->traits)
              . (empty($node->adaptations)
                 ? ';'
                 : ' {' . $this->pStmts($node->adaptations) . "\n" . '}');
     }
 
-    public function pStmt_TraitUseAdaptation_Precedence(Stmt\TraitUseAdaptation\Precedence $node) {
+    protected function pStmt_TraitUseAdaptation_Precedence(Stmt\TraitUseAdaptation\Precedence $node) {
         return $this->p($node->trait) . '::' . $node->method
              . ' insteadof ' . $this->pCommaSeparated($node->insteadof) . ';';
     }
 
-    public function pStmt_TraitUseAdaptation_Alias(Stmt\TraitUseAdaptation\Alias $node) {
+    protected function pStmt_TraitUseAdaptation_Alias(Stmt\TraitUseAdaptation\Alias $node) {
         return (null !== $node->trait ? $this->p($node->trait) . '::' : '')
              . $node->method . ' as'
              . (null !== $node->newModifier ? ' ' . rtrim($this->pModifiers($node->newModifier), ' ') : '')
@@ -574,17 +629,17 @@ class Standard extends PrettyPrinterAbstract
              . ';';
     }
 
-    public function pStmt_Property(Stmt\Property $node) {
+    protected function pStmt_Property(Stmt\Property $node) {
-        return (0 === $node->type ? 'var ' : $this->pModifiers($node->type)) . $this->pCommaSeparated($node->props) . ';';
+        return (0 === $node->flags ? 'var ' : $this->pModifiers($node->flags)) . $this->pCommaSeparated($node->props) . ';';
     }
 
-    public function pStmt_PropertyProperty(Stmt\PropertyProperty $node) {
+    protected function pStmt_PropertyProperty(Stmt\PropertyProperty $node) {
         return '$' . $node->name
              . (null !== $node->default ? ' = ' . $this->p($node->default) : '');
     }
 
-    public function pStmt_ClassMethod(Stmt\ClassMethod $node) {
+    protected function pStmt_ClassMethod(Stmt\ClassMethod $node) {
-        return $this->pModifiers($node->type)
+        return $this->pModifiers($node->flags)
              . 'function ' . ($node->byRef ? '&' : '') . $node->name
              . '(' . $this->pCommaSeparated($node->params) . ')'
              . (null !== $node->returnType ? ' : ' . $this->pType($node->returnType) : '')
@@ -593,49 +648,50 @@ class Standard extends PrettyPrinterAbstract
                 : ';');
     }
 
-    public function pStmt_ClassConst(Stmt\ClassConst $node) {
+    protected function pStmt_ClassConst(Stmt\ClassConst $node) {
-        return 'const ' . $this->pCommaSeparated($node->consts) . ';';
+        return $this->pModifiers($node->flags)
+             . 'const ' . $this->pCommaSeparated($node->consts) . ';';
     }
 
-    public function pStmt_Function(Stmt\Function_ $node) {
+    protected function pStmt_Function(Stmt\Function_ $node) {
         return 'function ' . ($node->byRef ? '&' : '') . $node->name
              . '(' . $this->pCommaSeparated($node->params) . ')'
              . (null !== $node->returnType ? ' : ' . $this->pType($node->returnType) : '')
              . "\n" . '{' . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_Const(Stmt\Const_ $node) {
+    protected function pStmt_Const(Stmt\Const_ $node) {
         return 'const ' . $this->pCommaSeparated($node->consts) . ';';
     }
 
-    public function pStmt_Declare(Stmt\Declare_ $node) {
+    protected function pStmt_Declare(Stmt\Declare_ $node) {
         return 'declare (' . $this->pCommaSeparated($node->declares) . ')'
              . (null !== $node->stmts ? ' {' . $this->pStmts($node->stmts) . "\n" . '}' : ';');
     }
 
-    public function pStmt_DeclareDeclare(Stmt\DeclareDeclare $node) {
+    protected function pStmt_DeclareDeclare(Stmt\DeclareDeclare $node) {
         return $node->key . '=' . $this->p($node->value);
     }
 
     // Control flow
 
-    public function pStmt_If(Stmt\If_ $node) {
+    protected function pStmt_If(Stmt\If_ $node) {
         return 'if (' . $this->p($node->cond) . ') {'
              . $this->pStmts($node->stmts) . "\n" . '}'
              . $this->pImplode($node->elseifs)
              . (null !== $node->else ? $this->p($node->else) : '');
     }
 
-    public function pStmt_ElseIf(Stmt\ElseIf_ $node) {
+    protected function pStmt_ElseIf(Stmt\ElseIf_ $node) {
         return ' elseif (' . $this->p($node->cond) . ') {'
              . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_Else(Stmt\Else_ $node) {
+    protected function pStmt_Else(Stmt\Else_ $node) {
         return ' else {' . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_For(Stmt\For_ $node) {
+    protected function pStmt_For(Stmt\For_ $node) {
         return 'for ('
              . $this->pCommaSeparated($node->init) . ';' . (!empty($node->cond) ? ' ' : '')
              . $this->pCommaSeparated($node->cond) . ';' . (!empty($node->loop) ? ' ' : '')
@@ -643,101 +699,108 @@ class Standard extends PrettyPrinterAbstract
              . ') {' . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_Foreach(Stmt\Foreach_ $node) {
+    protected function pStmt_Foreach(Stmt\Foreach_ $node) {
         return 'foreach (' . $this->p($node->expr) . ' as '
              . (null !== $node->keyVar ? $this->p($node->keyVar) . ' => ' : '')
              . ($node->byRef ? '&' : '') . $this->p($node->valueVar) . ') {'
              . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_While(Stmt\While_ $node) {
+    protected function pStmt_While(Stmt\While_ $node) {
         return 'while (' . $this->p($node->cond) . ') {'
              . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_Do(Stmt\Do_ $node) {
+    protected function pStmt_Do(Stmt\Do_ $node) {
         return 'do {' . $this->pStmts($node->stmts) . "\n"
              . '} while (' . $this->p($node->cond) . ');';
     }
 
-    public function pStmt_Switch(Stmt\Switch_ $node) {
+    protected function pStmt_Switch(Stmt\Switch_ $node) {
         return 'switch (' . $this->p($node->cond) . ') {'
              . $this->pStmts($node->cases) . "\n" . '}';
     }
 
-    public function pStmt_Case(Stmt\Case_ $node) {
+    protected function pStmt_Case(Stmt\Case_ $node) {
         return (null !== $node->cond ? 'case ' . $this->p($node->cond) : 'default') . ':'
              . $this->pStmts($node->stmts);
     }
 
-    public function pStmt_TryCatch(Stmt\TryCatch $node) {
+    protected function pStmt_TryCatch(Stmt\TryCatch $node) {
         return 'try {' . $this->pStmts($node->stmts) . "\n" . '}'
              . $this->pImplode($node->catches)
-             . ($node->finallyStmts !== null
+             . ($node->finally !== null ? $this->p($node->finally) : '');
-                ? ' finally {' . $this->pStmts($node->finallyStmts) . "\n" . '}'
-                : '');
     }
 
-    public function pStmt_Catch(Stmt\Catch_ $node) {
+    protected function pStmt_Catch(Stmt\Catch_ $node) {
-        return ' catch (' . $this->p($node->type) . ' $' . $node->var . ') {'
+        return ' catch (' . $this->pImplode($node->types, '|') . ' $' . $node->var . ') {'
              . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_Break(Stmt\Break_ $node) {
+    protected function pStmt_Finally(Stmt\Finally_ $node) {
+        return ' finally {' . $this->pStmts($node->stmts) . "\n" . '}';
+    }
+
+    protected function pStmt_Break(Stmt\Break_ $node) {
         return 'break' . ($node->num !== null ? ' ' . $this->p($node->num) : '') . ';';
     }
 
-    public function pStmt_Continue(Stmt\Continue_ $node) {
+    protected function pStmt_Continue(Stmt\Continue_ $node) {
         return 'continue' . ($node->num !== null ? ' ' . $this->p($node->num) : '') . ';';
     }
 
-    public function pStmt_Return(Stmt\Return_ $node) {
+    protected function pStmt_Return(Stmt\Return_ $node) {
         return 'return' . (null !== $node->expr ? ' ' . $this->p($node->expr) : '') . ';';
     }
 
-    public function pStmt_Throw(Stmt\Throw_ $node) {
+    protected function pStmt_Throw(Stmt\Throw_ $node) {
         return 'throw ' . $this->p($node->expr) . ';';
     }
 
-    public function pStmt_Label(Stmt\Label $node) {
+    protected function pStmt_Label(Stmt\Label $node) {
         return $node->name . ':';
     }
 
-    public function pStmt_Goto(Stmt\Goto_ $node) {
+    protected function pStmt_Goto(Stmt\Goto_ $node) {
         return 'goto ' . $node->name . ';';
     }
 
     // Other
 
-    public function pStmt_Echo(Stmt\Echo_ $node) {
+    protected function pStmt_Echo(Stmt\Echo_ $node) {
         return 'echo ' . $this->pCommaSeparated($node->exprs) . ';';
     }
 
-    public function pStmt_Static(Stmt\Static_ $node) {
+    protected function pStmt_Static(Stmt\Static_ $node) {
         return 'static ' . $this->pCommaSeparated($node->vars) . ';';
     }
 
-    public function pStmt_Global(Stmt\Global_ $node) {
+    protected function pStmt_Global(Stmt\Global_ $node) {
         return 'global ' . $this->pCommaSeparated($node->vars) . ';';
     }
 
-    public function pStmt_StaticVar(Stmt\StaticVar $node) {
+    protected function pStmt_StaticVar(Stmt\StaticVar $node) {
         return '$' . $node->name
              . (null !== $node->default ? ' = ' . $this->p($node->default) : '');
     }
 
-    public function pStmt_Unset(Stmt\Unset_ $node) {
+    protected function pStmt_Unset(Stmt\Unset_ $node) {
         return 'unset(' . $this->pCommaSeparated($node->vars) . ');';
     }
 
-    public function pStmt_InlineHTML(Stmt\InlineHTML $node) {
+    protected function pStmt_InlineHTML(Stmt\InlineHTML $node) {
-        return '?>' . $this->pNoIndent("\n" . $node->value) . '<?php ';
+        $newline = $node->getAttribute('hasLeadingNewline', true) ? "\n" : '';
+        return '?>' . $this->pNoIndent($newline . $node->value) . '<?php ';
     }
 
-    public function pStmt_HaltCompiler(Stmt\HaltCompiler $node) {
+    protected function pStmt_HaltCompiler(Stmt\HaltCompiler $node) {
         return '__halt_compiler();' . $node->remaining;
     }
 
+    protected function pStmt_Nop(Stmt\Nop $node) {
+        return '';
+    }
+
     // Helpers
 
     protected function pType($node) {
@@ -745,7 +808,7 @@ class Standard extends PrettyPrinterAbstract
     }
 
     protected function pClassCommon(Stmt\Class_ $node, $afterClassToken) {
-        return $this->pModifiers($node->type)
+        return $this->pModifiers($node->flags)
         . 'class' . $afterClassToken
         . (null !== $node->extends ? ' extends ' . $this->p($node->extends) : '')
         . (!empty($node->implements) ? ' implements ' . $this->pCommaSeparated($node->implements) : '')
@@ -773,7 +836,7 @@ class Standard extends PrettyPrinterAbstract
         $return = '';
         foreach ($encapsList as $element) {
             if ($element instanceof Scalar\EncapsedStringPart) {
-                $return .= addcslashes($element->value, "\n\r\t\f\v$" . $quote . "\\");
+                $return .= $this->escapeString($element->value, $quote);
             } else {
                 $return .= '{' . $this->p($element) . '}';
             }
@@ -782,6 +845,45 @@ class Standard extends PrettyPrinterAbstract
         return $return;
     }
 
+    protected function escapeString($string, $quote) {
+        if (null === $quote) {
+            // For doc strings, don't escape newlines
+            $escaped = addcslashes($string, "\t\f\v$\\");
+        } else {
+            $escaped = addcslashes($string, "\n\r\t\f\v$" . $quote . "\\");
+        }
+
+        // Escape other control characters
+        return preg_replace_callback('/([\0-\10\16-\37])(?=([0-7]?))/', function ($matches) {
+            $oct = decoct(ord($matches[1]));
+            if ($matches[2] !== '') {
+                // If there is a trailing digit, use the full three character form
+                return '\\' . str_pad($oct, 3, '0', STR_PAD_LEFT);
+            }
+            return '\\' . $oct;
+        }, $escaped);
+    }
+
+    protected function containsEndLabel($string, $label, $atStart = true, $atEnd = true) {
+        $start = $atStart ? '(?:^|[\r\n])' : '[\r\n]';
+        $end = $atEnd ? '(?:$|[;\r\n])' : '[;\r\n]';
+        return false !== strpos($string, $label)
+            && preg_match('/' . $start . $label . $end . '/', $string);
+    }
+
+    protected function encapsedContainsEndLabel(array $parts, $label) {
+        foreach ($parts as $i => $part) {
+            $atStart = $i === 0;
+            $atEnd = $i === count($parts) - 1;
+            if ($part instanceof Scalar\EncapsedStringPart
+                && $this->containsEndLabel($part->value, $label, $atStart, $atEnd)
+            ) {
+                return true;
+            }
+        }
+        return false;
+    }
+
     protected function pDereferenceLhs(Node $node) {
         if ($node instanceof Expr\Variable
             || $node instanceof Name

lib/PhpParser/PrettyPrinterAbstract.php

@@ -75,6 +75,7 @@ abstract class PrettyPrinterAbstract
     );
 
     protected $noIndentToken;
+    protected $docStringEndToken;
     protected $canUseSemicolonNamespaces;
     protected $options;
 
@@ -82,12 +83,14 @@ abstract class PrettyPrinterAbstract
      * Creates a pretty printer instance using the given options.
      *
      * Supported options:
-     *  * bool $shortArraySyntax = false: Whether to use [] instead of array()
+     *  * bool $shortArraySyntax = false: Whether to use [] instead of array() as the default array
+     *                                    syntax, if the node does not specify a format.
      *
      * @param array $options Dictionary of formatting options
      */
     public function __construct(array $options = []) {
         $this->noIndentToken = '_NO_INDENT_' . mt_rand();
+        $this->docStringEndToken = '_DOC_STRING_END_' . mt_rand();
 
         $defaultOptions = ['shortArraySyntax' => false];
         $this->options = $options + $defaultOptions;
@@ -103,7 +106,7 @@ abstract class PrettyPrinterAbstract
     public function prettyPrint(array $stmts) {
         $this->preprocessNodes($stmts);
 
-        return ltrim(str_replace("\n" . $this->noIndentToken, "\n", $this->pStmts($stmts, false)));
+        return ltrim($this->handleMagicTokens($this->pStmts($stmts, false)));
     }
 
     /**
@@ -114,7 +117,7 @@ abstract class PrettyPrinterAbstract
      * @return string Pretty printed node
      */
     public function prettyPrintExpr(Expr $node) {
-        return str_replace("\n" . $this->noIndentToken, "\n", $this->p($node));
+        return $this->handleMagicTokens($this->p($node));
     }
 
     /**
@@ -156,6 +159,17 @@ abstract class PrettyPrinterAbstract
         }
     }
 
+    protected function handleMagicTokens($str) {
+        // Drop no-indent tokens
+        $str = str_replace($this->noIndentToken, '', $str);
+
+        // Replace doc-string-end tokens with nothing or a newline
+        $str = str_replace($this->docStringEndToken . ";\n", ";\n", $str);
+        $str = str_replace($this->docStringEndToken, "\n", $str);
+
+        return $str;
+    }
+
     /**
      * Pretty prints an array of nodes (statements) and indents them optionally.
      *
@@ -167,10 +181,15 @@ abstract class PrettyPrinterAbstract
     protected function pStmts(array $nodes, $indent = true) {
         $result = '';
         foreach ($nodes as $node) {
-            $result .= "\n"
+            $comments = $node->getAttribute('comments', array());
-                    . $this->pComments($node->getAttribute('comments', array()))
+            if ($comments) {
-                    . $this->p($node)
+                $result .= "\n" . $this->pComments($comments);
-                    . ($node instanceof Expr ? ';' : '');
+                if ($node instanceof Stmt\Nop) {
+                    continue;
+                }
+            }
+
+            $result .= "\n" . $this->p($node) . ($node instanceof Expr ? ';' : '');
         }
 
         if ($indent) {
@@ -246,7 +265,11 @@ abstract class PrettyPrinterAbstract
     protected function pImplode(array $nodes, $glue = '') {
         $pNodes = array();
         foreach ($nodes as $node) {
-            $pNodes[] = $this->p($node);
+            if (null === $node) {
+                $pNodes[] = '';
+            } else {
+                $pNodes[] = $this->p($node);
+            }
         }
 
         return implode($glue, $pNodes);
@@ -282,12 +305,12 @@ abstract class PrettyPrinterAbstract
      * @return string Reformatted text of comments
      */
     protected function pComments(array $comments) {
-        $result = '';
+        $formattedComments = [];
 
         foreach ($comments as $comment) {
-            $result .= $comment->getReformattedText() . "\n";
+            $formattedComments[] = $comment->getReformattedText();
         }
 
-        return $result;
+        return implode("\n", $formattedComments);
     }
 }

test/PhpParser/Builder/FunctionTest.php

@@ -76,6 +76,24 @@ class FunctionTest extends \PHPUnit_Framework_TestCase
         )), $node);
     }
 
+    public function testReturnType() {
+        $node = $this->createFunctionBuilder('test')
+            ->setReturnType('void')
+            ->getNode();
+
+        $this->assertEquals(new Stmt\Function_('test', array(
+            'returnType' => 'void'
+        ), array()), $node);
+    }
+
+    /**
+     * @expectedException \LogicException
+     * @expectedExceptionMessage void type cannot be nullable
+     */
+    public function testInvalidNullableVoidType() {
+        $this->createFunctionBuilder('test')->setReturnType('?void');
+    }
+
     /**
      * @expectedException \LogicException
      * @expectedExceptionMessage Expected parameter node, got "Name"

test/PhpParser/Builder/MethodTest.php

@@ -120,6 +120,15 @@ class MethodTest extends \PHPUnit_Framework_TestCase
         )), $node);
     }
 
+    public function testReturnType() {
+        $node = $this->createMethodBuilder('test')
+            ->setReturnType('bool')
+            ->getNode();
+        $this->assertEquals(new Stmt\ClassMethod('test', array(
+            'returnType' => 'bool'
+        ), array()), $node);
+    }
+
     /**
      * @expectedException \LogicException
      * @expectedExceptionMessage Cannot add statements to an abstract method

test/PhpParser/Builder/ParamTest.php

@@ -78,36 +78,70 @@ class ParamTest extends \PHPUnit_Framework_TestCase
         );
     }
 
-    public function testTypeHints() {
+    /**
+     * @dataProvider provideTestTypeHints
+     */
+    public function testTypeHints($typeHint, $expectedType) {
         $node = $this->createParamBuilder('test')
-            ->setTypeHint('array')
+            ->setTypeHint($typeHint)
             ->getNode()
         ;
+        $type = $node->type;
 
-        $this->assertEquals(
+        /* Manually implement comparison to avoid __toString stupidity */
-            new Node\Param('test', null, 'array'),
+        if ($expectedType instanceof Node\NullableType) {
-            $node
+            $this->assertInstanceOf(get_class($expectedType), $type);
+            $expectedType = $expectedType->type;
+            $type = $type->type;
+        }
+
+        if ($expectedType instanceof Node\Name) {
+            $this->assertInstanceOf(get_class($expectedType), $type);
+            $this->assertEquals($expectedType, $type);
+        } else {
+            $this->assertSame($expectedType, $type);
+        }
+    }
+
+    public function provideTestTypeHints() {
+        return array(
+            array('array', 'array'),
+            array('callable', 'callable'),
+            array('bool', 'bool'),
+            array('int', 'int'),
+            array('float', 'float'),
+            array('string', 'string'),
+            array('iterable', 'iterable'),
+            array('Array', 'array'),
+            array('CALLABLE', 'callable'),
+            array('Some\Class', new Node\Name('Some\Class')),
+            array('\Foo', new Node\Name\FullyQualified('Foo')),
+            array('self', new Node\Name('self')),
+            array('?array', new Node\NullableType('array')),
+            array('?Some\Class', new Node\NullableType(new Node\Name('Some\Class'))),
+            array(new Node\Name('Some\Class'), new Node\Name('Some\Class')),
+            array(new Node\NullableType('int'), new Node\NullableType('int')),
+            array(
+                new Node\NullableType(new Node\Name('Some\Class')),
+                new Node\NullableType(new Node\Name('Some\Class'))
+            ),
         );
+    }
 
-        $node = $this->createParamBuilder('test')
+    /**
-            ->setTypeHint('callable')
+     * @expectedException \LogicException
-            ->getNode()
+     * @expectedExceptionMessage Parameter type cannot be void
-        ;
+     */
+    public function testVoidTypeError() {
+        $this->createParamBuilder('test')->setTypeHint('void');
+    }
 
-        $this->assertEquals(
+    /**
-            new Node\Param('test', null, 'callable'),
+     * @expectedException \LogicException
-            $node
+     * @expectedExceptionMessage Type must be a string, or an instance of Name or NullableType
-        );
+     */
-
+    public function testInvalidTypeError() {
-        $node = $this->createParamBuilder('test')
+        $this->createParamBuilder('test')->setTypeHint(new \stdClass);
-            ->setTypeHint('Some\Class')
-            ->getNode()
-        ;
-
-        $this->assertEquals(
-            new Node\Param('test', null, new Node\Name('Some\Class')),
-            $node
-        );
     }
 
     public function testByRef() {

test/PhpParser/Builder/TraitTest.php

@@ -27,7 +27,7 @@ class TraitTest extends \PHPUnit_Framework_TestCase
             ->addStmt($prop)
             ->getNode();
         $this->assertEquals(new Stmt\Trait_('TestTrait', array(
-            $prop, $method1, $method2, $method3
+            'stmts' => array($prop, $method1, $method2, $method3)
         ), array(
             'comments' => array(
                 new Comment\Doc('/** Nice trait */')

test/PhpParser/CodeParsingTest.php

@@ -13,7 +13,7 @@ class CodeParsingTest extends CodeTestAbstract
      */
     public function testParse($name, $code, $expected, $mode) {
         $lexer = new Lexer\Emulative(array('usedAttributes' => array(
-            'startLine', 'endLine', 'startFilePos', 'endFilePos'
+            'startLine', 'endLine', 'startFilePos', 'endFilePos', 'comments'
         )));
         $parser5 = new Parser\Php5($lexer, array(
             'throwOnError' => false,
@@ -47,7 +47,7 @@ class CodeParsingTest extends CodeTestAbstract
         }
 
         if (null !== $stmts) {
-            $dumper = new NodeDumper;
+            $dumper = new NodeDumper(['dumpComments' => true]);
             $output .= $dumper->dump($stmts);
         }
 

test/PhpParser/CodeTestAbstract.php

@@ -5,14 +5,16 @@ namespace PhpParser;
 abstract class CodeTestAbstract extends \PHPUnit_Framework_TestCase
 {
     protected function getTests($directory, $fileExtension) {
+        $directory = realpath($directory);
         $it = new \RecursiveDirectoryIterator($directory);
         $it = new \RecursiveIteratorIterator($it, \RecursiveIteratorIterator::LEAVES_ONLY);
         $it = new \RegexIterator($it, '(\.' . preg_quote($fileExtension) . '$)');
 
         $tests = array();
         foreach ($it as $file) {
-            $fileName = realpath($file->getPathname());
+            $fileName = $file->getPathname();
             $fileContents = file_get_contents($fileName);
+            $fileContents = canonicalize($fileContents);
 
             // evaluate @@{expr}@@ expressions
             $fileContents = preg_replace_callback(
@@ -24,17 +26,17 @@ abstract class CodeTestAbstract extends \PHPUnit_Framework_TestCase
             );
 
             // parse sections
-            $parts = array_map('trim', explode('-----', $fileContents));
+            $parts = preg_split("/\n-----(?:\n|$)/", $fileContents);
 
             // first part is the name
             $name = array_shift($parts) . ' (' . $fileName . ')';
-            $shortName = basename($fileName, '.test');
+            $shortName = ltrim(str_replace($directory, '', $fileName), '/\\');
 
             // multiple sections possible with always two forming a pair
             $chunks = array_chunk($parts, 2);
             foreach ($chunks as $i => $chunk) {
                 $dataSetName = $shortName . (count($chunks) > 1 ? '#' . $i : '');
-                list($expected, $mode) = $this->extractMode(canonicalize($chunk[1]));
+                list($expected, $mode) = $this->extractMode($chunk[1]);
                 $tests[$dataSetName] = array($name, $chunk[0], $expected, $mode);
             }
         }

test/PhpParser/CommentTest.php

@@ -5,18 +5,12 @@ namespace PhpParser;
 class CommentTest extends \PHPUnit_Framework_TestCase
 {
     public function testGetSet() {
-        $comment = new Comment('/* Some comment */', 1);
+        $comment = new Comment('/* Some comment */', 1, 10);
 
         $this->assertSame('/* Some comment */', $comment->getText());
         $this->assertSame('/* Some comment */', (string) $comment);
         $this->assertSame(1, $comment->getLine());
-
+        $this->assertSame(10, $comment->getFilePos());
-        $comment->setText('/* Some other comment */');
-        $comment->setLine(10);
-
-        $this->assertSame('/* Some other comment */', $comment->getText());
-        $this->assertSame('/* Some other comment */', (string) $comment);
-        $this->assertSame(10, $comment->getLine());
     }
 
     /**
@@ -58,6 +52,14 @@ class CommentTest extends \PHPUnit_Framework_TestCase
                 '/* Some text.
    More text.
    Even more text. */'
+            ),
+            array(
+                '/* Some text.
+       More text.
+         Indented text. */',
+                '/* Some text.
+   More text.
+     Indented text. */',
             ),
             // invalid comment -> no reformatting
             array(

test/PhpParser/LexerTest.php

@@ -72,7 +72,8 @@ class LexerTest extends \PHPUnit_Framework_TestCase
                     ),
                     array(
                         Tokens::T_INLINE_HTML, 'plaintext',
-                        array('startLine' => 1), array('endLine' => 1)
+                        array('startLine' => 1, 'hasLeadingNewline' => false),
+                        array('endLine' => 1)
                     ),
                 )
             ),
@@ -93,7 +94,9 @@ class LexerTest extends \PHPUnit_Framework_TestCase
                         ord('$'), '$',
                         array(
                             'startLine' => 3,
-                            'comments' => array(new Comment\Doc('/** doc' . "\n" . 'comment */', 2))
+                            'comments' => array(
+                                new Comment\Doc('/** doc' . "\n" . 'comment */', 2, 14),
+                            )
                         ),
                         array('endLine' => 3)
                     ),
@@ -109,10 +112,10 @@ class LexerTest extends \PHPUnit_Framework_TestCase
                         array(
                             'startLine' => 2,
                             'comments' => array(
-                                new Comment('/* comment */', 1),
+                                new Comment('/* comment */', 1, 6),
-                                new Comment('// comment' . "\n", 1),
+                                new Comment('// comment' . "\n", 1, 20),
-                                new Comment\Doc('/** docComment 1 */', 2),
+                                new Comment\Doc('/** docComment 1 */', 2, 31),
-                                new Comment\Doc('/** docComment 2 */', 2),
+                                new Comment\Doc('/** docComment 2 */', 2, 50),
                             ),
                         ),
                         array('endLine' => 2)

test/PhpParser/Node/NameTest.php

@@ -30,51 +30,6 @@ class NameTest extends \PHPUnit_Framework_TestCase
         $this->assertSame('foo_bar', $name->toString('_'));
     }
 
-    public function testSet() {
-        $name = new Name('foo');
-
-        $name->set('foo\bar');
-        $this->assertSame('foo\bar', $name->toString());
-
-        $name->set(array('foo', 'bar'));
-        $this->assertSame('foo\bar', $name->toString());
-
-        $name->set(new Name('foo\bar'));
-        $this->assertSame('foo\bar', $name->toString());
-    }
-
-    public function testSetFirst() {
-        $name = new Name('foo');
-
-        $name->setFirst('bar');
-        $this->assertSame('bar', $name->toString());
-
-        $name->setFirst('A\B');
-        $this->assertSame('A\B', $name->toString());
-
-        $name->setFirst('C');
-        $this->assertSame('C\B', $name->toString());
-
-        $name->setFirst('D\E');
-        $this->assertSame('D\E\B', $name->toString());
-    }
-
-    public function testSetLast() {
-        $name = new Name('foo');
-
-        $name->setLast('bar');
-        $this->assertSame('bar', $name->toString());
-
-        $name->setLast('A\B');
-        $this->assertSame('A\B', $name->toString());
-
-        $name->setLast('C');
-        $this->assertSame('A\C', $name->toString());
-
-        $name->setLast('D\E');
-        $this->assertSame('A\D\E', $name->toString());
-    }
-
     public function testAppend() {
         $name = new Name('foo');
 
@@ -96,20 +51,53 @@ class NameTest extends \PHPUnit_Framework_TestCase
     }
 
     public function testSlice() {
-        $name = new Name('foo\bar');
+        $name = new Name('foo\bar\baz');
-        $this->assertEquals(new Name('foo\bar'), $name->slice(0));
+        $this->assertEquals(new Name('foo\bar\baz'), $name->slice(0));
-        $this->assertEquals(new Name('bar'), $name->slice(1));
+        $this->assertEquals(new Name('bar\baz'), $name->slice(1));
-        $this->assertEquals(new Name([]), $name->slice(2));
+        $this->assertEquals(new Name([]), $name->slice(3));
+        $this->assertEquals(new Name('foo\bar\baz'), $name->slice(-3));
+        $this->assertEquals(new Name('bar\baz'), $name->slice(-2));
+        $this->assertEquals(new Name('foo\bar'), $name->slice(0, -1));
+        $this->assertEquals(new Name([]), $name->slice(0, -3));
+        $this->assertEquals(new Name('bar'), $name->slice(1, -1));
+        $this->assertEquals(new Name([]), $name->slice(1, -2));
+        $this->assertEquals(new Name('bar'), $name->slice(-2, 1));
+        $this->assertEquals(new Name('bar'), $name->slice(-2, -1));
+        $this->assertEquals(new Name([]), $name->slice(-2, -2));
     }
 
     /**
      * @expectedException \OutOfBoundsException
      * @expectedExceptionMessage Offset 4 is out of bounds
      */
-    public function testSliceException() {
+    public function testSliceOffsetTooLarge() {
         (new Name('foo\bar\baz'))->slice(4);
     }
 
+    /**
+     * @expectedException \OutOfBoundsException
+     * @expectedExceptionMessage Offset -4 is out of bounds
+     */
+    public function testSliceOffsetTooSmall() {
+        (new Name('foo\bar\baz'))->slice(-4);
+    }
+
+    /**
+     * @expectedException \OutOfBoundsException
+     * @expectedExceptionMessage Length 4 is out of bounds
+     */
+    public function testSliceLengthTooLarge() {
+        (new Name('foo\bar\baz'))->slice(0, 4);
+    }
+
+    /**
+     * @expectedException \OutOfBoundsException
+     * @expectedExceptionMessage Length -4 is out of bounds
+     */
+    public function testSliceLengthTooSmall() {
+        (new Name('foo\bar\baz'))->slice(0, -4);
+    }
+
     public function testConcat() {
         $this->assertEquals(new Name('foo\bar\baz'), Name::concat('foo', 'bar\baz'));
         $this->assertEquals(
@@ -159,6 +147,6 @@ class NameTest extends \PHPUnit_Framework_TestCase
      */
     public function testInvalidArg() {
         $name = new Name('foo');
-        $name->set(new \stdClass);
+        $name->append(new \stdClass);
     }
 }

test/PhpParser/Node/Stmt/ClassConstTest.php

@@ -0,0 +1,35 @@
+<?php
+
+namespace PhpParser\Node\Stmt;
+
+class ClassConstTest extends \PHPUnit_Framework_TestCase
+{
+    /**
+     * @dataProvider provideModifiers
+     */
+    public function testModifiers($modifier) {
+        $node = new ClassConst(
+            array(), // invalid
+            constant('PhpParser\Node\Stmt\Class_::MODIFIER_' . strtoupper($modifier))
+        );
+
+        $this->assertTrue($node->{'is' . $modifier}());
+    }
+
+    public function testNoModifiers() {
+        $node = new ClassConst(array(), 0);
+
+        $this->assertTrue($node->isPublic());
+        $this->assertFalse($node->isProtected());
+        $this->assertFalse($node->isPrivate());
+        $this->assertFalse($node->isStatic());
+    }
+
+    public function provideModifiers() {
+        return array(
+            array('public'),
+            array('protected'),
+            array('private'),
+        );
+    }
+}

test/PhpParser/Node/Stmt/ClassTest.php

@@ -56,4 +56,11 @@ class ClassTest extends \PHPUnit_Framework_TestCase
         $this->assertSame($methodTest, $class->getMethod('test'));
         $this->assertNull($class->getMethod('nonExisting'));
     }
+
+    public function testDeprecatedTypeNode() {
+        $class = new Class_('Foo', array('type' => Class_::MODIFIER_ABSTRACT));
+        $this->assertTrue($class->isAbstract());
+        $this->assertSame(Class_::MODIFIER_ABSTRACT, $class->flags);
+        $this->assertSame(Class_::MODIFIER_ABSTRACT, $class->type);
+    }
 }

test/PhpParser/NodeAbstractTest.php

@@ -144,4 +144,109 @@ class NodeAbstractTest extends \PHPUnit_Framework_TestCase
             $node->getAttributes()
         );
     }
+
+    public function testJsonSerialization() {
+        $code = <<<'PHP'
+<?php
+// comment
+/** doc comment */
+function functionName(&$a = 0, $b = 1.0) {
+    echo 'Foo';
+}
+PHP;
+        $expected = <<<'JSON'
+[
+    {
+        "nodeType": "Stmt_Function",
+        "byRef": false,
+        "name": "functionName",
+        "params": [
+            {
+                "nodeType": "Param",
+                "type": null,
+                "byRef": true,
+                "variadic": false,
+                "name": "a",
+                "default": {
+                    "nodeType": "Scalar_LNumber",
+                    "value": 0,
+                    "attributes": {
+                        "startLine": 4,
+                        "endLine": 4,
+                        "kind": 10
+                    }
+                },
+                "attributes": {
+                    "startLine": 4,
+                    "endLine": 4
+                }
+            },
+            {
+                "nodeType": "Param",
+                "type": null,
+                "byRef": false,
+                "variadic": false,
+                "name": "b",
+                "default": {
+                    "nodeType": "Scalar_DNumber",
+                    "value": 1,
+                    "attributes": {
+                        "startLine": 4,
+                        "endLine": 4
+                    }
+                },
+                "attributes": {
+                    "startLine": 4,
+                    "endLine": 4
+                }
+            }
+        ],
+        "returnType": null,
+        "stmts": [
+            {
+                "nodeType": "Stmt_Echo",
+                "exprs": [
+                    {
+                        "nodeType": "Scalar_String",
+                        "value": "Foo",
+                        "attributes": {
+                            "startLine": 5,
+                            "endLine": 5,
+                            "kind": 1
+                        }
+                    }
+                ],
+                "attributes": {
+                    "startLine": 5,
+                    "endLine": 5
+                }
+            }
+        ],
+        "attributes": {
+            "startLine": 4,
+            "comments": [
+                {
+                    "nodeType": "Comment",
+                    "text": "\/\/ comment\n",
+                    "line": 2,
+                    "filePos": 6
+                },
+                {
+                    "nodeType": "Comment_Doc",
+                    "text": "\/** doc comment *\/",
+                    "line": 3,
+                    "filePos": 17
+                }
+            ],
+            "endLine": 6
+        }
+    }
+]
+JSON;
+
+        $parser = new Parser\Php7(new Lexer());
+        $stmts = $parser->parse(canonicalize($code));
+        $json = json_encode($stmts, JSON_PRETTY_PRINT);
+        $this->assertEquals(canonicalize($expected), canonicalize($json));
+    }
 }

test/PhpParser/NodeTraverserTest.php

@@ -200,4 +200,19 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
 
         $this->assertSame($stmts, $traverser->traverse($stmts));
     }
+
+    /**
+     * @expectedException \LogicException
+     * @expectedExceptionMessage leaveNode() may only return an array if the parent structure is an array
+     */
+    public function testReplaceByArrayOnlyAllowedIfParentIsArray() {
+        $stmts = array(new Node\Expr\UnaryMinus(new Node\Scalar\LNumber(42)));
+
+        $visitor = $this->getMock('PhpParser\NodeVisitor');
+        $visitor->method('leaveNode')->willReturn(array(new Node\Scalar\DNumber(42.0)));
+
+        $traverser = new NodeTraverser();
+        $traverser->addVisitor($visitor);
+        $traverser->traverse($stmts);
+    }
 }

test/PhpParser/Parser/MultipleTest.php

@@ -61,7 +61,7 @@ class MultipleTest extends ParserTest {
                 $this->getPrefer5(),
                 [
                     new Expr\Variable(
-                        new Expr\ArrayDimFetch(new Expr\Variable('a'), new LNumber(0))
+                        new Expr\ArrayDimFetch(new Expr\Variable('a'), LNumber::fromString('0'))
                     )
                 ]
             ],
@@ -71,7 +71,7 @@ class MultipleTest extends ParserTest {
                 $this->getPrefer7(),
                 [
                     new Expr\ArrayDimFetch(
-                        new Expr\Variable(new Expr\Variable('a')), new LNumber(0)
+                        new Expr\Variable(new Expr\Variable('a')), LNumber::fromString('0')
                     )
                 ]
             ],

test/PhpParser/ParserTest.php

@@ -3,6 +3,9 @@
 namespace PhpParser;
 
 use PhpParser\Comment;
+use PhpParser\Node\Expr;
+use PhpParser\Node\Scalar;
+use PhpParser\Node\Scalar\String_;
 
 abstract class ParserTest extends \PHPUnit_Framework_TestCase
 {
@@ -54,7 +57,7 @@ EOC;
         $this->assertInstanceOf('PhpParser\Node\Stmt\Function_', $fn);
         $this->assertEquals(array(
             'comments' => array(
-                new Comment\Doc('/** Doc comment */', 2),
+                new Comment\Doc('/** Doc comment */', 2, 6),
             ),
             'startLine' => 3,
             'endLine' => 7,
@@ -76,8 +79,8 @@ EOC;
         $this->assertInstanceOf('PhpParser\Node\Stmt\Echo_', $echo);
         $this->assertEquals(array(
             'comments' => array(
-                new Comment("// Line\n", 4),
+                new Comment("// Line\n", 4, 49),
-                new Comment("// Comments\n", 5),
+                new Comment("// Comments\n", 5, 61),
             ),
             'startLine' => 6,
             'endLine' => 6,
@@ -105,6 +108,60 @@ EOC;
         $parser = $this->getParser($lexer);
         $parser->parse('dummy');
     }
+
+    /**
+     * @dataProvider provideTestExtraAttributes
+     */
+    public function testExtraAttributes($code, $expectedAttributes) {
+        $parser = $this->getParser(new Lexer);
+        $stmts = $parser->parse("<?php $code;");
+        $attributes = $stmts[0]->getAttributes();
+        foreach ($expectedAttributes as $name => $value) {
+            $this->assertSame($value, $attributes[$name]);
+        }
+    }
+
+    public function provideTestExtraAttributes() {
+        return array(
+            array('0', ['kind' => Scalar\LNumber::KIND_DEC]),
+            array('9', ['kind' => Scalar\LNumber::KIND_DEC]),
+            array('07', ['kind' => Scalar\LNumber::KIND_OCT]),
+            array('0xf', ['kind' => Scalar\LNumber::KIND_HEX]),
+            array('0XF', ['kind' => Scalar\LNumber::KIND_HEX]),
+            array('0b1', ['kind' => Scalar\LNumber::KIND_BIN]),
+            array('0B1', ['kind' => Scalar\LNumber::KIND_BIN]),
+            array('[]', ['kind' => Expr\Array_::KIND_SHORT]),
+            array('array()', ['kind' => Expr\Array_::KIND_LONG]),
+            array("'foo'", ['kind' => String_::KIND_SINGLE_QUOTED]),
+            array("b'foo'", ['kind' => String_::KIND_SINGLE_QUOTED]),
+            array("B'foo'", ['kind' => String_::KIND_SINGLE_QUOTED]),
+            array('"foo"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('b"foo"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('B"foo"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('"foo$bar"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('b"foo$bar"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('B"foo$bar"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array("<<<'STR'\nSTR\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']),
+            array("<<<STR\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("<<<\"STR\"\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("b<<<'STR'\nSTR\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']),
+            array("B<<<'STR'\nSTR\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']),
+            array("<<< \t 'STR'\nSTR\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']),
+            // HHVM doesn't support this due to a lexer bug
+            // (https://github.com/facebook/hhvm/issues/6970)
+            // array("<<<'\xff'\n\xff\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => "\xff"]),
+            array("<<<\"STR\"\n\$a\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("b<<<\"STR\"\n\$a\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("B<<<\"STR\"\n\$a\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("<<< \t \"STR\"\n\$a\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("die", ['kind' => Expr\Exit_::KIND_DIE]),
+            array("die('done')", ['kind' => Expr\Exit_::KIND_DIE]),
+            array("exit", ['kind' => Expr\Exit_::KIND_EXIT]),
+            array("exit(1)", ['kind' => Expr\Exit_::KIND_EXIT]),
+            array("?>Foo", ['hasLeadingNewline' => false]),
+            array("?>\nFoo", ['hasLeadingNewline' => true]),
+        );
+    }
 }
 
 class InvalidTokenLexer extends Lexer {

test/PhpParser/PrettyPrinterTest.php

@@ -4,6 +4,8 @@ namespace PhpParser;
 
 use PhpParser\Comment;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Scalar\Encapsed;
+use PhpParser\Node\Scalar\EncapsedStringPart;
 use PhpParser\Node\Scalar\String_;
 use PhpParser\Node\Stmt;
 use PhpParser\PrettyPrinter\Standard;
@@ -102,4 +104,61 @@ class PrettyPrinterTest extends CodeTestAbstract
         $options = isset($parts[1]) ? json_decode($parts[1], true) : [];
         return [$version, $options];
     }
+
+    public function testArraySyntaxDefault() {
+        $prettyPrinter = new Standard(['shortArraySyntax' => true]);
+        $expr = new Expr\Array_([
+            new Expr\ArrayItem(new String_('val'), new String_('key'))
+        ]);
+        $expected = "['key' => 'val']";
+        $this->assertSame($expected, $prettyPrinter->prettyPrintExpr($expr));
+    }
+
+    /**
+     * @dataProvider provideTestKindAttributes
+     */
+    public function testKindAttributes($node, $expected) {
+        $prttyPrinter = new PrettyPrinter\Standard;
+        $result = $prttyPrinter->prettyPrintExpr($node);
+        $this->assertSame($expected, $result);
+    }
+
+    public function provideTestKindAttributes() {
+        $nowdoc = ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR'];
+        $heredoc = ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR'];
+        return [
+            // Defaults to single quoted
+            [new String_('foo'), "'foo'"],
+            // Explicit single/double quoted
+            [new String_('foo', ['kind' => String_::KIND_SINGLE_QUOTED]), "'foo'"],
+            [new String_('foo', ['kind' => String_::KIND_DOUBLE_QUOTED]), '"foo"'],
+            // Fallback from doc string if no label
+            [new String_('foo', ['kind' => String_::KIND_NOWDOC]), "'foo'"],
+            [new String_('foo', ['kind' => String_::KIND_HEREDOC]), '"foo"'],
+            // Fallback if string contains label
+            [new String_("A\nB\nC", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'A']), "'A\nB\nC'"],
+            [new String_("A\nB\nC", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'B']), "'A\nB\nC'"],
+            [new String_("A\nB\nC", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'C']), "'A\nB\nC'"],
+            [new String_("STR;", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']), "'STR;'"],
+            // Doc string if label not contained (or not in ending position)
+            [new String_("foo", $nowdoc), "<<<'STR'\nfoo\nSTR\n"],
+            [new String_("foo", $heredoc), "<<<STR\nfoo\nSTR\n"],
+            [new String_("STRx", $nowdoc), "<<<'STR'\nSTRx\nSTR\n"],
+            [new String_("xSTR", $nowdoc), "<<<'STR'\nxSTR\nSTR\n"],
+            // Empty doc string variations (encapsed variant does not occur naturally)
+            [new String_("", $nowdoc), "<<<'STR'\nSTR\n"],
+            [new String_("", $heredoc), "<<<STR\nSTR\n"],
+            [new Encapsed([new EncapsedStringPart('')], $heredoc), "<<<STR\nSTR\n"],
+            // Encapsed doc string variations
+            [new Encapsed([new EncapsedStringPart('foo')], $heredoc), "<<<STR\nfoo\nSTR\n"],
+            [new Encapsed([new EncapsedStringPart('foo'), new Expr\Variable('y')], $heredoc), "<<<STR\nfoo{\$y}\nSTR\n"],
+            [new Encapsed([new EncapsedStringPart("\nSTR"), new Expr\Variable('y')], $heredoc), "<<<STR\n\nSTR{\$y}\nSTR\n"],
+            [new Encapsed([new EncapsedStringPart("\nSTR"), new Expr\Variable('y')], $heredoc), "<<<STR\n\nSTR{\$y}\nSTR\n"],
+            [new Encapsed([new Expr\Variable('y'), new EncapsedStringPart("STR\n")], $heredoc), "<<<STR\n{\$y}STR\n\nSTR\n"],
+            // Encapsed doc string fallback
+            [new Encapsed([new Expr\Variable('y'), new EncapsedStringPart("\nSTR")], $heredoc), '"{$y}\\nSTR"'],
+            [new Encapsed([new EncapsedStringPart("STR\n"), new Expr\Variable('y')], $heredoc), '"STR\\n{$y}"'],
+            [new Encapsed([new EncapsedStringPart("STR")], $heredoc), '"STR"'],
+        ];
+    }
 }

test/PhpParser/Serializer/XMLTest.php

@@ -23,6 +23,9 @@ CODE;
 <AST xmlns:node="http://nikic.github.com/PHPParser/XML/node" xmlns:subNode="http://nikic.github.com/PHPParser/XML/subNode" xmlns:attribute="http://nikic.github.com/PHPParser/XML/attribute" xmlns:scalar="http://nikic.github.com/PHPParser/XML/scalar">
  <scalar:array>
   <node:Stmt_Function>
+   <attribute:startLine>
+    <scalar:int>4</scalar:int>
+   </attribute:startLine>
    <attribute:comments>
     <scalar:array>
      <comment isDocComment="false" line="2">// comment
@@ -30,9 +33,6 @@ CODE;
      <comment isDocComment="true" line="3">/** doc comment */</comment>
     </scalar:array>
    </attribute:comments>
-   <attribute:startLine>
-    <scalar:int>4</scalar:int>
-   </attribute:startLine>
    <attribute:endLine>
     <scalar:int>6</scalar:int>
    </attribute:endLine>
@@ -71,6 +71,9 @@ CODE;
         <attribute:endLine>
          <scalar:int>4</scalar:int>
         </attribute:endLine>
+        <attribute:kind>
+         <scalar:int>10</scalar:int>
+        </attribute:kind>
         <subNode:value>
          <scalar:int>0</scalar:int>
         </subNode:value>
@@ -133,6 +136,9 @@ CODE;
          <attribute:endLine>
           <scalar:int>5</scalar:int>
          </attribute:endLine>
+         <attribute:kind>
+          <scalar:int>1</scalar:int>
+         </attribute:kind>
          <subNode:value>
           <scalar:string>Foo</scalar:string>
          </subNode:value>

test/bootstrap.php

@@ -5,12 +5,16 @@ namespace PhpParser;
 require __DIR__ . '/../vendor/autoload.php';
 
 function canonicalize($str) {
-    // trim from both sides
+    // normalize EOL style
-    $str = trim($str);
+    $str = str_replace("\r\n", "\n", $str);
 
-    // normalize EOL to \n
+    // trim newlines at end
-    $str = str_replace(array("\r\n", "\r"), "\n", $str);
+    $str = rtrim($str, "\n");
 
-    // trim right side of all lines
+    // remove trailing whitespace on all lines
-    return implode("\n", array_map('rtrim', explode("\n", $str)));
+    $lines = explode("\n", $str);
+    $lines = array_map(function($line) {
+        return rtrim($line, " \t");
+    }, $lines);
+    return implode("\n", $lines);
 }

test/code/parser/blockComments.test

@@ -0,0 +1,23 @@
+Comments on blocks
+-----
+<?php
+
+// foo
+{
+    // bar
+    {
+        // baz
+        $a;
+    }
+}
+-----
+array(
+    0: Expr_Variable(
+        name: a
+        comments: array(
+            0: // foo
+            1: // bar
+            2: // baz
+        )
+    )
+)

test/code/parser/comments.test

@@ -0,0 +1,100 @@
+Comments
+-----
+<?php
+
+/** doc 1 */
+/* foobar 1 */
+// foo 1
+// bar 1
+$var;
+
+if ($cond) {
+    /** doc 2 */
+    /* foobar 2 */
+    // foo 2
+    // bar 2
+}
+
+/** doc 3 */
+/* foobar 3 */
+// foo 3
+// bar 3
+-----
+array(
+    0: Expr_Variable(
+        name: var
+        comments: array(
+            0: /** doc 1 */
+            1: /* foobar 1 */
+            2: // foo 1
+            3: // bar 1
+        )
+    )
+    1: Stmt_If(
+        cond: Expr_Variable(
+            name: cond
+        )
+        stmts: array(
+            0: Stmt_Nop(
+                comments: array(
+                    0: /** doc 2 */
+                    1: /* foobar 2 */
+                    2: // foo 2
+                    3: // bar 2
+                )
+            )
+        )
+        elseifs: array(
+        )
+        else: null
+    )
+    2: Stmt_Nop(
+        comments: array(
+            0: /** doc 3 */
+            1: /* foobar 3 */
+            2: // foo 3
+            3: // bar 3
+        )
+    )
+)
+-----
+<?php
+
+/** doc */
+/* foobar */
+// foo
+// bar
+
+?>
+-----
+array(
+    0: Stmt_Nop(
+        comments: array(
+            0: /** doc */
+            1: /* foobar */
+            2: // foo
+            3: // bar
+        )
+    )
+)
+-----
+<?php
+
+// comment
+if (42) {}
+-----
+array(
+    0: Stmt_If(
+        cond: Scalar_LNumber(
+            value: 42
+        )
+        stmts: array(
+        )
+        elseifs: array(
+        )
+        else: null
+        comments: array(
+            0: // comment
+        )
+    )
+)

test/code/parser/errorHandling/eofError.test

@@ -4,10 +4,29 @@ Error positions
 -----
 Syntax error, unexpected EOF from 1:10 to 1:10
 array(
+    0: Expr_ConstFetch(
+        name: Name(
+            parts: array(
+                0: foo
+            )
+        )
+    )
 )
 -----
 <?php foo /* bar */
 -----
 Syntax error, unexpected EOF from 1:20 to 1:20
 array(
-)
+    0: Expr_ConstFetch(
+        name: Name(
+            parts: array(
+                0: foo
+            )
+        )
+    )
+    1: Stmt_Nop(
+        comments: array(
+            0: /* bar */
+        )
+    )
+)

test/code/parser/errorHandling/recovery.test

@@ -10,6 +10,33 @@ Syntax error, unexpected T_STRING from 4:1 to 4:3
 Syntax error, unexpected T_STRING from 5:1 to 5:3
 Syntax error, unexpected EOF from 5:6 to 5:6
 array(
+    0: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: foo
+            )
+        )
+        args: array(
+        )
+    )
+    1: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: bar
+            )
+        )
+        args: array(
+        )
+    )
+    2: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: baz
+            )
+        )
+        args: array(
+        )
+    )
 )
 -----
 <?php
@@ -23,13 +50,22 @@ array(
     0: Expr_FuncCall(
         name: Name(
             parts: array(
-                0: bar
+                0: foo
             )
         )
         args: array(
         )
     )
     1: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: bar
+            )
+        )
+        args: array(
+        )
+    )
+    2: Expr_FuncCall(
         name: Name(
             parts: array(
                 0: baz
@@ -58,6 +94,15 @@ array(
         )
     )
     1: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: bar
+            )
+        )
+        args: array(
+        )
+    )
+    2: Expr_FuncCall(
         name: Name(
             parts: array(
                 0: baz
@@ -81,6 +126,9 @@ array(
             )
         )
     )
+    1: Scalar_LNumber(
+        value: 1
+    )
 )
 -----
 <?php
@@ -97,6 +145,9 @@ array(
         )
         returnType: null
         stmts: array(
+            0: Scalar_LNumber(
+                value: 1
+            )
         )
     )
 )
@@ -172,6 +223,11 @@ array(
             value: 2
         )
     )
+    3: Stmt_Nop(
+        comments: array(
+            0: // The output here drops the loop - would require Error node to handle this
+        )
+    )
 )
 -----
 <?php
@@ -184,3 +240,43 @@ while (true) {
     $i = 2;
 -----
 Syntax error, unexpected EOF from 8:12 to 8:12
+-----
+<?php
+$foo->
+;
+-----
+Syntax error, unexpected ';' from 3:1 to 3:1
+array(
+    0: Expr_PropertyFetch(
+        var: Expr_Variable(
+            name: foo
+        )
+        name: Expr_Error(
+        )
+    )
+)
+-----
+<?php
+function foo() {
+    $bar->
+}
+-----
+Syntax error, unexpected '}' from 4:1 to 4:1
+array(
+    0: Stmt_Function(
+        byRef: false
+        name: foo
+        params: array(
+        )
+        returnType: null
+        stmts: array(
+            0: Expr_PropertyFetch(
+                var: Expr_Variable(
+                    name: bar
+                )
+                name: Expr_Error(
+                )
+            )
+        )
+    )
+)

test/code/parser/expr/arrayDef.test

@@ -97,6 +97,9 @@ array(
     5: Expr_Array(
         items: array(
         )
+        comments: array(
+            0: // short array syntax
+        )
     )
     6: Expr_Array(
         items: array(

test/code/parser/expr/arrayDestructuring.test

@@ -0,0 +1,144 @@
+Array destructuring
+-----
+<?php
+
+[$a, $b] = [$c, $d];
+[, $a, , , $b, ,] = $foo;
+[, [[$a]], $b] = $bar;
+['a' => $b, 'b' => $a] = $baz;
+-----
+!!php7
+array(
+    0: Expr_Assign(
+        var: Expr_Array(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Variable(
+                        name: a
+                    )
+                    byRef: false
+                )
+                1: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Variable(
+                        name: b
+                    )
+                    byRef: false
+                )
+            )
+        )
+        expr: Expr_Array(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Variable(
+                        name: c
+                    )
+                    byRef: false
+                )
+                1: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Variable(
+                        name: d
+                    )
+                    byRef: false
+                )
+            )
+        )
+    )
+    1: Expr_Assign(
+        var: Expr_Array(
+            items: array(
+                0: null
+                1: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Variable(
+                        name: a
+                    )
+                    byRef: false
+                )
+                2: null
+                3: null
+                4: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Variable(
+                        name: b
+                    )
+                    byRef: false
+                )
+                5: null
+            )
+        )
+        expr: Expr_Variable(
+            name: foo
+        )
+    )
+    2: Expr_Assign(
+        var: Expr_Array(
+            items: array(
+                0: null
+                1: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Array(
+                        items: array(
+                            0: Expr_ArrayItem(
+                                key: null
+                                value: Expr_Array(
+                                    items: array(
+                                        0: Expr_ArrayItem(
+                                            key: null
+                                            value: Expr_Variable(
+                                                name: a
+                                            )
+                                            byRef: false
+                                        )
+                                    )
+                                )
+                                byRef: false
+                            )
+                        )
+                    )
+                    byRef: false
+                )
+                2: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Variable(
+                        name: b
+                    )
+                    byRef: false
+                )
+            )
+        )
+        expr: Expr_Variable(
+            name: bar
+        )
+    )
+    3: Expr_Assign(
+        var: Expr_Array(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: Scalar_String(
+                        value: a
+                    )
+                    value: Expr_Variable(
+                        name: b
+                    )
+                    byRef: false
+                )
+                1: Expr_ArrayItem(
+                    key: Scalar_String(
+                        value: b
+                    )
+                    value: Expr_Variable(
+                        name: a
+                    )
+                    byRef: false
+                )
+            )
+        )
+        expr: Expr_Variable(
+            name: baz
+        )
+    )
+)

test/code/parser/expr/assign.test

@@ -23,7 +23,6 @@ $a = $b *= $c **= $d;
 
 // by ref assign
 $a =& $b;
-$a =& new B;
 
 // list() assign
 list($a) = $b;
@@ -40,18 +39,30 @@ array(
     0: Expr_Assign(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // simple assign
+            )
         )
         expr: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // simple assign
+        )
     )
     1: Expr_AssignOp_BitwiseAnd(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // combined assign
+            )
         )
         expr: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // combined assign
+        )
     )
     2: Expr_AssignOp_BitwiseOr(
         var: Expr_Variable(
@@ -144,6 +155,9 @@ array(
     13: Expr_Assign(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // chained assign
+            )
         )
         expr: Expr_AssignOp_Mul(
             var: Expr_Variable(
@@ -158,50 +172,63 @@ array(
                 )
             )
         )
+        comments: array(
+            0: // chained assign
+        )
     )
     14: Expr_AssignRef(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // by ref assign
+            )
         )
         expr: Expr_Variable(
             name: b
         )
-    )
+        comments: array(
-    15: Expr_AssignRef(
+            0: // by ref assign
-        var: Expr_Variable(
-            name: a
         )
-        expr: Expr_New(
+    )
-            class: Name(
+    15: Expr_Assign(
-                parts: array(
+        var: Expr_List(
-                    0: B
+            items: array(
+                0: Expr_ArrayItem(
+                    key: null
+                    value: Expr_Variable(
+                        name: a
+                    )
+                    byRef: false
                 )
             )
-            args: array(
+            comments: array(
+                0: // list() assign
             )
         )
+        expr: Expr_Variable(
+            name: b
+        )
+        comments: array(
+            0: // list() assign
+        )
     )
     16: Expr_Assign(
         var: Expr_List(
-            vars: array(
+            items: array(
-                0: Expr_Variable(
+                0: Expr_ArrayItem(
-                    name: a
+                    key: null
-                )
+                    value: Expr_Variable(
-            )
+                        name: a
-        )
+                    )
-        expr: Expr_Variable(
+                    byRef: false
-            name: b
-        )
-    )
-    17: Expr_Assign(
-        var: Expr_List(
-            vars: array(
-                0: Expr_Variable(
-                    name: a
                 )
                 1: null
-                2: Expr_Variable(
+                2: Expr_ArrayItem(
-                    name: b
+                    key: null
+                    value: Expr_Variable(
+                        name: b
+                    )
+                    byRef: false
                 )
             )
         )
@@ -209,22 +236,34 @@ array(
             name: c
         )
     )
-    18: Expr_Assign(
+    17: Expr_Assign(
         var: Expr_List(
-            vars: array(
+            items: array(
-                0: Expr_Variable(
+                0: Expr_ArrayItem(
-                    name: a
+                    key: null
+                    value: Expr_Variable(
+                        name: a
+                    )
+                    byRef: false
                 )
                 1: Expr_List(
-                    vars: array(
+                    items: array(
                         0: null
-                        1: Expr_Variable(
+                        1: Expr_ArrayItem(
-                            name: c
+                            key: null
+                            value: Expr_Variable(
+                                name: c
+                            )
+                            byRef: false
                         )
                     )
                 )
-                2: Expr_Variable(
+                2: Expr_ArrayItem(
-                    name: d
+                    key: null
+                    value: Expr_Variable(
+                        name: d
+                    )
+                    byRef: false
                 )
             )
         )
@@ -232,22 +271,25 @@ array(
             name: e
         )
     )
-    19: Expr_PreInc(
+    18: Expr_PreInc(
+        var: Expr_Variable(
+            name: a
+        )
+        comments: array(
+            0: // inc/dec
+        )
+    )
+    19: Expr_PostInc(
         var: Expr_Variable(
             name: a
         )
     )
-    20: Expr_PostInc(
+    20: Expr_PreDec(
         var: Expr_Variable(
             name: a
         )
     )
-    21: Expr_PreDec(
+    21: Expr_PostDec(
-        var: Expr_Variable(
-            name: a
-        )
-    )
-    22: Expr_PostDec(
         var: Expr_Variable(
             name: a
         )

test/code/parser/expr/assignNewByRef.test

@@ -0,0 +1,39 @@
+Assigning new by reference (PHP 5 only)
+-----
+<?php
+$a =& new B;
+-----
+!!php5
+array(
+    0: Expr_AssignRef(
+        var: Expr_Variable(
+            name: a
+        )
+        expr: Expr_New(
+            class: Name(
+                parts: array(
+                    0: B
+                )
+            )
+            args: array(
+            )
+        )
+    )
+)
+-----
+<?php
+$a =& new B;
+-----
+!!php7
+Syntax error, unexpected T_NEW from 2:7 to 2:9
+array(
+    0: Expr_New(
+        class: Name(
+            parts: array(
+                0: B
+            )
+        )
+        args: array(
+        )
+    )
+)

test/code/parser/expr/fetchAndCall/funcCall.test

@@ -21,9 +21,15 @@ array(
             parts: array(
                 0: a
             )
+            comments: array(
+                0: // function name variations
+            )
         )
         args: array(
         )
+        comments: array(
+            0: // function name variations
+        )
     )
     1: Expr_FuncCall(
         name: Expr_Variable(
@@ -106,12 +112,21 @@ array(
                 parts: array(
                     0: a
                 )
+                comments: array(
+                    0: // array dereferencing
+                )
             )
             args: array(
             )
+            comments: array(
+                0: // array dereferencing
+            )
         )
         dim: Scalar_String(
             value: b
         )
+        comments: array(
+            0: // array dereferencing
+        )
     )
 )

test/code/parser/expr/fetchAndCall/objectAccess.test

@@ -22,8 +22,14 @@ array(
     0: Expr_PropertyFetch(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // property fetch variations
+            )
         )
         name: b
+        comments: array(
+            0: // property fetch variations
+        )
     )
     1: Expr_ArrayDimFetch(
         var: Expr_PropertyFetch(
@@ -50,10 +56,16 @@ array(
     3: Expr_MethodCall(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // method call variations
+            )
         )
         name: b
         args: array(
         )
+        comments: array(
+            0: // method call variations
+        )
     )
     4: Expr_MethodCall(
         var: Expr_Variable(
@@ -94,14 +106,23 @@ array(
         var: Expr_MethodCall(
             var: Expr_Variable(
                 name: a
+                comments: array(
+                    0: // array dereferencing
+                )
             )
             name: b
             args: array(
             )
+            comments: array(
+                0: // array dereferencing
+            )
         )
         dim: Scalar_String(
             value: c
         )
+        comments: array(
+            0: // array dereferencing
+        )
     )
     8: Expr_ArrayDimFetch(
         var: Expr_MethodCall(
@@ -116,4 +137,9 @@ array(
             value: c
         )
     )
-)
+    9: Stmt_Nop(
+        comments: array(
+            0: // invalid PHP: drop Support?
+        )
+    )
+)

test/code/parser/expr/fetchAndCall/staticCall.test

@@ -25,10 +25,16 @@ array(
             parts: array(
                 0: A
             )
+            comments: array(
+                0: // method name variations
+            )
         )
         name: b
         args: array(
         )
+        comments: array(
+            0: // method name variations
+        )
     )
     1: Expr_StaticCall(
         class: Name(
@@ -99,24 +105,39 @@ array(
                 parts: array(
                     0: A
                 )
+                comments: array(
+                    0: // array dereferencing
+                )
             )
             name: b
             args: array(
             )
+            comments: array(
+                0: // array dereferencing
+            )
         )
         dim: Scalar_String(
             value: c
         )
+        comments: array(
+            0: // array dereferencing
+        )
     )
     6: Expr_StaticCall(
         class: Name(
             parts: array(
                 0: static
             )
+            comments: array(
+                0: // class name variations
+            )
         )
         name: b
         args: array(
         )
+        comments: array(
+            0: // class name variations
+        )
     )
     7: Expr_StaticCall(
         class: Expr_Variable(
@@ -149,4 +170,4 @@ array(
         args: array(
         )
     )
-)
+)

test/code/parser/expr/fetchAndCall/staticPropertyFetch.test

@@ -19,8 +19,14 @@ array(
             parts: array(
                 0: A
             )
+            comments: array(
+                0: // property name variations
+            )
         )
         name: b
+        comments: array(
+            0: // property name variations
+        )
     )
     1: Expr_StaticPropertyFetch(
         class: Name(
@@ -48,12 +54,21 @@ array(
                 parts: array(
                     0: A
                 )
+                comments: array(
+                    0: // array access
+                )
             )
             name: b
+            comments: array(
+                0: // array access
+            )
         )
         dim: Scalar_String(
             value: c
         )
+        comments: array(
+            0: // array access
+        )
     )
     4: Expr_ArrayDimFetch(
         var: Expr_StaticPropertyFetch(
@@ -68,4 +83,9 @@ array(
             value: c
         )
     )
+    5: Stmt_Nop(
+        comments: array(
+            0: // class name variations can be found in staticCall.test
+        )
+    )
 )

test/code/parser/expr/includeAndEval.test

@@ -12,25 +12,25 @@ array(
         expr: Scalar_String(
             value: A.php
         )
-        type: 1
+        type: TYPE_INCLUDE (1)
     )
     1: Expr_Include(
         expr: Scalar_String(
             value: A.php
         )
-        type: 2
+        type: TYPE_INCLUDE_ONCE (2)
     )
     2: Expr_Include(
         expr: Scalar_String(
             value: A.php
         )
-        type: 3
+        type: TYPE_REQUIRE (3)
     )
     3: Expr_Include(
         expr: Scalar_String(
             value: A.php
         )
-        type: 4
+        type: TYPE_REQURE_ONCE (4)
     )
     4: Expr_Eval(
         expr: Scalar_String(

test/code/parser/expr/listWithKeys.test

@@ -0,0 +1,75 @@
+List destructing with keys
+-----
+<?php
+
+list('a' => $b) = ['a' => 'b'];
+list('a' => list($b => $c), 'd' => $e) = $x;
+-----
+!!php7
+array(
+    0: Expr_Assign(
+        var: Expr_List(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: Scalar_String(
+                        value: a
+                    )
+                    value: Expr_Variable(
+                        name: b
+                    )
+                    byRef: false
+                )
+            )
+        )
+        expr: Expr_Array(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: Scalar_String(
+                        value: a
+                    )
+                    value: Scalar_String(
+                        value: b
+                    )
+                    byRef: false
+                )
+            )
+        )
+    )
+    1: Expr_Assign(
+        var: Expr_List(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: Scalar_String(
+                        value: a
+                    )
+                    value: Expr_List(
+                        items: array(
+                            0: Expr_ArrayItem(
+                                key: Expr_Variable(
+                                    name: b
+                                )
+                                value: Expr_Variable(
+                                    name: c
+                                )
+                                byRef: false
+                            )
+                        )
+                    )
+                    byRef: false
+                )
+                1: Expr_ArrayItem(
+                    key: Scalar_String(
+                        value: d
+                    )
+                    value: Expr_Variable(
+                        name: e
+                    )
+                    byRef: false
+                )
+            )
+        )
+        expr: Expr_Variable(
+            name: x
+        )
+    )
+)

test/code/parser/expr/logic.test

@@ -24,10 +24,16 @@ array(
     0: Expr_BinaryOp_BooleanAnd(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // boolean ops
+            )
         )
         right: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // boolean ops
+        )
     )
     1: Expr_BinaryOp_BooleanOr(
         left: Expr_Variable(
@@ -52,10 +58,16 @@ array(
     4: Expr_BinaryOp_LogicalAnd(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // logical ops
+            )
         )
         right: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // logical ops
+        )
     )
     5: Expr_BinaryOp_LogicalOr(
         left: Expr_Variable(
@@ -77,10 +89,16 @@ array(
         left: Expr_BinaryOp_BooleanAnd(
             left: Expr_Variable(
                 name: a
+                comments: array(
+                    0: // precedence
+                )
             )
             right: Expr_Variable(
                 name: b
             )
+            comments: array(
+                0: // precedence
+            )
         )
         right: Expr_BinaryOp_BooleanAnd(
             left: Expr_Variable(
@@ -90,6 +108,9 @@ array(
                 name: d
             )
         )
+        comments: array(
+            0: // precedence
+        )
     )
     8: Expr_BinaryOp_BooleanAnd(
         left: Expr_BinaryOp_BooleanAnd(

test/code/parser/expr/math.test

@@ -38,6 +38,9 @@ array(
         expr: Expr_Variable(
             name: a
         )
+        comments: array(
+            0: // unary ops
+        )
     )
     1: Expr_UnaryPlus(
         expr: Expr_Variable(
@@ -52,10 +55,16 @@ array(
     3: Expr_BinaryOp_BitwiseAnd(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // binary ops
+            )
         )
         right: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // binary ops
+        )
     )
     4: Expr_BinaryOp_BitwiseOr(
         left: Expr_Variable(
@@ -149,14 +158,23 @@ array(
         left: Expr_BinaryOp_Mul(
             left: Expr_Variable(
                 name: a
+                comments: array(
+                    0: // associativity
+                )
             )
             right: Expr_Variable(
                 name: b
             )
+            comments: array(
+                0: // associativity
+            )
         )
         right: Expr_Variable(
             name: c
         )
+        comments: array(
+            0: // associativity
+        )
     )
     16: Expr_BinaryOp_Mul(
         left: Expr_Variable(
@@ -174,6 +192,9 @@ array(
     17: Expr_BinaryOp_Plus(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // precedence
+            )
         )
         right: Expr_BinaryOp_Mul(
             left: Expr_Variable(
@@ -183,6 +204,9 @@ array(
                 name: c
             )
         )
+        comments: array(
+            0: // precedence
+        )
     )
     18: Expr_BinaryOp_Mul(
         left: Expr_BinaryOp_Plus(
@@ -200,6 +224,9 @@ array(
     19: Expr_BinaryOp_Pow(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // pow is special
+            )
         )
         right: Expr_BinaryOp_Pow(
             left: Expr_Variable(
@@ -209,6 +236,9 @@ array(
                 name: c
             )
         )
+        comments: array(
+            0: // pow is special
+        )
     )
     20: Expr_BinaryOp_Pow(
         left: Expr_BinaryOp_Pow(

test/code/parser/expr/new.test

@@ -50,6 +50,9 @@ array(
         )
         args: array(
         )
+        comments: array(
+            0: // class name variations
+        )
     )
     3: Expr_New(
         class: Expr_ArrayDimFetch(
@@ -84,6 +87,9 @@ array(
         )
         args: array(
         )
+        comments: array(
+            0: // DNCR object access
+        )
     )
     6: Expr_New(
         class: Expr_PropertyFetch(

test/code/parser/expr/ternaryAndCoalesce.test

@@ -20,6 +20,9 @@ array(
     0: Expr_Ternary(
         cond: Expr_Variable(
             name: a
+            comments: array(
+                0: // ternary
+            )
         )
         if: Expr_Variable(
             name: b
@@ -27,6 +30,9 @@ array(
         else: Expr_Variable(
             name: c
         )
+        comments: array(
+            0: // ternary
+        )
     )
     1: Expr_Ternary(
         cond: Expr_Variable(
@@ -41,6 +47,9 @@ array(
         cond: Expr_Ternary(
             cond: Expr_Variable(
                 name: a
+                comments: array(
+                    0: // precedence
+                )
             )
             if: Expr_Variable(
                 name: b
@@ -48,6 +57,9 @@ array(
             else: Expr_Variable(
                 name: c
             )
+            comments: array(
+                0: // precedence
+            )
         )
         if: Expr_Variable(
             name: d
@@ -55,6 +67,9 @@ array(
         else: Expr_Variable(
             name: e
         )
+        comments: array(
+            0: // precedence
+        )
     )
     3: Expr_Ternary(
         cond: Expr_Variable(
@@ -78,10 +93,16 @@ array(
     4: Expr_BinaryOp_Coalesce(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // null coalesce
+            )
         )
         right: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // null coalesce
+        )
     )
     5: Expr_BinaryOp_Coalesce(
         left: Expr_Variable(
@@ -125,4 +146,4 @@ array(
             name: c
         )
     )
-)
+)

test/code/parser/scalar/constantString.test

@@ -8,8 +8,10 @@ b'';
 b"";
 'Hi';
 b'Hi';
+B'Hi';
 "Hi";
 b"Hi";
+B"Hi";
 '!\'!\\!\a!';
 "!\"!\\!\$!\n!\r!\t!\f!\v!\e!\a";
 "!\xFF!\377!\400!\0!";
@@ -40,14 +42,19 @@ array(
         value: Hi
     )
     8: Scalar_String(
-        value: !'!\!\a!
+        value: Hi
     )
     9: Scalar_String(
-        value: !"!\!$!
+        value: Hi
-    !
-!@@{ "\t" }@@!@@{ "\f" }@@!@@{ "\v" }@@!@@{ chr(27) /* "\e" */ }@@!\a
     )
     10: Scalar_String(
+        value: !'!\!\a!
+    )
+    11: Scalar_String(
+        value: !"!\!$!
+    !@@{ "\r" }@@!@@{ "\t" }@@!@@{ "\f" }@@!@@{ "\v" }@@!@@{ chr(27) /* "\e" */ }@@!\a
+    )
+    12: Scalar_String(
         value: !@@{ chr(255) }@@!@@{ chr(255) }@@!@@{ chr(0) }@@!@@{ chr(0) }@@!
     )
 )

test/code/parser/scalar/docString.test

@@ -24,17 +24,26 @@ EOS;
 Test $a and $b->c test
 EOS;
 
-// comment to force line break before EOF
+b<<<EOS
+Binary
+EOS;
+
 -----
 array(
     0: Scalar_String(
         value:
+        comments: array(
+            0: // empty strings
+        )
     )
     1: Scalar_String(
         value:
     )
     2: Scalar_String(
         value: Test '" $a \n
+        comments: array(
+            0: // constant encapsed strings
+        )
     )
     3: Scalar_String(
         value: Test '" $a
@@ -49,6 +58,9 @@ array(
                 name: a
             )
         )
+        comments: array(
+            0: // encapsed strings
+        )
     )
     5: Scalar_Encapsed(
         parts: array(
@@ -72,4 +84,7 @@ array(
             )
         )
     )
+    6: Scalar_String(
+        value: Binary
+    )
 )

test/code/parser/scalar/docStringNewlines.test

@@ -12,7 +12,6 @@ Trailing newlines in doc strings
 <<<'EOF'@@{ "\r\nFoo\r\n\r\n" }@@EOF;
 <<<EOF@@{ "\r\n\$var\r\n\r\n" }@@EOF;
 
-// comment to force line break before EOF
 -----
 array(
     0: Scalar_String(

test/code/parser/scalar/encapsedString.test

@@ -21,6 +21,7 @@ Encapsed strings
 "$$A[B]";
 "A $B C";
 b"$A";
+B"$A";
 -----
 array(
     0: Scalar_Encapsed(
@@ -231,4 +232,11 @@ array(
             )
         )
     )
+    19: Scalar_Encapsed(
+        parts: array(
+            0: Expr_Variable(
+                name: A
+            )
+        )
+    )
 )

test/code/parser/scalar/float.test

@@ -53,18 +53,22 @@ array(
         value: INF
     )
     10: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
+        comments: array(
+            0: // various integer -> float overflows
+            1: // (all are actually the same number, just in different representations)
+        )
     )
     11: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
     )
     12: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
     )
     13: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
     )
     14: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
     )
 )

test/code/parser/scalar/int.test

@@ -10,7 +10,6 @@ Different integer syntaxes
 0xfff;
 0XfFf;
 0777;
-0787;
 0b111000111000;
 -----
 array(
@@ -39,9 +38,6 @@ array(
         value: 511
     )
     8: Scalar_LNumber(
-        value: 7
-    )
-    9: Scalar_LNumber(
         value: 3640
     )
 )

test/code/parser/scalar/invalidOctal.test

@@ -0,0 +1,17 @@
+Invalid octal literals
+-----
+<?php
+0787;
+-----
+!!php7
+Invalid numeric literal from 2:1 to 2:4
+-----
+<?php
+0787;
+-----
+!!php5
+array(
+    0: Scalar_LNumber(
+        value: 7
+    )
+)

test/code/parser/semiReserved.test

@@ -54,14 +54,14 @@ class Foo {
 -----
 array(
     0: Stmt_Class(
-        type: 0
+        flags: 0
         name: Test
         extends: null
         implements: array(
         )
         stmts: array(
             0: Stmt_ClassMethod(
-                type: 0
+                flags: 0
                 byRef: false
                 name: array
                 params: array(
@@ -71,7 +71,7 @@ array(
                 )
             )
             1: Stmt_ClassMethod(
-                type: 0
+                flags: 0
                 byRef: false
                 name: public
                 params: array(
@@ -81,7 +81,7 @@ array(
                 )
             )
             2: Stmt_ClassMethod(
-                type: 8
+                flags: MODIFIER_STATIC (8)
                 byRef: false
                 name: list
                 params: array(
@@ -91,7 +91,7 @@ array(
                 )
             )
             3: Stmt_ClassMethod(
-                type: 8
+                flags: MODIFIER_STATIC (8)
                 byRef: false
                 name: protected
                 params: array(
@@ -101,7 +101,7 @@ array(
                 )
             )
             4: Stmt_Property(
-                type: 1
+                flags: MODIFIER_PUBLIC (1)
                 props: array(
                     0: Stmt_PropertyProperty(
                         name: class
@@ -110,7 +110,7 @@ array(
                 )
             )
             5: Stmt_Property(
-                type: 1
+                flags: MODIFIER_PUBLIC (1)
                 props: array(
                     0: Stmt_PropertyProperty(
                         name: private
@@ -119,6 +119,7 @@ array(
                 )
             )
             6: Stmt_ClassConst(
+                flags: 0
                 consts: array(
                     0: Const(
                         name: TRAIT
@@ -135,6 +136,7 @@ array(
                 )
             )
             7: Stmt_ClassConst(
+                flags: 0
                 consts: array(
                     0: Const(
                         name: __CLASS__
@@ -267,7 +269,7 @@ array(
         name: FINAL
     )
     10: Stmt_Class(
-        type: 0
+        flags: 0
         name: Foo
         extends: null
         implements: array(
@@ -319,7 +321,7 @@ array(
                             )
                         )
                         method: throw
-                        newModifier: 2
+                        newModifier: MODIFIER_PROTECTED (2)
                         newName: public
                     )
                     3: Stmt_TraitUseAdaptation_Alias(
@@ -329,7 +331,7 @@ array(
                             )
                         )
                         method: self
-                        newModifier: 2
+                        newModifier: MODIFIER_PROTECTED (2)
                         newName: null
                     )
                     4: Stmt_TraitUseAdaptation_Alias(

test/code/parser/stmt/class/abstract.test

@@ -9,14 +9,14 @@ abstract class A {
 -----
 array(
     0: Stmt_Class(
-        type: 16
+        flags: MODIFIER_ABSTRACT (16)
         name: A
         extends: null
         implements: array(
         )
         stmts: array(
             0: Stmt_ClassMethod(
-                type: 1
+                flags: MODIFIER_PUBLIC (1)
                 byRef: false
                 name: a
                 params: array(
@@ -26,7 +26,7 @@ array(
                 )
             )
             1: Stmt_ClassMethod(
-                type: 17
+                flags: MODIFIER_PUBLIC | MODIFIER_ABSTRACT (17)
                 byRef: false
                 name: b
                 params: array(
@@ -36,4 +36,4 @@ array(
             )
         )
     )
-)
+)

test/code/parser/stmt/class/anonymous.test

@@ -24,14 +24,14 @@ class A {
 array(
     0: Expr_New(
         class: Stmt_Class(
-            type: 0
+            flags: 0
             name: null
             extends: null
             implements: array(
             )
             stmts: array(
                 0: Stmt_ClassMethod(
-                    type: 1
+                    flags: MODIFIER_PUBLIC (1)
                     byRef: false
                     name: test
                     params: array(
@@ -47,7 +47,7 @@ array(
     )
     1: Expr_New(
         class: Stmt_Class(
-            type: 0
+            flags: 0
             name: null
             extends: Name(
                 parts: array(
@@ -74,14 +74,14 @@ array(
     )
     2: Expr_New(
         class: Stmt_Class(
-            type: 0
+            flags: 0
             name: null
             extends: null
             implements: array(
             )
             stmts: array(
                 0: Stmt_Property(
-                    type: 1
+                    flags: MODIFIER_PUBLIC (1)
                     props: array(
                         0: Stmt_PropertyProperty(
                             name: foo
@@ -96,7 +96,7 @@ array(
     )
     3: Expr_New(
         class: Stmt_Class(
-            type: 0
+            flags: 0
             name: null
             extends: Name(
                 parts: array(
@@ -137,14 +137,14 @@ array(
         )
     )
     4: Stmt_Class(
-        type: 0
+        flags: 0
         name: A
         extends: null
         implements: array(
         )
         stmts: array(
             0: Stmt_ClassMethod(
-                type: 1
+                flags: MODIFIER_PUBLIC (1)
                 byRef: false
                 name: test
                 params: array(
@@ -154,7 +154,7 @@ array(
                     0: Stmt_Return(
                         expr: Expr_New(
                             class: Stmt_Class(
-                                type: 0
+                                flags: 0
                                 name: null
                                 extends: Name(
                                     parts: array(
@@ -165,6 +165,7 @@ array(
                                 )
                                 stmts: array(
                                     0: Stmt_ClassConst(
+                                        flags: 0
                                         consts: array(
                                             0: Const(
                                                 name: A
@@ -191,4 +192,4 @@ array(
             )
         )
     )
-)
+)

test/code/parser/stmt/class/conditional.test

@@ -17,7 +17,7 @@ array(
         )
         stmts: array(
             0: Stmt_Class(
-                type: 0
+                flags: 0
                 name: A
                 extends: null
                 implements: array(