Release PHP-Parser 3.1.0

It doesn't look like this is going away for now, and we currently
don't have the APIs to cover all existing usages elegantly.

.gitignore

@@ -0,0 +1,4 @@
+vendor/
+composer.lock
+grammar/kmyacc.exe
+grammar/y.output

.travis.yml

@@ -1,13 +1,31 @@
 language: php
+dist: trusty
+sudo: false
+
+cache:
+  directories:
+    - $HOME/.composer/cache
 
 php:
-  - 5.3
-  - 5.4
   - 5.5
   - 5.6
+  - 7.0
+  - nightly
   - hhvm
 
+install:
+  - if [ $TRAVIS_PHP_VERSION = '5.6' ]; then composer require satooshi/php-coveralls '~1.0'; fi
+  - composer install --prefer-dist
+
 matrix:
   allow_failures:
-    - php: hhvm
+    - php: nightly
   fast_finish: true
+
+script:
+  - if [ $TRAVIS_PHP_VERSION = '5.6' ]; then vendor/bin/phpunit --coverage-clover build/logs/clover.xml; else vendor/bin/phpunit; fi
+  - if [ $TRAVIS_PHP_VERSION = '7.0' ]; then test_old/run-php-src.sh; fi
+
+after_success:
+  if [ $TRAVIS_PHP_VERSION = '5.6' ]; then php vendor/bin/coveralls; fi
+

CHANGELOG.md

@@ -1,108 +1,378 @@
-Version 1.0.2-dev
+Version 3.1.1-dev
 -----------------
 
 Nothing yet.
 
-Version 1.0.1 (14.10.2014)
+Version 3.1.0 (2017-07-28)
 --------------------------
 
-* Disallow `new` expressions without a class name. Previously `new;` was accidentally considered to be valid code.
+### Added
 
-* Support T_ONUMBER token used by HHVM.
+* [PHP 7.2] Added support for trailing comma in group use statements.
+* [PHP 7.2] Added support for `object` type. This means `object` types will now be represented as a
+  builtin type (a simple `"object"` string), rather than a class `Name`.
+  
+### Fixed
 
-* Add ability to directly pass code to the `php-parse.php` script.
+* Floating-point numbers are now printed correctly if the LC_NUMERIC locale uses a comma as decimal
+  separator.
 
-* Prevent truncation of `var_dump()` output in the `php-parse.php` script if XDebug is used.
+### Changed
 
-Version 1.0.0 (12.09.2014)
+* `Name::$parts` is no longer deprecated.
+
+Version 3.0.6 (2017-06-28)
 --------------------------
 
-* [BC] Removed deprecated `Template` and `TemplateLoader` classes.
+### Fixed
 
-* Fixed XML unserializer to properly work with new namespaced node names.
+* Fixed the spelling of `Class_::VISIBILITY_MODIFIER_MASK`. The previous spelling of
+  `Class_::VISIBILITY_MODIFER_MASK` is preserved for backwards compatibility.
+* The pretty printing will now preserve comments inside array literals and function calls by
+  printing the array items / function arguments on separate lines. Array literals and functions that
+  do not contain comments are not affected.
 
-Version 1.0.0-beta2 (31.08.2014)
+### Added
+
+* Added `Builder\Param::makeVariadic()`.
+
+### Deprecated
+
+* The `Node::setLine()` method has been deprecated.
+
+Version 3.0.5 (2017-03-05)
+--------------------------
+
+### Fixed
+
+* Name resolution of `NullableType`s is now performed earlier, so that a fully resolved signature is
+  available when a function is entered. (#360)
+* `Error` nodes are now considered empty, while previously they extended until the token where the
+  error occurred. This made some nodes larger than expected. (#359)
+* Fixed notices being thrown during error recovery in some situations. (#362)
+
+Version 3.0.4 (2017-02-10)
+--------------------------
+
+### Fixed
+
+* Fixed some extensibility issues in pretty printer (`pUseType()` is now public and `pPrec()` calls
+  into `p()`, instead of directly dispatching to the type-specific printing method).
+* Fixed notice in `bin/php-parse` script.
+
+### Added
+
+* Error recovery from missing semicolons is now supported in more cases.
+* Error recovery from trailing commas in positions where PHP does not support them is now supported.
+
+Version 3.0.3 (2017-02-03)
+--------------------------
+
+### Fixed
+
+* In `"$foo[0]"` the `0` is now parsed as an `LNumber` rather than `String`. (#325)
+* Ensure integers and floats are always pretty printed preserving semantics, even if the particular
+  value can only be manually constructed.
+* Throw a `LogicException` when trying to pretty-print an `Error` node. Previously this resulted in
+  an undefined method exception or fatal error.
+
+### Added
+
+* [PHP 7.1] Added support for negative interpolated offsets: `"$foo[-1]"`
+* Added `preserveOriginalNames` option to `NameResolver`. If this option is enabled, an
+  `originalName` attribute, containing the unresolved name, will be added to each resolved name.
+* Added `php-parse --with-positions` option, which dumps nodes with position information.
+
+### Deprecated
+
+* The XML serializer has been deprecated. In particular, the classes `Serializer\XML`,
+  `Unserializer\XML`, as well as the interfaces `Serializer` and `Unserializer` are deprecated.
+
+Version 3.0.2 (2016-12-06)
+--------------------------
+
+### Fixed
+
+* Fixed name resolution of nullable types. (#324)
+* Fixed pretty-printing of nullable types.
+
+Version 3.0.1 (2016-12-01)
+--------------------------
+
+### Fixed
+
+* Fixed handling of nested `list()`s: If the nested list was unkeyed, it was directly included in
+  the list items. If it was keyed, it was wrapped in `ArrayItem`. Now nested `List_` nodes are
+  always wrapped in `ArrayItem`s. (#321)
+
+Version 3.0.0 (2016-11-30)
+--------------------------
+
+### Added
+
+* Added support for dumping node positions in the NodeDumper through the `dumpPositions` option.
+* Added error recovery support for `$`, `new`, `Foo::`.
+
+Version 3.0.0-beta2 (2016-10-29)
 --------------------------------
 
-* [PHP 5.6] Updated support for constant scalar expressions to comply with latest changes. This means that arrays
-  and array dimension fetches are now supported as well.
+This release primarily improves our support for error recovery.
 
-* [PHP 5.6] Direct array dereferencing of constants is supported now, i.e. both `FOO[0]` and `Foo::BAR[0]` are valid
-  now.
+### Added
 
-* Fixed handling of special class names (`self`, `parent` and `static`) in the name resolver to be case insensitive.
-  Additionally the name resolver now enforces that special class names are only used as unqualified names, e.g. `\self`
-  is considered invalid.
+* Added `Node::setDocComment()` method.
+* Added `Error::getMessageWithColumnInfo()` method.
+* Added support for recovery from lexer errors.
+* Added support for recovering from "special" errors (i.e. non-syntax parse errors).
+* Added precise location information for lexer errors.
+* Added `ErrorHandler` interface, and `ErrorHandler\Throwing` and `ErrorHandler\Collecting` as
+  specific implementations. These provide a general mechanism for handling error recovery.
+* Added optional `ErrorHandler` argument to `Parser::parse()`, `Lexer::startLexing()` and
+  `NameResolver::__construct()`.
+* The `NameResolver` now adds a `namespacedName` attribute on name nodes that cannot be statically
+  resolved (unqualified unaliased function or constant names in namespaces).
+  
+### Fixed
 
-* The case of references to the `static` class name is now preserved. Previously `static` was always lowercased,
-  regardless of the case used in the source code.
+* Fixed attribute assignment for `GroupUse` prefix and variables in interpolated strings.
 
-* The autoloader now only requires a file if it exists. This allows usages like
-  `class_exists('PhpParser\NotExistingClass')`.
+### Changed
 
-* Added experimental `bin/php-parse.php` script, which is intended to help exploring and debugging the node tree.
+* The constants on `NameTraverserInterface` have been moved into the `NameTraverser` class.
+* Due to the error handling changes, the `Parser` interface and `Lexer` API have changed.
+* The emulative lexer now directly postprocesses tokens, instead of using `~__EMU__~` sequences.
+  This changes the protected API of the lexer.
+* The `Name::slice()` method now returns `null` for empty slices, previously `new Name([])` was
+  used. `Name::concat()` now also supports concatenation with `null`.
 
-* Separated the parser implemention (in `lib/PhpParser/ParserAbstract.php`) and the generated data (in
-  `lib/PhpParser/Parser.php`). Furthermore the parser now uses meaningful variable names and contains comments
-  explaining their usage.
+### Removed
 
-Version 1.0.0-beta1 (27.03.2014)
+* Removed `Name::append()` and `Name::prepend()`. These mutable methods have been superseded by
+  the immutable `Name::concat()`.
+* Removed `Error::getRawLine()` and `Error::setRawLine()`. These methods have been superseded by
+  `Error::getStartLine()` and `Error::setStartLine()`.
+* Removed support for node cloning in the `NodeTraverser`.
+* Removed `$separator` argument from `Name::toString()`.
+* Removed `throw_on_error` parser option and `Parser::getErrors()` method. Use the `ErrorHandler`
+  mechanism instead.
+
+Version 3.0.0-beta1 (2016-09-16)
 --------------------------------
 
-* [BC] PHP-Parser now requires PHP 5.3 or newer to run. It is however still possible to *parse* PHP 5.2 source code,
+### Added
+
+* [7.1] Function/method and parameter builders now support PHP 7.1 type hints (void, iterable and
+  nullable types).
+* Nodes and Comments now implement `JsonSerializable`. The node kind is stored in a `nodeType`
+  property.
+* The `InlineHTML` node now has an `hasLeadingNewline` attribute, that specifies whether the
+  preceding closing tag contained a newline. The pretty printer honors this attribute.
+* Partial parsing of `$obj->` (with missing property name) is now supported in error recovery mode.
+* The error recovery mode is now exposed in the `php-parse` script through the `--with-recovery`
+  or `-r` flags.
+
+The following changes are also part of PHP-Parser 2.1.1:
+
+* The PHP 7 parser will now generate a parse error for `$var =& new Obj` assignments.
+* Comments on free-standing code blocks will now be retained as comments on the first statement in
+  the code block.
+
+Version 3.0.0-alpha1 (2016-07-25)
+---------------------------------
+
+### Added
+
+* [7.1] Added support for `void` and `iterable` types. These will now be represented as strings
+  (instead of `Name` instances) similar to other builtin types.
+* [7.1] Added support for class constant visibility. The `ClassConst` node now has a `flags` subnode
+  holding the visibility modifier, as well as `isPublic()`, `isProtected()` and `isPrivate()`
+  methods. The constructor changed to accept the additional subnode.
+* [7.1] Added support for nullable types. These are represented using a new `NullableType` node
+  with a single `type` subnode.
+* [7.1] Added support for short array destructuring syntax. This means that `Array` nodes may now
+  appear as the left-hand-side of assignments and foreach value targets. Additionally the array
+  items may now contain `null` values if elements are skipped.
+* [7.1] Added support for keys in list() destructuring. The `List` subnode `vars` has been renamed
+  to `items` and now contains `ArrayItem`s instead of plain variables.
+* [7.1] Added support for multi-catch. The `Catch` subnode `type` has been renamed to `types` and
+  is now an array of `Name`s.
+* `Name::slice()` now supports lengths and negative offsets. This brings it in line with
+  `array_slice()` functionality.
+
+### Changed
+
+Due to PHP 7.1 support additions described above, the node structure changed as follows:
+
+* `void` and `iterable` types are now stored as strings if the PHP 7 parser is used.
+* The `ClassConst` constructor changed to accept an additional `flags` subnode.
+* The `Array` subnode `items` may now contain `null` elements (destructuring).
+* The `List` subnode `vars` has been renamed to `items` and now contains `ArrayItem`s instead of
+  plain variables.
+* The `Catch` subnode `type` has been renamed to `types` and is now an array of `Name`s.
+
+Additionally the following changes were made:
+
+* The `type` subnode on `Class`, `ClassMethod` and `Property` has been renamed to `flags`. The
+  `type` subnode has retained for backwards compatibility and is populated to the same value as
+  `flags`. However, writes to `type` will not update `flags`.
+* The `TryCatch` subnode `finallyStmts` has been replaced with a `finally` subnode that holds an
+  explicit `Finally` node. This allows for more accurate attribute assignment.
+* The `Trait` constructor now has the same form as the `Class` and `Interface` constructors: It
+  takes an array of subnodes. Unlike classes/interfaces, traits can only have a `stmts` subnode.
+* The `NodeDumper` now prints class/method/property/constant modifiers, as well as the include and
+  use type in a textual representation, instead of only showing the number.
+* All methods on `PrettyPrinter\Standard` are now protected. Previoulsy most of them were public.
+
+### Removed
+
+* Removed support for running on PHP 5.4. It is however still possible to parse PHP 5.2-5.4 code
   while running on a newer version.
+* The deprecated `Comment::setLine()` and `Comment::setText()` methods have been removed.
+* The deprecated `Name::set()`, `Name::setFirst()` and `Name::setLast()` methods have been removed.
 
-* [BC] The library has been moved to use namespaces with the `PhpParser` vendor prefix. However, the old names using
-  underscores are still available as aliases, as such most code should continue running on the new version without
-  further changes.
+Version 2.1.1 (2016-09-16)
+--------------------------
 
-  However, code performing dispatch operations on `Node::getType()` may be affected by some of the name changes. For
-  example a `+` node will now return type `Expr_BinaryOp_Plus` instead of `Expr_Plus`. In particular this may affect
-  custom pretty printers.
+### Changed
 
-  Due to conflicts with reserved keywords, some class names now end with an underscore, e.g. `PHPParser_Node_Stmt_Class`
-  is now `PhpParser\Node\Stmt\Class_`. (But as usual, the old name is still available)
+* The pretty printer will now escape all control characters in the range `\x00-\x1F` inside double
+  quoted strings. If no special escape sequence is available, an octal escape will be used.
+* The quality of the error recovery has been improved. In particular unterminated expressions should
+  be handled more gracefully.
+* The PHP 7 parser will now generate a parse error for `$var =& new Obj` assignments.
+* Comments on free-standing code blocks will no be retained as comments on the first statement in
+  the code block.
 
-* [PHP 5.6] Added support for the power operator `**` (node `Expr\BinaryOp\Pow`) and the compound power assignment
-  operator `**=` (node `Expr\AssignOp\Pow`).
+Version 2.1.0 (2016-04-19)
+--------------------------
 
-* [PHP 5.6] Added support for variadic functions: `Param` nodes now have `variadic` as a boolean subnode.
+### Fixed
 
-* [PHP 5.6] Added support for argument unpacking: `Arg` nodes now have `unpack` as a boolean subnode.
+* Properly support `B""` strings (with uppercase `B`) in a number of places.
+* Fixed reformatting of indented parts in a certain non-standard comment style.
 
-* [PHP 5.6] Added support for aliasing of functions and constants. `Stmt\Use_` nodes now have an integral `type`
-  subnode, which is one of `Stmt\Use_::TYPE_NORMAL` (`use`), `Stmt\Use_::TYPE_FUNCTION` (`use function`) or
-  `Stmt\Use_::TYPE_CONSTANT` (`use const`).
+### Added
 
-  The `NameResolver` now also supports resolution of such aliases.
+* Added `dumpComments` option to node dumper, to enable dumping of comments associated with nodes.
+* Added `Stmt\Nop` node, that is used to collect comments located at the end of a block or at the
+  end of a file (without a following node with which they could otherwise be associated).
+* Added `kind` attribute to `Expr\Exit` to distinguish between `exit` and `die`.
+* Added `kind` attribute to `Scalar\LNumber` to distinguish between decimal, binary, octal and
+  hexadecimal numbers.
+* Added `kind` attribtue to `Expr\Array` to distinguish between `array()` and `[]`.
+* Added `kind` attribute to `Scalar\String` and `Scalar\Encapsed` to distinguish between
+  single-quoted, double-quoted, heredoc and nowdoc string.
+* Added `docLabel` attribute to `Scalar\String` and `Scalar\Encapsed`, if it is a heredoc or
+  nowdoc string.
+* Added start file offset information to `Comment` nodes.
+* Added `setReturnType()` method to function and method builders.
+* Added `-h` and `--help` options to `php-parse` script.
 
-* [PHP 5.6] Added support for constant scalar expressions. This means that certain expressions are now allowed as the
-  initializer for constants, properties, parameters, static variables, etc.
+### Changed
 
-* [BC] Improved pretty printing of empty statements lists, which are now printed as `{\n}` instead of `{\n    \n}`.
-  This changes the behavior of the protected `PrettyPrinterAbstract::pStmts()` method, so custom pretty printing code
-  making use it of may need to be adjusted.
+* Invalid octal literals now throw a parse error in PHP 7 mode.
+* The pretty printer takes all the new attributes mentioned in the previous section into account.
+* The protected `AbstractPrettyPrinter::pComments()` method no longer returns a trailing newline.
+* The bundled autoloader supports library files being stored in a different directory than
+  `PhpParser` for easier downstream distribution.
 
-* Changed the order of some subnodes to be consistent with their order in the sour code. For example `Stmt\If->cond`
-  will now appear before `Stmt\If->stmts` etc.
+### Deprecated
 
-* Added `Scalar\MagicConstant->getName()`, which returns the name of the magic constant (e.g. `__CLASS__`).
+* The `Comment::setLine()` and `Comment::setText()` methods have been deprecated. Construct new
+  objects instead.
 
-**The following changes are also included in 0.9.5**:
+### Removed
 
-* [BC] Deprecated `PHPParser_Template` and `PHPParser_TemplateLoader`. This functionality does not belong in the main project
-  and - as far as I know - nobody is using it.
+* The internal (but public) method `Scalar\LNumber::parse()` has been removed. A non-internal
+  `LNumber::fromString()` method has been added instead.
 
-* Add `NodeTraverser::removeVisitor()` method, which removes a visitor from the node traverser. This also modifies the
-  corresponding `NodeTraverserInterface`.
+Version 2.0.1 (2016-02-28)
+--------------------------
 
-* Fix alias resolution in `NameResolver`: Class names are now correctly handled as case-insensitive.
+### Fixed
 
-* The undefined variable error, which is used to the lexer to reset the error state, will no longer interfere with
-  custom error handlers.
+* `declare() {}` and `declare();` are not semantically equivalent and will now result in different
+  ASTs. The format case will have an empty `stmts` array, while the latter will set `stmts` to
+  `null`.
+* Magic constants are now supported as semi-reserved keywords.
+* A shebang line like `#!/usr/bin/env php` is now allowed at the start of a namespaced file.
+  Previously this generated an exception.
+* The `prettyPrintFile()` method will not strip a trailing `?>` from the raw data that follows a
+  `__halt_compiler()` statement.
+* The `prettyPrintFile()` method will not strip an opening `<?php` if the file starts with a
+  comment followed by InlineHTML.
+
+Version 2.0.0 (2015-12-04)
+--------------------------
+
+### Changed
+
+* String parts of encapsed strings are now represented using `Scalar\EncapsStringPart` nodes.
+  Previously raw strings were used. This affects the `parts` child of `Scalar\Encaps` and
+  `Expr\ShellExec`. The change has been done to allow assignment of attributes to encapsed string
+  parts.
+
+Version 2.0.0-beta1 (2015-10-21)
+--------------------------------
+
+### Fixed
+
+* Fixed issue with too many newlines being stripped at the end of heredoc/nowdoc strings in some
+  cases. (#227)
+
+### Changed
+
+* Update group use support to be in line with recent PHP 7.0 builds.
+* Renamed `php-parse.php` to `php-parse` and registered it as a composer bin.
+* Use composer PSR-4 autoloader instead of custom autoloader.
+* Specify phpunit as a dev dependency.
+
+### Added
+
+* Added `shortArraySyntax` option to pretty printer, to print all arrays using short syntax.
+
+Version 2.0.0-alpha1 (2015-07-14)
+---------------------------------
+
+A more detailed description of backwards incompatible changes can be found in the
+[upgrading guide](UPGRADE-2.0.md).
+
+### Removed
+
+* Removed support for running on PHP 5.3. It is however still possible to parse PHP 5.2 and PHP 5.3
+  code while running on a newer version.
+* Removed legacy class name aliases. This includes the old non-namespaced class names and the old
+  names for classes that were renamed for PHP 7 compatibility.
+* Removed support for legacy node format. All nodes must have a `getSubNodeNames()` method now.
+
+### Added
+
+* Added support for remaining PHP 7 features that were not present in 1.x:
+  * Group use declarations. These are represented using `Stmt\GroupUse` nodes. Furthermore a `type`
+    attribute was added to `Stmt\UseUse` to handle mixed group use declarations.
+  * Uniform variable syntax.
+  * Generalized yield operator.
+  * Scalar type declarations. These are presented using `'bool'`, `'int'`, `'float'` and `'string'`
+    as the type. The PHP 5 parser also accepts these, however they'll be `Name` instances there.
+  * Unicode escape sequences.
+* Added `PhpParser\ParserFactory` class, which should be used to create parser instances.
+* Added `Name::concat()` which concatenates two names.
+* Added `Name->slice()` which takes a subslice of a name.
+
+### Changed
+
+* `PhpParser\Parser` is now an interface, implemented by `Parser\Php5`, `Parser\Php7` and
+  `Parser\Multiple`. The `Multiple` parser will try multiple parsers, until one succeeds.
+* Token constants are now defined on `PhpParser\Parser\Tokens` rather than `PhpParser\Parser`.
+* The `Name->set()`, `Name->append()`, `Name->prepend()` and `Name->setFirst()` methods are
+  deprecated in favor of `Name::concat()` and `Name->slice()`.
+* The `NodeTraverser` no longer clones nodes by default. The old behavior can be restored by
+  passing `true` to the constructor.
+* The constructor for `Scalar` nodes no longer has a default value. E.g. `new LNumber()` should now
+  be written as `new LNumber(0)`.
 
 ---
 
-**This changelog only includes changes from the 1.0 series. For older changes see the [0.9 series changelog][1].**
-
- [1]: https://github.com/nikic/PHP-Parser/blob/0.9/CHANGELOG.md
+**This changelog only includes changes from the 2.0 series. For older changes see the
+[1.x series changelog](https://github.com/nikic/PHP-Parser/blob/1.x/CHANGELOG.md) and the
+[0.9 series changelog](https://github.com/nikic/PHP-Parser/blob/0.9/CHANGELOG.md).**

README.md

@@ -1,12 +1,16 @@
 PHP Parser
 ==========
 
-This is a PHP 5.2 to PHP 5.6 parser written in PHP. It's purpose is to simplify static code analysis and
+[![Build Status](https://travis-ci.org/nikic/PHP-Parser.svg?branch=master)](https://travis-ci.org/nikic/PHP-Parser) [![Coverage Status](https://coveralls.io/repos/github/nikic/PHP-Parser/badge.svg?branch=master)](https://coveralls.io/github/nikic/PHP-Parser?branch=master)
+
+This is a PHP 5.2 to PHP 7.1 parser written in PHP. Its purpose is to simplify static code analysis and
 manipulation.
 
-[**Documentation for version 1.0.x**][doc_master] (stable; for running on PHP >= 5.3).
+[**Documentation for version 3.x**][doc_master] (stable; for running on PHP >= 5.5; for parsing PHP 5.2 to PHP 7.1).
 
-[Documentation for version 0.9.x][doc_0_9] (unsupported; for running on PHP 5.2).
+[Documentation for version 2.x][doc_2_x] (stable; for running on PHP >= 5.4; for parsing PHP 5.2 to PHP 7.0).
+
+[Documentation for version 1.x][doc_1_x] (unsupported; for running on PHP >= 5.3; for parsing PHP 5.2 to PHP 5.6).
 
 In a Nutshell
 -------------
@@ -22,7 +26,7 @@ hello\world('foo', 'bar' . 'baz');
 
 You'll get a syntax tree looking roughly like this:
 
-```
+```php
 array(
     0: Stmt_Echo(
         exprs: array(
@@ -70,7 +74,26 @@ programming errors or security issues).
 Additionally, you can convert a syntax tree back to PHP code. This allows you to do code preprocessing
 (like automatedly porting code to older PHP versions).
 
-So, that's it, in a nutshell. You can find everything else in the [docs][doc_master].
+Installation
+------------
 
- [doc_0_9]: https://github.com/nikic/PHP-Parser/tree/0.9/doc
- [doc_master]: https://github.com/nikic/PHP-Parser/tree/master/doc
+The preferred installation method is [composer](https://getcomposer.org):
+
+    php composer.phar require nikic/php-parser
+
+Documentation
+-------------
+
+ 1. [Introduction](doc/0_Introduction.markdown)
+ 2. [Usage of basic components](doc/2_Usage_of_basic_components.markdown)
+ 3. [Other node tree representations](doc/3_Other_node_tree_representations.markdown)
+ 4. [Code generation](doc/4_Code_generation.markdown)
+
+Component documentation:
+
+ 1. [Error handling](doc/component/Error_handling.markdown)
+ 2. [Lexer](doc/component/Lexer.markdown)
+
+ [doc_1_x]: https://github.com/nikic/PHP-Parser/tree/1.x/doc
+ [doc_2_x]: https://github.com/nikic/PHP-Parser/tree/2.x/doc
+ [doc_master]: https://github.com/nikic/PHP-Parser/tree/master/doc

UPGRADE-2.0.md

@@ -0,0 +1,74 @@
+Upgrading from PHP-Parser 1.x to 2.0
+====================================
+
+### PHP version requirements
+
+PHP-Parser now requires PHP 5.4 or newer to run. It is however still possible to *parse* PHP 5.2 and
+PHP 5.3 source code, while running on a newer version.
+
+### Creating a parser instance
+
+Parser instances should now be created through the `ParserFactory`. Old direct instantiation code
+will not work, because the parser class was renamed.
+
+Old:
+
+```php
+use PhpParser\Parser, PhpParser\Lexer;
+$parser = new Parser(new Lexer\Emulative);
+```
+
+New:
+
+```php
+use PhpParser\ParserFactory;
+$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
+```
+
+The first argument to `ParserFactory` determines how different PHP versions are handled. The
+possible values are:
+
+ * `ParserFactory::PREFER_PHP7`: Try to parse code as PHP 7. If this fails, try to parse it as PHP 5.
+ * `ParserFactory::PREFER_PHP5`: Try to parse code as PHP 5. If this fails, try to parse it as PHP 7.
+ * `ParserFactory::ONLY_PHP7`: Parse code as PHP 7.
+ * `ParserFactory::ONLY_PHP5`: Parse code as PHP 5.
+
+For most practical purposes the difference between `PREFER_PHP7` and `PREFER_PHP5` is mainly whether
+a scalar type hint like `string` will be stored as `'string'` (PHP 7) or as `new Name('string')`
+(PHP 5).
+
+To use a custom lexer, pass it as the second argument to the `create()` method:
+
+```php
+use PhpParser\ParserFactory;
+$lexer = new MyLexer;
+$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7, $lexer);
+```
+
+### Rename of the `PhpParser\Parser` class
+
+`PhpParser\Parser` is now an interface, which is implemented by `Parser\Php5`, `Parser\Php7` and
+`Parser\Multiple`. Parser tokens are now defined in `Parser\Tokens`. If you use the `ParserFactory`
+described above to create your parser instance, these changes should have no further impact on you.
+
+### Removal of legacy aliases
+
+All legacy aliases for classes have been removed. This includes the old non-namespaced `PHPParser_`
+classes, as well as the classes that had to be renamed for PHP 7 support.
+
+### Deprecations
+
+The `set()`, `setFirst()`, `append()` and `prepend()` methods of the `Node\Name` class have been
+deprecated. Instead `Name::concat()` and `Name->slice()` should be used.
+
+### Miscellaneous
+
+* The `NodeTraverser` no longer clones nodes by default. If you want to restore the old behavior,
+  pass `true` to the constructor.
+* The legacy node format has been removed. If you use custom nodes, they are now expected to
+  implement a `getSubNodeNames()` method.
+* The default value for `Scalar` node constructors was removed. This means that something like
+  `new LNumber()` should be replaced by `new LNumber(0)`.
+* String parts of encapsed strings are now represented using `Scalar\EncapsStringPart` nodes, while
+  previously raw strings were used. This affects the `parts` child of `Scalar\Encaps` and
+  `Expr\ShellExec`.

UPGRADE-3.0.md

@@ -0,0 +1,160 @@
+Upgrading from PHP-Parser 2.x to 3.0
+====================================
+
+The backwards-incompatible changes in this release may be summarized as follows:
+
+ * The specific details of the node representation have changed in some cases, primarily to
+   accomodate new PHP 7.1 features.
+ * There have been significant changes to the error recovery implementation. This may affect you,
+   if you used the error recovery mode or have a custom lexer implementation.
+ * A number of deprecated methods were removed.
+
+### PHP version requirements
+
+PHP-Parser now requires PHP 5.5 or newer to run. It is however still possible to *parse* PHP 5.2,
+5.3 and 5.4 source code, while running on a newer version.
+
+### Changes to the node structure
+
+The following changes are likely to require code changes if the respective nodes are used:
+
+ * The `List` subnode `vars` has been renamed to `items` and now contains `ArrayItem`s instead of
+   plain variables.
+ * The `Catch` subnode `type` has been renamed to `types` and is now an array of `Name`s.
+ * The `TryCatch` subnode `finallyStmts` has been replaced with a `finally` subnode that holds an
+   explicit `Finally` node.
+ * The `type` subnode on `Class`, `ClassMethod` and `Property` has been renamed to `flags`. The
+   `type` subnode has retained for backwards compatibility and is populated to the same value as
+   `flags`. However, writes to `type` will not update `flags` and use of `type` is discouraged.
+
+The following changes are unlikely to require code changes:
+
+ * The `ClassConst` constructor changed to accept an additional `flags` subnode.
+ * The `Trait` constructor now has the same form as the `Class` and `Interface` constructors: It
+   takes an array of subnodes. Unlike classes/interfaces, traits can only have a `stmts` subnode.
+ * The `Array` subnode `items` may now contain `null` elements (due to destructuring).
+ * `void` and `iterable` types are now stored as strings if the PHP 7 parser is used. Previously
+   these would have been represented as `Name` instances.
+
+### Changes to error recovery mode
+
+Previously, error recovery mode was enabled by setting the `throwOnError` option to `false` when
+creating the parser, while collected errors were retrieved using the `getErrors()` method:
+
+```php
+$lexer = ...;
+$parser = (new ParserFactory)->create(ParserFactor::ONLY_PHP7, $lexer, [
+    'throwOnError' => true,
+]);
+
+$stmts = $parser->parse($code);
+$errors = $parser->getErrors();
+if ($errors) {
+    handleErrors($errors);
+}
+processAst($stmts);
+```
+
+Both the `throwOnError` option and the `getErrors()` method have been removed in PHP-Parser 3.0.
+Instead an instance of `ErrorHandler\Collecting` should be passed to the `parse()` method:
+
+```php
+$lexer = ...;
+$parser = (new ParserFactory)->create(ParserFactor::ONLY_PHP7, $lexer);
+
+$errorHandler = new ErrorHandler\Collecting;
+$stmts = $parser->parse($code, $errorHandler);
+if ($errorHandler->hasErrors()) {
+    handleErrors($errorHandler->getErrors());
+}
+processAst($stmts);
+```
+
+#### Multiple parser fallback in error recovery mode
+
+As a result of this change, if a `Multiple` parser is used (e.g. through the `ParserFactory` using
+`PREFER_PHP7` or `PREFER_PHP5`), it will now return the result of the first *non-throwing* parse. As
+parsing never throws in error recovery mode, the result from the first parser will always be
+returned.
+
+The PHP 7 parser is a superset of the PHP 5 parser, with the exceptions that `=& new` and
+`global $$foo->bar` are not supported (other differences are in representation only). The PHP 7
+parser will be able to recover from the error in both cases. For this reason, this change will
+likely pass unnoticed if you do not specifically test for this syntax.
+
+It is possible to restore the precise previous behavior with the following code:
+
+```php
+$lexer = ...;
+$parser7 = new Parser\Php7($lexer);
+$parser5 = new Parser\Php5($lexer);
+
+$errors7 = new ErrorHandler\Collecting();
+$stmts7 = $parser7->parse($code, $errors7);
+if ($errors7->hasErrors()) {
+    $errors5 = new ErrorHandler\Collecting();
+    $stmts5 = $parser5->parse($code, $errors5);
+    if (!$errors5->hasErrors()) {
+        // If PHP 7 parse has errors but PHP 5 parse has no errors, use PHP 5 result
+        return [$stmts5, $errors5];
+    }
+}
+// If PHP 7 succeeds or both fail use PHP 7 result
+return [$stmts7, $errors7];
+```
+
+#### Error handling in the lexer
+
+In order to support recovery from lexer errors, the signature of the `startLexing()` method changed
+to optionally accept an `ErrorHandler`:
+
+```php
+// OLD
+public function startLexing($code);
+// NEW
+public function startLexing($code, ErrorHandler $errorHandler = null);
+```
+
+If you use a custom lexer with overriden `startLexing()` method, it needs to be changed to accept
+the extra parameter. The value should be passed on to the parent method.
+
+#### Error checks in node constructors
+
+The constructors of certain nodes used to contain additional checks for semantic errors, such as
+creating a try block without either catch or finally. These checks have been moved from the node
+constructors into the parser. This allows recovery from such errors, as well as representing the
+resulting (invalid) AST.
+
+This means that certain error conditions are no longer checked for manually constructed nodes.
+
+### Removed methods, arguments, options
+
+The following methods, arguments or options have been removed:
+
+ * `Comment::setLine()`, `Comment::setText()`: Create new `Comment` instances instead.
+ * `Name::set()`, `Name::setFirst()`, `Name::setLast()`, `Name::append()`, `Name::prepend()`:
+    Use `Name::concat()` in combination with `Name::slice()` instead.
+ * `Error::getRawLine()`, `Error::setRawLine()`. Use `Error::getStartLine()` and
+   `Error::setStartLine()` instead.
+ * `Parser::getErrors()`. Use `ErrorHandler\Collecting` instead.
+ * `$separator` argument of `Name::toString()`. Use `strtr()` instead, if you really need it.
+ * `$cloneNodes` argument of `NodeTraverser::__construct()`. Explicitly clone nodes in the visitor
+   instead.
+ * `throwOnError` parser option. Use `ErrorHandler\Collecting` instead.
+
+### Miscellaneous
+
+ * The `NameResolver` will now resolve unqualified function and constant names in the global
+   namespace into fully qualified names. For example `foo()` in the global namespace resolves to
+   `\foo()`. For names where no static resolution is possible, a `namespacedName` attribute is
+   added now, containing the namespaced variant of the name.
+ * All methods on `PrettyPrinter\Standard` are now protected. Previoulsy most of them were public.
+   The pretty printer should only be invoked using the `prettyPrint()`, `prettyPrintFile()` and
+   `prettyPrintExpr()` methods.
+ * The node dumper now prints numeric values that act as enums/flags in a string representation.
+   If node dumper results are used in tests, updates may be needed to account for this.
+ * The constants on `NameTraverserInterface` have been moved into the `NameTraverser` class.
+ * The emulative lexer now directly postprocesses tokens, instead of using `~__EMU__~` sequences.
+   This changes the protected API of the emulative lexer.
+ * The `Name::slice()` method now returns `null` for empty slices, previously `new Name([])` was
+   used. `Name::concat()` now also supports concatenation with `null`.

bin/php-parse

@@ -0,0 +1,202 @@
+#!/usr/bin/env php
+<?php
+
+foreach ([__DIR__ . '/../../../autoload.php', __DIR__ . '/../vendor/autoload.php'] as $file) {
+    if (file_exists($file)) {
+        require $file;
+        break;
+    }
+}
+
+ini_set('xdebug.max_nesting_level', 3000);
+
+// Disable XDebug var_dump() output truncation
+ini_set('xdebug.var_display_max_children', -1);
+ini_set('xdebug.var_display_max_data', -1);
+ini_set('xdebug.var_display_max_depth', -1);
+
+list($operations, $files, $attributes) = parseArgs($argv);
+
+/* Dump nodes by default */
+if (empty($operations)) {
+    $operations[] = 'dump';
+}
+
+if (empty($files)) {
+    showHelp("Must specify at least one file.");
+}
+
+$lexer = new PhpParser\Lexer\Emulative(array('usedAttributes' => array(
+    'startLine', 'endLine', 'startFilePos', 'endFilePos', 'comments'
+)));
+$parser = (new PhpParser\ParserFactory)->create(
+    PhpParser\ParserFactory::PREFER_PHP7,
+    $lexer
+);
+$dumper = new PhpParser\NodeDumper([
+    'dumpComments' => true,
+    'dumpPositions' => $attributes['with-positions'],
+]);
+$prettyPrinter = new PhpParser\PrettyPrinter\Standard;
+$serializer = new PhpParser\Serializer\XML;
+
+$traverser = new PhpParser\NodeTraverser();
+$traverser->addVisitor(new PhpParser\NodeVisitor\NameResolver);
+
+foreach ($files as $file) {
+    if (strpos($file, '<?php') === 0) {
+        $code = $file;
+        echo "====> Code $code\n";
+    } else {
+        if (!file_exists($file)) {
+            die("File $file does not exist.\n");
+        }
+
+        $code = file_get_contents($file);
+        echo "====> File $file:\n";
+    }
+
+    if ($attributes['with-recovery']) {
+        $errorHandler = new PhpParser\ErrorHandler\Collecting;
+        $stmts = $parser->parse($code, $errorHandler);
+        foreach ($errorHandler->getErrors() as $error) {
+            $message = formatErrorMessage($error, $code, $attributes['with-column-info']);
+            echo $message . "\n";
+        }
+        if (null === $stmts) {
+            continue;
+        }
+    } else {
+        try {
+            $stmts = $parser->parse($code);
+        } catch (PhpParser\Error $error) {
+            $message = formatErrorMessage($error, $code, $attributes['with-column-info']);
+            die($message . "\n");
+        }
+    }
+
+    foreach ($operations as $operation) {
+        if ('dump' === $operation) {
+            echo "==> Node dump:\n";
+            echo $dumper->dump($stmts, $code), "\n";
+        } elseif ('pretty-print' === $operation) {
+            echo "==> Pretty print:\n";
+            echo $prettyPrinter->prettyPrintFile($stmts), "\n";
+        } elseif ('serialize-xml' === $operation) {
+            echo "==> Serialized XML:\n";
+            echo $serializer->serialize($stmts), "\n";
+        } elseif ('var-dump' === $operation) {
+            echo "==> var_dump():\n";
+            var_dump($stmts);
+        } elseif ('resolve-names' === $operation) {
+            echo "==> Resolved names.\n";
+            $stmts = $traverser->traverse($stmts);
+        }
+    }
+}
+
+function formatErrorMessage(PhpParser\Error $e, $code, $withColumnInfo) {
+    if ($withColumnInfo && $e->hasColumnInfo()) {
+        return $e->getMessageWithColumnInfo($code);
+    } else {
+        return $e->getMessage();
+    }
+}
+
+function showHelp($error = '') {
+    if ($error) {
+        echo $error . "\n\n";
+    }
+    die(<<<OUTPUT
+Usage: php-parse [operations] file1.php [file2.php ...]
+   or: php-parse [operations] "<?php code"
+Turn PHP source code into an abstract syntax tree.
+
+Operations is a list of the following options (--dump by default):
+
+    -d, --dump              Dump nodes using NodeDumper
+    -p, --pretty-print      Pretty print file using PrettyPrinter\Standard
+        --serialize-xml     Serialize nodes using Serializer\XML
+        --var-dump          var_dump() nodes (for exact structure)
+    -N, --resolve-names     Resolve names using NodeVisitor\NameResolver
+    -c, --with-column-info  Show column-numbers for errors (if available)
+    -P, --with-positions    Show positions in node dumps
+    -r, --with-recovery     Use parsing with error recovery
+    -h, --help              Display this page
+
+Example:
+    php-parse -d -p -N -d file.php
+
+    Dumps nodes, pretty prints them, then resolves names and dumps them again.
+
+
+OUTPUT
+    );
+}
+
+function parseArgs($args) {
+    $operations = array();
+    $files = array();
+    $attributes = array(
+        'with-column-info' => false,
+        'with-positions' => false,
+        'with-recovery' => false,
+    );
+
+    array_shift($args);
+    $parseOptions = true;
+    foreach ($args as $arg) {
+        if (!$parseOptions) {
+            $files[] = $arg;
+            continue;
+        }
+
+        switch ($arg) {
+            case '--dump':
+            case '-d':
+                $operations[] = 'dump';
+                break;
+            case '--pretty-print':
+            case '-p':
+                $operations[] = 'pretty-print';
+                break;
+            case '--serialize-xml':
+                $operations[] = 'serialize-xml';
+                break;
+            case '--var-dump':
+                $operations[] = 'var-dump';
+                break;
+            case '--resolve-names':
+            case '-N';
+                $operations[] = 'resolve-names';
+                break;
+            case '--with-column-info':
+            case '-c';
+                $attributes['with-column-info'] = true;
+                break;
+            case '--with-positions':
+            case '-P':
+                $attributes['with-positions'] = true;
+                break;
+            case '--with-recovery':
+            case '-r':
+                $attributes['with-recovery'] = true;
+                break;
+            case '--help':
+            case '-h';
+                showHelp();
+                break;
+            case '--':
+                $parseOptions = false;
+                break;
+            default:
+                if ($arg[0] === '-') {
+                    showHelp("Invalid operation $arg.");
+                } else {
+                    $files[] = $arg;
+                }
+        }
+    }
+
+    return array($operations, $files, $attributes);
+}

bin/php-parse.php

@@ -1,142 +0,0 @@
-<?php
-
-require __DIR__ . '/../lib/bootstrap.php';
-
-ini_set('xdebug.max_nesting_level', 2000);
-
-// Disable XDebug var_dump() output truncation
-ini_set('xdebug.var_display_max_children', -1);
-ini_set('xdebug.var_display_max_data', -1);
-ini_set('xdebug.var_display_max_depth', -1);
-
-list($operations, $files) = parseArgs($argv);
-
-/* Dump nodes by default */
-if (empty($operations)) {
-    $operations[] = 'dump';
-}
-
-if (empty($files)) {
-    showHelp("Must specify at least one file.");
-}
-
-$parser = new PhpParser\Parser(new PhpParser\Lexer\Emulative);
-$dumper = new PhpParser\NodeDumper;
-$prettyPrinter = new PhpParser\PrettyPrinter\Standard;
-$serializer = new PhpParser\Serializer\XML;
-
-$traverser = new PhpParser\NodeTraverser();
-$traverser->addVisitor(new PhpParser\NodeVisitor\NameResolver);
-
-foreach ($files as $file) {
-    if (strpos($file, '<?php') === 0) {
-        $code = $file;
-        echo "====> Code $code\n";
-    } else {
-        if (!file_exists($file)) {
-            die("File $file does not exist.\n");
-        }
-
-        $code = file_get_contents($file);
-        echo "====> File $file:\n";
-    }
-
-    try {
-        $stmts = $parser->parse($code);
-    } catch (PhpParser\Error $e) {
-        die("==> Parse Error: {$e->getMessage()}\n");
-    }
-
-    foreach ($operations as $operation) {
-        if ('dump' === $operation) {
-            echo "==> Node dump:\n";
-            echo $dumper->dump($stmts), "\n";
-        } elseif ('pretty-print' === $operation) {
-            echo "==> Pretty print:\n";
-            echo $prettyPrinter->prettyPrintFile($stmts), "\n";
-        } elseif ('serialize-xml' === $operation) {
-            echo "==> Serialized XML:\n";
-            echo $serializer->serialize($stmts), "\n";
-        } elseif ('var-dump' === $operation) {
-            echo "==> var_dump():\n";
-            var_dump($stmts);
-        } elseif ('resolve-names' === $operation) {
-            echo "==> Resolved names.\n";
-            $stmts = $traverser->traverse($stmts);
-        }
-    }
-}
-
-function showHelp($error) {
-    die($error . "\n\n" .
-        <<<OUTPUT
-Usage:
-
-    php php-parse.php [operations] file1.php [file2.php ...]
-
-The file arguments can also be replaced with a code string:
-
-    php php-parse.php [operations] "<?php code"
-
-Operations is a list of the following options (--dump by default):
-
-    --dump           -d  Dump nodes using NodeDumper
-    --pretty-print   -p  Pretty print file using PrettyPrinter\Standard
-    --serialize-xml      Serialize nodes using Serializer\XML
-    --var-dump           var_dump() nodes (for exact structure)
-    --resolve-names  -N  Resolve names using NodeVisitor\NameResolver
-
-Example:
-
-    php php-parse.php -d -p -N -d file.php
-
-    Dumps nodes, pretty prints them, then resolves names and dumps them again.
-OUTPUT
-    );
-}
-
-function parseArgs($args) {
-    $operations = array();
-    $files = array();
-
-    array_shift($args);
-    $parseOptions = true;
-    foreach ($args as $arg) {
-        if (!$parseOptions) {
-            $files[] = $arg;
-            continue;
-        }
-
-        switch ($arg) {
-            case '--dump':
-            case '-d':
-                $operations[] = 'dump';
-                break;
-            case '--pretty-print':
-            case '-p':
-                $operations[] = 'pretty-print';
-                break;
-            case '--serialize-xml':
-                $operations[] = 'serialize-xml';
-                break;
-            case '--var-dump':
-                $operations[] = 'var-dump';
-                break;
-            case '--resolve-names':
-            case '-N';
-                $operations[] = 'resolve-names';
-                break;
-            case '--':
-                $parseOptions = false;
-                break;
-            default:
-                if ($arg[0] === '-') {
-                    showHelp("Invalid operation $arg.");
-                } else {
-                    $files[] = $arg;
-                }
-        }
-    }
-
-    return array($operations, $files);
-}

composer.json

@@ -10,15 +10,21 @@
         }
     ],
     "require": {
-        "php": ">=5.3",
+        "php": ">=5.5",
         "ext-tokenizer": "*"
     },
-    "autoload": {
-        "files": ["lib/bootstrap.php"]
+    "require-dev": {
+        "phpunit/phpunit": "~4.0|~5.0"
     },
+    "autoload": {
+        "psr-4": {
+            "PhpParser\\": "lib/PhpParser"
+        }
+    },
+    "bin": ["bin/php-parse"],
     "extra": {
         "branch-alias": {
-            "dev-master": "1.0-dev"
+            "dev-master": "3.0-dev"
         }
     }
 }

doc/0_Introduction.markdown

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-This project is a PHP 5.2 to PHP 5.6 parser **written in PHP itself**.
+This project is a PHP 5.2 to PHP 7.1 parser **written in PHP itself**.
 
 What is this for?
 -----------------
@@ -18,7 +18,7 @@ For example an AST abstracts away the fact that in PHP variables can be written
 as `$$bar`, `${'foobar'}` or even `${!${''}=barfoo()}`. You don't have to worry about recognizing
 all the different syntaxes from a stream of tokens.
 
-Another questions is: Why would I want to have a PHP parser *written in PHP*? Well, PHP might not be
+Another question is: Why would I want to have a PHP parser *written in PHP*? Well, PHP might not be
 a language especially suited for fast parsing, but processing the AST is much easier in PHP than it
 would be in other, faster languages like C. Furthermore the people most probably wanting to do
 programmatic PHP code analysis are incidentally PHP developers, not C developers.
@@ -26,13 +26,12 @@ programmatic PHP code analysis are incidentally PHP developers, not C developers
 What can it parse?
 ------------------
 
-The parser uses a PHP 5.6 compliant grammar, which is backwards compatible with all PHP version from PHP 5.2
-upwards (and maybe older).
+The parser supports parsing PHP 5.2-5.6 and PHP 7.
 
 As the parser is based on the tokens returned by `token_get_all` (which is only able to lex the PHP
-version it runs on), additionally a wrapper for emulating new tokens from 5.3, 5.4, 5.5 and 5.6 is provided.
-This allows to parse PHP 5.6 source code running on PHP 5.3, for example. This emulation is very hacky and not
-perfect, but it should work well on any sane code.
+version it runs on), additionally a wrapper for emulating tokens from newer versions is provided.
+This allows to parse PHP 7.1 source code running on PHP 5.5, for example. This emulation is somewhat
+hacky and not perfect, but it should work well on any sane code.
 
 What output does it produce?
 ----------------------------

doc/1_Installation.markdown

@@ -1,39 +0,0 @@
-Installation
-============
-
-There are multiple ways to include the PHP parser into your project:
-
-Installing via Composer
------------------------
-
-Create a `composer.json` file in your project root and use it to define your dependencies:
-
-    {
-        "require": {
-            "nikic/php-parser": "~1.0.1"
-        }
-    }
-
-Then install Composer in your project (or [download the composer.phar][1] directly):
-
-    curl -s http://getcomposer.org/installer | php
-
-And finally ask Composer to install the dependencies:
-
-    php composer.phar install
-
-Installing as a Git Submodule
------------------------------
-
-Run the following command to install the parser into the `vendor/PHP-Parser` folder:
-
-    git submodule add git://github.com/nikic/PHP-Parser.git vendor/PHP-Parser
-
-Installing from the Zip- or Tarball
------------------------------------
-
-Download the latest version from [the download page][2], unpack it and move the files somewhere into your project.
-
-
- [1]: http://getcomposer.org/composer.phar
- [2]: https://github.com/nikic/PHP-Parser/tags

doc/2_Usage_of_basic_components.markdown

@@ -6,55 +6,60 @@ This document explains how to use the parser, the pretty printer and the node tr
 Bootstrapping
 -------------
 
-The library needs to register a class autoloader. You can either use the ''vendor/autoload.php'' file generated by
-Composer or by including the bundled `lib/bootstrap.php` file:
+To bootstrap the library, include the autoloader generated by composer:
 
 ```php
-<?php
-require 'path/to/PHP-Parser/lib/bootstrap.php';
-// Or, if you're using Composer:
 require 'path/to/vendor/autoload.php';
 ```
 
 Additionally you may want to set the `xdebug.max_nesting_level` ini option to a higher value:
 
 ```php
-<?php
-ini_set('xdebug.max_nesting_level', 2000);
+ini_set('xdebug.max_nesting_level', 3000);
 ```
 
-This ensures that there will be no errors when traversing highly nested node trees.
+This ensures that there will be no errors when traversing highly nested node trees. However, it is
+preferable to disable XDebug completely, as it can easily make this library more than five times
+slower.
 
 Parsing
 -------
 
-In order to parse some source code you first have to create a `PhpParser\Parser` object, which
-needs to be passed a `PhpParser\Lexer` instance:
+In order to parse code, you first have to create a parser instance:
 
 ```php
-<?php
-
-$parser = new PhpParser\Parser(new PhpParser\Lexer);
-// or
-$parser = new PhpParser\Parser(new PhpParser\Lexer\Emulative);
+use PhpParser\ParserFactory;
+$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
 ```
 
-Use of the emulative lexer is required if you want to parse PHP code from newer versions than the one
-you're running on. For example it will allow you to parse PHP 5.6 code while running on PHP 5.3.
+The factory accepts a kind argument, that determines how different PHP versions are treated:
+
+Kind | Behavior
+-----|---------
+`ParserFactory::PREFER_PHP7` | Try to parse code as PHP 7. If this fails, try to parse it as PHP 5.
+`ParserFactory::PREFER_PHP5` | Try to parse code as PHP 5. If this fails, try to parse it as PHP 7.
+`ParserFactory::ONLY_PHP7` | Parse code as PHP 7.
+`ParserFactory::ONLY_PHP5` | Parse code as PHP 5.
+
+Unless you have strong reason to use something else, `PREFER_PHP7` is a reasonable default.
+
+The `create()` method optionally accepts a `Lexer` instance as the second argument. Some use cases
+that require customized lexers are discussed in the [lexer documentation](component/Lexer.markdown).
 
 Subsequently you can pass PHP code (including the opening `<?php` tag) to the `parse` method in order to
 create a syntax tree. If a syntax error is encountered, an `PhpParser\Error` exception will be thrown:
 
 ```php
-<?php
-$code = '<?php // some code';
+use PhpParser\Error;
+use PhpParser\ParserFactory;
 
-$parser = new PhpParser\Parser(new PhpParser\Lexer\Emulative);
+$code = '<?php // some code';
+$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
 
 try {
     $stmts = $parser->parse($code);
     // $stmts is an array of statement nodes
-} catch (PhpParser\Error $e) {
+} catch (Error $e) {
     echo 'Parse Error: ', $e->getMessage();
 }
 ```
@@ -102,7 +107,7 @@ with them easier they are grouped into three categories:
    and thus can occur in other expressions. Examples of expressions are `$var`
    (`PhpParser\Node\Expr\Variable`) and `func()` (`PhpParser\Node\Expr\FuncCall`).
  * `PhpParser\Node\Scalar`s are nodes representing scalar values, like `'string'`
-   (`PhpParser\Node\Scalar\String`), `0` (`PhpParser\Node\Scalar\LNumber`) or magic constants
+   (`PhpParser\Node\Scalar\String_`), `0` (`PhpParser\Node\Scalar\LNumber`) or magic constants
    like `__FILE__` (`PhpParser\Node\Scalar\MagicConst\File`). All `PhpParser\Node\Scalar`s extend
    `PhpParser\Node\Expr`, as scalars are expressions, too.
  * There are some nodes not in either of these groups, for example names (`PhpParser\Node\Name`)
@@ -137,11 +142,14 @@ information the formatting is done using a specified scheme. Currently there is
 namely `PhpParser\PrettyPrinter\Standard`.
 
 ```php
-<?php
+use PhpParser\Error;
+use PhpParser\ParserFactory;
+use PhpParser\PrettyPrinter;
+
 $code = "<?php echo 'Hi ', hi\\getTarget();";
 
-$parser = new PhpParser\Parser(new PhpParser\Lexer);
-$prettyPrinter = new PhpParser\PrettyPrinter\Standard;
+$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
+$prettyPrinter = new PrettyPrinter\Standard;
 
 try {
     // parse
@@ -158,7 +166,7 @@ try {
     $code = $prettyPrinter->prettyPrint($stmts);
 
     echo $code;
-} catch (PhpParser\Error $e) {
+} catch (Error $e) {
     echo 'Parse Error: ', $e->getMessage();
 }
 ```
@@ -188,11 +196,13 @@ For this purpose the parser provides a component for traversing and visiting the
 structure of a program using this `PhpParser\NodeTraverser` looks like this:
 
 ```php
-<?php
+use PhpParser\NodeTraverser;
+use PhpParser\ParserFactory;
+use PhpParser\PrettyPrinter;
 
-$parser        = new PhpParser\Parser(new PhpParser\Lexer\Emulative);
-$traverser     = new PhpParser\NodeTraverser;
-$prettyPrinter = new PhpParser\PrettyPrinter\Standard;
+$parser        = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
+$traverser     = new NodeTraverser;
+$prettyPrinter = new PrettyPrinter\Standard;
 
 // add your visitor
 $traverser->addVisitor(new MyNodeVisitor);
@@ -218,13 +228,13 @@ try {
 The corresponding node visitor might look like this:
 
 ```php
-<?php
 use PhpParser\Node;
+use PhpParser\NodeVisitorAbstract;
 
-class MyNodeVisitor extends PhpParser\NodeVisitorAbstract
+class MyNodeVisitor extends NodeVisitorAbstract
 {
     public function leaveNode(Node $node) {
-        if ($node instanceof Node\Scalar\String) {
+        if ($node instanceof Node\Scalar\String_) {
             $node->value = 'foo';
         }
     }
@@ -236,28 +246,34 @@ The above node visitor would change all string literals in the program to `'foo'
 All visitors must implement the `PhpParser\NodeVisitor` interface, which defines the following four
 methods:
 
-    public function beforeTraverse(array $nodes);
-    public function enterNode(PhpParser\Node $node);
-    public function leaveNode(PhpParser\Node $node);
-    public function afterTraverse(array $nodes);
+```php
+public function beforeTraverse(array $nodes);
+public function enterNode(\PhpParser\Node $node);
+public function leaveNode(\PhpParser\Node $node);
+public function afterTraverse(array $nodes);
+```
 
-The `beforeTraverse` method is called once before the traversal begins and is passed the nodes the
+The `beforeTraverse()` method is called once before the traversal begins and is passed the nodes the
 traverser was called with. This method can be used for resetting values before traversation or
 preparing the tree for traversal.
 
-The `afterTraverse` method is similar to the `beforeTraverse` method, with the only difference that
+The `afterTraverse()` method is similar to the `beforeTraverse()` method, with the only difference that
 it is called once after the traversal.
 
-The `enterNode` and `leaveNode` methods are called on every node, the former when it is entered,
+The `enterNode()` and `leaveNode()` methods are called on every node, the former when it is entered,
 i.e. before its subnodes are traversed, the latter when it is left.
 
 All four methods can either return the changed node or not return at all (i.e. `null`) in which
-case the current node is not changed. The `leaveNode` method can additionally return two special
-values:
+case the current node is not changed.
 
-If `false` is returned the current node will be removed from the parent array. If an array is returned
-it will be merged into the parent array at the offset of the current node. I.e. if in `array(A, B, C)`
-the node `B` should be replaced with `array(X, Y, Z)` the result will be `array(A, X, Y, Z, C)`.
+The `enterNode()` method can additionally return the value `NodeTraverser::DONT_TRAVERSE_CHILDREN`,
+which instructs the traverser to skip all children of the current node.
+
+The `leaveNode()` method can additionally return the value `NodeTraverser::REMOVE_NODE`, in which
+case the current node will be removed from the parent array. Furthermore it is possible to return
+an array of nodes, which will be merged into the parent array at the offset of the current node.
+I.e. if in `array(A, B, C)` the node `B` should be replaced with `array(X, Y, Z)` the result will
+be `array(A, X, Y, Z, C)`.
 
 Instead of manually implementing the `NodeVisitor` interface you can also extend the `NodeVisitorAbstract`
 class, which will define empty default implementations for all the above methods.
@@ -295,20 +311,24 @@ assume that no dynamic features are used.
 We start off with the following base code:
 
 ```php
-<?php
+use PhpParser\ParserFactory;
+use PhpParser\PrettyPrinter;
+use PhpParser\NodeTraverser;
+use PhpParser\NodeVisitor\NameResolver;
+
 $inDir  = '/some/path';
 $outDir = '/some/other/path';
 
-$parser        = new PhpParser\Parser(new PhpParser\Lexer\Emulative);
-$traverser     = new PhpParser\NodeTraverser;
-$prettyPrinter = new PhpParser\PrettyPrinter\Standard;
+$parser        = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
+$traverser     = new NodeTraverser;
+$prettyPrinter = new PrettyPrinter\Standard;
 
-$traverser->addVisitor(new PhpParser\NodeVisitor\NameResolver); // we will need resolved names
-$traverser->addVisitor(new NodeVisitor\NamespaceConverter);     // our own node visitor
+$traverser->addVisitor(new NameResolver); // we will need resolved names
+$traverser->addVisitor(new NamespaceConverter); // our own node visitor
 
 // iterate over all .php files in the directory
-$files = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($inDir));
-$files = new RegexIterator($files, '/\.php$/');
+$files = new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($inDir));
+$files = new \RegexIterator($files, '/\.php$/');
 
 foreach ($files as $file) {
     try {
@@ -339,9 +359,9 @@ Now lets start with the main code, the `NodeVisitor\NamespaceConverter`. One thi
 is convert `A\\B` style names to `A_B` style ones.
 
 ```php
-<?php
 use PhpParser\Node;
-class NodeVisitor_NamespaceConverter extends PhpParser\NodeVisitorAbstract
+
+class NamespaceConverter extends \PhpParser\NodeVisitorAbstract
 {
     public function leaveNode(Node $node) {
         if ($node instanceof Node\Name) {
@@ -358,14 +378,14 @@ create a name with backslashes either write `$node->toString()` or `(string) $no
 a new name from the string and return it. Returning a new node replaces the old node.
 
 Another thing we need to do is change the class/function/const declarations. Currently they contain
-only the shortname (i.e. the last part of the name), but they need to contain the complete name inclduing
+only the shortname (i.e. the last part of the name), but they need to contain the complete name including
 the namespace prefix:
 
 ```php
-<?php
 use PhpParser\Node;
 use PhpParser\Node\Stmt;
-class NodeVisitor_NamespaceConverter extends PhpParser\NodeVisitorAbstract
+
+class NodeVisitor_NamespaceConverter extends \PhpParser\NodeVisitorAbstract
 {
     public function leaveNode(Node $node) {
         if ($node instanceof Node\Name) {
@@ -388,10 +408,10 @@ There is not much more to it than converting the namespaced name to string with
 The last thing we need to do is remove the `namespace` and `use` statements:
 
 ```php
-<?php
 use PhpParser\Node;
 use PhpParser\Node\Stmt;
-class NodeVisitor_NamespaceConverter extends PhpParser\NodeVisitorAbstract
+
+class NodeVisitor_NamespaceConverter extends \PhpParser\NodeVisitorAbstract
 {
     public function leaveNode(Node $node) {
         if ($node instanceof Node\Name) {
@@ -415,4 +435,4 @@ class NodeVisitor_NamespaceConverter extends PhpParser\NodeVisitorAbstract
 }
 ```
 
-That's all.
+That's all.

doc/3_Other_node_tree_representations.markdown

@@ -8,7 +8,7 @@ Simple serialization
 
 It is possible to serialize the node tree using `serialize()` and also unserialize it using
 `unserialize()`. The output is not human readable and not easily processable from anything
-but PHP, but it is compact and generates fast. The main application thus is in caching.
+but PHP, but it is compact and generates quickly. The main application thus is in caching.
 
 Human readable dumping
 ----------------------
@@ -17,7 +17,6 @@ Furthermore it is possible to dump nodes into a human readable format using the
 `PhpParser\NodeDumper`. This can be used for debugging.
 
 ```php
-<?php
 $code = <<<'CODE'
 <?php
 
@@ -28,7 +27,7 @@ function printLine($msg) {
 printLine('Hello World!!!');
 CODE;
 
-$parser = new PhpParser\Parser(new PhpParser\Lexer);
+$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7);
 $nodeDumper = new PhpParser\NodeDumper;
 
 try {
@@ -87,6 +86,134 @@ array(
 )
 ```
 
+JSON encoding
+-------------
+
+Nodes (and comments) implement the `JsonSerializable` interface. As such, it is possible to JSON
+encode the AST directly using `json_encode()`:
+
+```php
+$code = <<<'CODE'
+<?php
+
+function printLine($msg) {
+    echo $msg, "\n";
+}
+
+printLine('Hello World!!!');
+CODE;
+
+$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7);
+$nodeDumper = new PhpParser\NodeDumper;
+
+try {
+    $stmts = $parser->parse($code);
+
+    echo json_encode($stmts, JSON_PRETTY_PRINT), "\n";
+} catch (PhpParser\Error $e) {
+    echo 'Parse Error: ', $e->getMessage();
+}
+```
+
+This will result in the following output (which includes attributes):
+
+```json
+[
+    {
+        "nodeType": "Stmt_Function",
+        "byRef": false,
+        "name": "printLine",
+        "params": [
+            {
+                "nodeType": "Param",
+                "type": null,
+                "byRef": false,
+                "variadic": false,
+                "name": "msg",
+                "default": null,
+                "attributes": {
+                    "startLine": 3,
+                    "endLine": 3
+                }
+            }
+        ],
+        "returnType": null,
+        "stmts": [
+            {
+                "nodeType": "Stmt_Echo",
+                "exprs": [
+                    {
+                        "nodeType": "Expr_Variable",
+                        "name": "msg",
+                        "attributes": {
+                            "startLine": 4,
+                            "endLine": 4
+                        }
+                    },
+                    {
+                        "nodeType": "Scalar_String",
+                        "value": "\n",
+                        "attributes": {
+                            "startLine": 4,
+                            "endLine": 4,
+                            "kind": 2
+                        }
+                    }
+                ],
+                "attributes": {
+                    "startLine": 4,
+                    "endLine": 4
+                }
+            }
+        ],
+        "attributes": {
+            "startLine": 3,
+            "endLine": 5
+        }
+    },
+    {
+        "nodeType": "Expr_FuncCall",
+        "name": {
+            "nodeType": "Name",
+            "parts": [
+                "printLine"
+            ],
+            "attributes": {
+                "startLine": 7,
+                "endLine": 7
+            }
+        },
+        "args": [
+            {
+                "nodeType": "Arg",
+                "value": {
+                    "nodeType": "Scalar_String",
+                    "value": "Hello World!!!",
+                    "attributes": {
+                        "startLine": 7,
+                        "endLine": 7,
+                        "kind": 1
+                    }
+                },
+                "byRef": false,
+                "unpack": false,
+                "attributes": {
+                    "startLine": 7,
+                    "endLine": 7
+                }
+            }
+        ],
+        "attributes": {
+            "startLine": 7,
+            "endLine": 7
+        }
+    }
+]
+```
+
+There is currently no mechanism to convert JSON back into a node tree. Furthermore, not all ASTs
+can be JSON encoded. In particular, JSON only supports UTF-8 strings.
+
 Serialization to XML
 --------------------
 
@@ -106,7 +233,7 @@ function printLine($msg) {
 printLine('Hello World!!!');
 CODE;
 
-$parser = new PhpParser\Parser(new PhpParser\Lexer);
+$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7);
 $serializer = new PhpParser\Serializer\XML;
 
 try {

doc/4_Code_generation.markdown

@@ -2,52 +2,80 @@ Code generation
 ===============
 
 It is also possible to generate code using the parser, by first creating an Abstract Syntax Tree and then using the
-pretty printer to convert it to PHP code. To simplify code generation, the project comes with a set of builders for
-classes, interfaces, methods, functions, parameters and properties. The builders allow creating node trees using a
-fluid interface, instead of instantiating all nodes manually.
+pretty printer to convert it to PHP code. To simplify code generation, the project comes with builders which allow
+creating node trees using a fluid interface, instead of instantiating all nodes manually. Builders are available for
+the following syntactic elements:
+
+ * namespaces and use statements
+ * classes, interfaces and traits
+ * methods, functions and parameters
+ * properties
 
 Here is an example:
 
 ```php
-<?php
-$factory = new PhpParser\BuilderFactory;
-$node = $factory->class('SomeClass')
-    ->extend('SomeOtherClass')
-    ->implement('A\Few', 'Interfaces')
-    ->makeAbstract() // ->makeFinal()
+use PhpParser\BuilderFactory;
+use PhpParser\PrettyPrinter;
+use PhpParser\Node;
 
-    ->addStmt($factory->method('someMethod')
+$factory = new BuilderFactory;
+$node = $factory->namespace('Name\Space')
+    ->addStmt($factory->use('Some\Other\Thingy')->as('SomeOtherClass'))
+    ->addStmt($factory->class('SomeClass')
+        ->extend('SomeOtherClass')
+        ->implement('A\Few', '\Interfaces')
         ->makeAbstract() // ->makeFinal()
-        ->addParam($factory->param('someParam')->setTypeHint('SomeClass'))
-    )
 
-    ->addStmt($factory->method('anotherMethod')
-        ->makeProtected() // ->makePublic() [default], ->makePrivate()
-        ->addParam($factory->param('someParam')->setDefault('test'))
-        // it is possible to add manually created nodes
-        ->addStmt(new PhpParser\Node\Expr\Print_(new PhpParser\Node\Expr\Variable('someParam')))
-    )
+        ->addStmt($factory->method('someMethod')
+            ->makePublic()
+            ->makeAbstract() // ->makeFinal()
+            ->setReturnType('bool')
+            ->addParam($factory->param('someParam')->setTypeHint('SomeClass'))
+            ->setDocComment('/**
+                              * This method does something.
+                              *
+                              * @param SomeClass And takes a parameter
+                              */')
+        )
 
-    // properties will be correctly reordered above the methods
-    ->addStmt($factory->property('someProperty')->makeProtected())
-    ->addStmt($factory->property('anotherProperty')->makePrivate()->setDefault(array(1, 2, 3)))
+        ->addStmt($factory->method('anotherMethod')
+            ->makeProtected() // ->makePublic() [default], ->makePrivate()
+            ->addParam($factory->param('someParam')->setDefault('test'))
+            // it is possible to add manually created nodes
+            ->addStmt(new Node\Expr\Print_(new Node\Expr\Variable('someParam')))
+        )
+
+        // properties will be correctly reordered above the methods
+        ->addStmt($factory->property('someProperty')->makeProtected())
+        ->addStmt($factory->property('anotherProperty')->makePrivate()->setDefault(array(1, 2, 3)))
+    )
 
     ->getNode()
 ;
 
 $stmts = array($node);
-echo $prettyPrinter->prettyPrint($stmts);
+$prettyPrinter = new PrettyPrinter\Standard();
+echo $prettyPrinter->prettyPrintFile($stmts);
 ```
 
-This will produce the following output with the default pretty printer:
+This will produce the following output with the standard pretty printer:
 
 ```php
 <?php
-abstract class SomeClass extends SomeOtherClass implements A\Few, Interfaces
+
+namespace Name\Space;
+
+use Some\Other\Thingy as SomeClass;
+abstract class SomeClass extends SomeOtherClass implements A\Few, \Interfaces
 {
     protected $someProperty;
     private $anotherProperty = array(1, 2, 3);
-    abstract function someMethod(SomeClass $someParam);
+    /**
+     * This method does something.
+     *
+     * @param SomeClass And takes a parameter
+     */
+    public abstract function someMethod(SomeClass $someParam) : bool;
     protected function anotherMethod($someParam = 'test')
     {
         print $someParam;

doc/component/Error_handling.markdown

@@ -0,0 +1,75 @@
+Error handling
+==============
+
+Errors during parsing or analysis are represented using the `PhpParser\Error` exception class. In addition to an error
+message, an error can also store additional information about the location the error occurred at.
+
+How much location information is available depends on the origin of the error and how many lexer attributes have been
+enabled. At a minimum the start line of the error is usually available.
+
+Column information
+------------------
+
+In order to receive information about not only the line, but also the column span an error occurred at, the file
+position attributes in the lexer need to be enabled:
+
+```php
+$lexer = new PhpParser\Lexer(array(
+    'usedAttributes' => array('comments', 'startLine', 'endLine', 'startFilePos', 'endFilePos'),
+));
+$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7, $lexer);
+
+try {
+    $stmts = $parser->parse($code);
+    // ...
+} catch (PhpParser\Error $e) {
+    // ...
+}
+```
+
+Before using column information its availability needs to be checked with `$e->hasColumnInfo()`, as the precise
+location of an error cannot always be determined. The methods for retrieving column information also have to be passed
+the source code of the parsed file. An example for printing an error:
+
+```php
+if ($e->hasColumnInfo()) {
+    echo $e->getRawMessage() . ' from ' . $e->getStartLine() . ':' . $e->getStartColumn($code)
+        . ' to ' . $e->getEndLine() . ':' . $e->getEndColumn($code);
+    // or:
+    echo $e->getMessageWithColumnInfo();
+} else {
+    echo $e->getMessage();
+}
+```
+
+Both line numbers and column numbers are 1-based. EOF errors will be located at the position one past the end of the
+file.
+
+Error recovery
+--------------
+
+The error behavior of the parser (and other components) is controlled by an `ErrorHandler`. Whenever an error is
+encountered, `ErrorHandler::handleError()` is invoked. The default error handling strategy is `ErrorHandler\Throwing`,
+which will immediately throw when an error is encountered.
+
+To instead collect all encountered errors into an array, while trying to continue parsing the rest of the source code,
+an instance of `ErrorHandler\Collecting` can be passed to the `Parser::parse()` method. A usage example:
+
+```php
+$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::ONLY_PHP7);
+$errorHandler = new PhpParser\ErrorHandler\Collecting;
+
+$stmts = $parser->parse($code, $errorHandler);
+
+if ($errorHandler->hasErrors()) {
+    foreach ($errorHandler->getErrors() as $error) {
+        // $error is an ordinary PhpParser\Error
+    }
+}
+
+if (null !== $stmts) {
+    // $stmts is a best-effort partial AST
+}
+```
+
+The `NameResolver` visitor also accepts an `ErrorHandler` as a constructor argument.

doc/component/Lexer.markdown

@@ -5,83 +5,47 @@ The lexer is responsible for providing tokens to the parser. The project comes w
 `PhpParser\Lexer\Emulative`. The latter is an extension of the former, which adds the ability to emulate tokens of
 newer PHP versions and thus allows parsing of new code on older versions.
 
-A lexer has to define the following public interface:
+This documentation discusses options available for the default lexers and explains how lexers can be extended.
 
-    void startLexing(string $code);
-    string handleHaltCompiler();
-    int getNextToken(string &$value = null, array &$startAttributes = null, array &$endAttributes = null);
+Lexer options
+-------------
 
-The `startLexing()` method is invoked with the source code that is to be lexed (including the opening tag) whenever the
-`parse()` method of the parser is called. It can be used to reset state or preprocess the source code or tokens.
-
-The `handleHaltCompiler()` method is called whenever a `T_HALT_COMPILER` token is encountered. It has to return the
-remaining string after the construct (not including `();`).
-
-The `getNextToken()` method returns the ID of the next token (as defined by the `Parser::T_*` constants). If no more
-tokens are available it must return `0`, which is the ID of the `EOF` token. Furthermore the string content of the
-token should be written into the by-reference `$value` parameter (which will then be available as `$n` in the parser).
-
-Attribute handling
-------------------
-
-The other two by-ref variables `$startAttributes` and `$endAttributes` define which attributes will eventually be
-assigned to the generated nodes: The parser will take the `$startAttributes` from the first token which is part of the
-node and the `$endAttributes` from the last token that is part of the node.
-
-E.g. if the tokens `T_FUNCTION T_STRING ... '{' ... '}'` constitute a node, then the `$startAttributes` from the
-`T_FUNCTION` token will be taken and the `$endAttributes` from the `'}'` token.
-
-By default the lexer creates the attributes `startLine`, `comments` (both part of `$startAttributes`) and `endLine`
-(part of `$endAttributes`).
-
-If you don't want all these attributes to be added (to reduce memory usage of the AST) you can simply remove them by
-overriding the method:
+The two default lexers accept an `$options` array in the constructor. Currently only the `'usedAttributes'` option is
+supported, which allows you to specify which attributes will be added to the AST nodes. The attributes can then be
+accessed using `$node->getAttribute()`, `$node->setAttribute()`, `$node->hasAttribute()` and `$node->getAttributes()`
+methods. A sample options array:
 
 ```php
-<?php
-
-class LessAttributesLexer extends PhpParser\Lexer {
-    public function getNextToken(&$value = null, &$startAttributes = null, &$endAttributes = null) {
-        $tokenId = parent::getNextToken($value, $startAttributes, $endAttributes);
-
-        // only keep startLine attribute
-        unset($startAttributes['comments']);
-        unset($endAttributes['endLine']);
-
-        return $tokenId;
-    }
-}
+$lexer = new PhpParser\Lexer(array(
+    'usedAttributes' => array(
+        'comments', 'startLine', 'endLine'
+    )
+));
 ```
 
-Token offset lexer
-------------------
+The attributes used in this example match the default behavior of the lexer. The following attributes are supported:
 
-A useful application for custom attributes is the token offset lexer, which provides the start and end token for a node
-as attributes:
+ * `comments`: Array of `PhpParser\Comment` or `PhpParser\Comment\Doc` instances, representing all comments that occurred
+   between the previous non-discarded token and the current one. Use of this attribute is required for the
+   `$node->getDocComment()` method to work. The attribute is also needed if you wish the pretty printer to retain
+   comments present in the original code.
+ * `startLine`: Line in which the node starts. This attribute is required for the `$node->getLine()` to work. It is also
+   required if syntax errors should contain line number information.
+ * `endLine`: Line in which the node ends.
+ * `startTokenPos`: Offset into the token array of the first token in the node.
+ * `endTokenPos`: Offset into the token array of the last token in the node.
+ * `startFilePos`: Offset into the code string of the first character that is part of the node.
+ * `endFilePos`: Offset into the code string of the last character that is part of the node.
 
-```php
-<?php
+### Using token positions
 
-class TokenOffsetLexer extends PhpParser\Lexer {
-    public function getNextToken(&$value = null, &$startAttributes = null, &$endAttributes = null) {
-        $tokenId = parent::getNextToken($value, $startAttributes, $endAttributes);
-        $startAttributes['startOffset'] = $endAttributes['endOffset'] = $this->pos;
-        return $tokenId;
-    }
-
-    public function getTokens() {
-        return $this->tokens;
-    }
-}
-```
-
-This information can now be used to examine the exact formatting used for a node. For example the AST does not
-distinguish whether a property was declared using `public` or using `var`, but you can retrieve this information based
-on the token offset:
+The token offset information is useful if you wish to examine the exact formatting used for a node. For example the AST
+does not distinguish whether a property was declared using `public` or using `var`, but you can retrieve this
+information based on the token position:
 
 ```php
 function isDeclaredUsingVar(array $tokens, PhpParser\Node\Stmt\Property $prop) {
-    $i = $prop->getAttribute('startOffset');
+    $i = $prop->getAttribute('startTokenPos');
     return $tokens[$i][0] === T_VAR;
 }
 ```
@@ -103,8 +67,12 @@ class MyNodeVisitor extends PhpParser\NodeVisitorAbstract {
     }
 }
 
-$lexer = new TokenOffsetLexer();
-$parser = new PhpParser\Parser($lexer);
+$lexer = new PhpParser\Lexer(array(
+    'usedAttributes' => array(
+        'comments', 'startLine', 'endLine', 'startTokenPos', 'endTokenPos'
+    )
+));
+$parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7, $lexer);
 
 $visitor = new MyNodeVisitor();
 $traverser = new PhpParser\NodeTraverser();
@@ -121,3 +89,64 @@ try {
 
 The same approach can also be used to perform specific modifications in the code, without changing the formatting in
 other places (which is the case when using the pretty printer).
+
+Lexer extension
+---------------
+
+A lexer has to define the following public interface:
+
+    void startLexing(string $code, ErrorHandler $errorHandler = null);
+    array getTokens();
+    string handleHaltCompiler();
+    int getNextToken(string &$value = null, array &$startAttributes = null, array &$endAttributes = null);
+
+The `startLexing()` method is invoked with the source code that is to be lexed (including the opening tag) whenever the
+`parse()` method of the parser is called. It can be used to reset state or preprocess the source code or tokens. The
+passes `ErrorHandler` should be used to report lexing errors.
+
+The `getTokens()` method returns the current token array, in the usual `token_get_all()` format. This method is not
+used by the parser (which uses `getNextToken()`), but is useful in combination with the token position attributes.
+
+The `handleHaltCompiler()` method is called whenever a `T_HALT_COMPILER` token is encountered. It has to return the
+remaining string after the construct (not including `();`).
+
+The `getNextToken()` method returns the ID of the next token (as defined by the `Parser::T_*` constants). If no more
+tokens are available it must return `0`, which is the ID of the `EOF` token. Furthermore the string content of the
+token should be written into the by-reference `$value` parameter (which will then be available as `$n` in the parser).
+
+### Attribute handling
+
+The other two by-ref variables `$startAttributes` and `$endAttributes` define which attributes will eventually be
+assigned to the generated nodes: The parser will take the `$startAttributes` from the first token which is part of the
+node and the `$endAttributes` from the last token that is part of the node.
+
+E.g. if the tokens `T_FUNCTION T_STRING ... '{' ... '}'` constitute a node, then the `$startAttributes` from the
+`T_FUNCTION` token will be taken and the `$endAttributes` from the `'}'` token.
+
+An application of custom attributes is storing the exact original formatting of literals: While the parser does retain
+some information about the formatting of integers (like decimal vs. hexadecimal) or strings (like used quote type), it
+does not preserve the exact original formatting (e.g. leading zeros for integers or escape sequences in strings). This
+can be remedied by storing the original value in an attribute:
+
+```php
+use PhpParser\Lexer;
+use PhpParser\Parser\Tokens;
+
+class KeepOriginalValueLexer extends Lexer // or Lexer\Emulative
+{
+    public function getNextToken(&$value = null, &$startAttributes = null, &$endAttributes = null) {
+        $tokenId = parent::getNextToken($value, $startAttributes, $endAttributes);
+
+        if ($tokenId == Tokens::T_CONSTANT_ENCAPSED_STRING   // non-interpolated string
+            || $tokenId == Tokens::T_ENCAPSED_AND_WHITESPACE // interpolated string
+            || $tokenId == Tokens::T_LNUMBER                 // integer
+            || $tokenId == Tokens::T_DNUMBER                 // floating point number
+        ) {
+            // could also use $startAttributes, doesn't really matter here
+            $endAttributes['originalValue'] = $value;
+        }
+
+        return $tokenId;
+    }
+}
+```

grammar/README.md

@@ -1,29 +1,28 @@
 What do all those files mean?
 =============================
 
- * `zend_language_parser.phpy`: PHP grammer written in a pseudo language
- * `analyze.php`:               Analyzes the `.phpy`-grammer and outputs some info about it
- * `rebuildParser.php`:         Preprocesses the `.phpy`-grammar and builds the parser using `kmyacc`
- * `kmyacc.php.parser`:         A `kmyacc` parser prototype file for PHP
+ * `php5.y`:             PHP 5 grammar written in a pseudo language
+ * `php7.y`:             PHP 7 grammar written in a pseudo language
+ * `tokens.y`:           Tokens definition shared between PHP 5 and PHP 7 grammars
+ * `parser.template`:    A `kmyacc` parser prototype file for PHP
+ * `tokens.template`:    A `kmyacc` prototype file for the `Tokens` class
+ * `rebuildParsers.php`: Preprocesses the grammar and builds the parser using `kmyacc`
 
 .phpy pseudo language
 =====================
 
-The `.phpy` file is a normal grammer in `kmyacc` (`yacc`) style, with some transformations
+The `.y` file is a normal grammer in `kmyacc` (`yacc`) style, with some transformations
 applied to it:
 
  * Nodes are created using the syntax `Name[..., ...]`. This is transformed into
-   `new Node\Name(..., ..., $attributes)`
- * `Name::abc` is transformed to `Node\Name::abc`
- * Some function-like constructs are resolved (see `rebuildParser.php` for a list)
- * Associative arrays are written as `[key: value, ...]`, which is transformed to
-   `array('key' => value, ...)`
+   `new Name(..., ..., attributes())`
+ * Some function-like constructs are resolved (see `rebuildParsers.php` for a list)
 
 Building the parser
 ===================
 
 In order to rebuild the parser, you need [moriyoshi's fork of kmyacc](https://github.com/moriyoshi/kmyacc-forked).
-After you compiled/installed it, run the `rebuildParser.php` script.
+After you compiled/installed it, run the `rebuildParsers.php` script.
 
-By default only the `Parser.php` is built. If you want to additionally build `Parser/Debug.php` and `y.output` run the
-script with `--debug`. If you want to retain the preprocessed grammar pass `--keep-tmp-grammar`.
+By default only the `Parser.php` is built. If you want to additionally emit debug symbols and create `y.output`, run the
+script with `--debug`. If you want to retain the preprocessed grammar pass `--keep-tmp-grammar`.

grammar/analyze.php

@@ -1,96 +0,0 @@
-<?php
-
-const GRAMMAR_FILE = './zend_language_parser.phpy';
-
-const LIB = '(?(DEFINE)
-    (?<singleQuotedString>\'[^\\\\\']*+(?:\\\\.[^\\\\\']*+)*+\')
-    (?<doubleQuotedString>"[^\\\\"]*+(?:\\\\.[^\\\\"]*+)*+")
-    (?<string>(?&singleQuotedString)|(?&doubleQuotedString))
-    (?<comment>/\*[^*]*+(?:\*(?!/)[^*]*+)*+\*/)
-    (?<code>\{[^\'"/{}]*+(?:(?:(?&string)|(?&comment)|(?&code)|/)[^\'"/{}]*+)*+})
-)';
-
-const RULE_BLOCK = '(?<name>[a-z_]++):(?<rules>[^\'"/{};]*+(?:(?:(?&string)|(?&comment)|(?&code)|/|})[^\'"/{};]*+)*+);';
-
-$usedTerminals = array_flip(array(
-    'T_VARIABLE', 'T_STRING', 'T_INLINE_HTML', 'T_ENCAPSED_AND_WHITESPACE',
-    'T_LNUMBER', 'T_DNUMBER', 'T_CONSTANT_ENCAPSED_STRING', 'T_STRING_VARNAME', 'T_NUM_STRING'
-));
-$unusedNonterminals = array_flip(array(
-    'case_separator', 'optional_comma'
-));
-
-function regex($regex) {
-    return '~' . LIB . '(?:' . str_replace('~', '\~', $regex) . ')~';
-}
-
-function magicSplit($regex, $string) {
-    $pieces = preg_split(regex('(?:(?&string)|(?&comment)|(?&code))(*SKIP)(*FAIL)|' . $regex), $string);
-
-    foreach ($pieces as &$piece) {
-        $piece = trim($piece);
-    }
-
-    return array_filter($pieces);
-}
-
-echo '<pre>';
-
-////////////////////
-////////////////////
-////////////////////
-
-list($defs, $ruleBlocks) = magicSplit('%%', file_get_contents(GRAMMAR_FILE));
-
-if ('' !== trim(preg_replace(regex(RULE_BLOCK), '', $ruleBlocks))) {
-    die('Not all rule blocks were properly recognized!');
-}
-
-preg_match_all(regex(RULE_BLOCK), $ruleBlocks, $ruleBlocksMatches, PREG_SET_ORDER);
-foreach ($ruleBlocksMatches as $match) {
-    $ruleBlockName = $match['name'];
-    $rules = magicSplit('\|', $match['rules']);
-
-    foreach ($rules as &$rule) {
-        $parts = magicSplit('\s+', $rule);
-        $usedParts = array();
-
-        foreach ($parts as $part) {
-            if ('{' === $part[0]) {
-                preg_match_all('~\$([0-9]+)~', $part, $backReferencesMatches, PREG_SET_ORDER);
-                foreach ($backReferencesMatches as $match) {
-                    $usedParts[$match[1]] = true;
-                }
-            }
-        }
-
-        $i = 1;
-        foreach ($parts as &$part) {
-            if ('/' === $part[0]) {
-                continue;
-            }
-
-            if (isset($usedParts[$i])) {
-                if ('\'' === $part[0] || '{' === $part[0]
-                    || (ctype_upper($part[0]) && !isset($usedTerminals[$part]))
-                    || (ctype_lower($part[0]) && isset($unusedNonterminals[$part]))
-                ) {
-                    $part = '<span style="background-color: red; color: white;">' . $part . '</span>';
-                } else {
-                    $part = '<strong><em>' . $part . '</em></strong>';
-                }
-            } elseif ((ctype_upper($part[0]) && isset($usedTerminals[$part]))
-                      || (ctype_lower($part[0]) && !isset($unusedNonterminals[$part]))
-
-            ) {
-                $part = '<span style="background-color: blue; color: white;">' . $part . '</span>';
-            }
-
-            ++$i;
-        }
-
-        $rule = implode(' ', $parts);
-    }
-
-    echo $ruleBlockName, ':', "\n", '      ', implode("\n" . '    | ', $rules), "\n", ';', "\n\n";
-}

grammar/parser.template

@@ -2,35 +2,39 @@
 $meta #
 #semval($) $this->semValue
 #semval($,%t) $this->semValue
-#semval(%n) $this->semStack[$this->stackPos-(%l-%n)]
-#semval(%n,%t) $this->semStack[$this->stackPos-(%l-%n)]
-#include;
+#semval(%n) $this->stackPos-(%l-%n)
+#semval(%n,%t) $this->stackPos-(%l-%n)
 
-namespace PhpParser;
+namespace PhpParser\Parser;
+
+use PhpParser\Error;
+use PhpParser\Node;
+use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
+use PhpParser\Node\Scalar;
+use PhpParser\Node\Stmt;
+#include;
 
 /* This is an automatically GENERATED file, which should not be manually edited.
  * Instead edit one of the following:
- *  * the grammar file grammar/zend_language_parser.phpy
- *  * the skeleton file grammar/kymacc.php.parser
- *  * the preprocessing script grammar/rebuildParser.php
+ *  * the grammar files grammar/php5.y or grammar/php7.y
+ *  * the skeleton file grammar/parser.template
+ *  * the preprocessing script grammar/rebuildParsers.php
  */
-class Parser extends ParserAbstract
+class #(-p) extends \PhpParser\ParserAbstract
 {
     protected $tokenToSymbolMapSize = #(YYMAXLEX);
     protected $actionTableSize = #(YYLAST);
     protected $gotoTableSize = #(YYGLAST);
 
-    protected $invalidToken = #(YYBADCH);
+    protected $invalidSymbol = #(YYBADCH);
+    protected $errorSymbol = #(YYINTERRTOK);
     protected $defaultAction = #(YYDEFAULT);
     protected $unexpectedTokenRule = #(YYUNEXPECTED);
 
     protected $YY2TBLSTATE  = #(YY2TBLSTATE);
     protected $YYNLSTATES   = #(YYNLSTATES);
 
-#tokenval
-    const %s = %n;
-#endtokenval
-
     protected $symbolToName = array(
         #listvar terminals
     );
@@ -86,7 +90,7 @@ class Parser extends ParserAbstract
 #endif
 #reduce
 
-    protected function reduceRule%n($attributes) {
+    protected function reduceRule%n() {
         %b
     }
 #noact

grammar/php5.y

@@ -1,113 +1,7 @@
 %pure_parser
-%expect 2
+%expect 6
 
-%left T_INCLUDE T_INCLUDE_ONCE T_EVAL T_REQUIRE T_REQUIRE_ONCE
-%left ','
-%left T_LOGICAL_OR
-%left T_LOGICAL_XOR
-%left T_LOGICAL_AND
-%right T_PRINT
-%right T_YIELD
-%left '=' T_PLUS_EQUAL T_MINUS_EQUAL T_MUL_EQUAL T_DIV_EQUAL T_CONCAT_EQUAL T_MOD_EQUAL T_AND_EQUAL T_OR_EQUAL T_XOR_EQUAL T_SL_EQUAL T_SR_EQUAL T_POW_EQUAL
-%left '?' ':'
-%left T_BOOLEAN_OR
-%left T_BOOLEAN_AND
-%left '|'
-%left '^'
-%left '&'
-%nonassoc T_IS_EQUAL T_IS_NOT_EQUAL T_IS_IDENTICAL T_IS_NOT_IDENTICAL
-%nonassoc '<' T_IS_SMALLER_OR_EQUAL '>' T_IS_GREATER_OR_EQUAL
-%left T_SL T_SR
-%left '+' '-' '.'
-%left '*' '/' '%'
-%right '!'
-%nonassoc T_INSTANCEOF
-%right '~' T_INC T_DEC T_INT_CAST T_DOUBLE_CAST T_STRING_CAST T_ARRAY_CAST T_OBJECT_CAST T_BOOL_CAST T_UNSET_CAST '@'
-%right T_POW
-%right '['
-%nonassoc T_NEW T_CLONE
-%token T_EXIT
-%token T_IF
-%left T_ELSEIF
-%left T_ELSE
-%left T_ENDIF
-%token T_LNUMBER
-%token T_DNUMBER
-%token T_STRING
-%token T_STRING_VARNAME
-%token T_VARIABLE
-%token T_NUM_STRING
-%token T_INLINE_HTML
-%token T_CHARACTER
-%token T_BAD_CHARACTER
-%token T_ENCAPSED_AND_WHITESPACE
-%token T_CONSTANT_ENCAPSED_STRING
-%token T_ECHO
-%token T_DO
-%token T_WHILE
-%token T_ENDWHILE
-%token T_FOR
-%token T_ENDFOR
-%token T_FOREACH
-%token T_ENDFOREACH
-%token T_DECLARE
-%token T_ENDDECLARE
-%token T_AS
-%token T_SWITCH
-%token T_ENDSWITCH
-%token T_CASE
-%token T_DEFAULT
-%token T_BREAK
-%token T_CONTINUE
-%token T_GOTO
-%token T_FUNCTION
-%token T_CONST
-%token T_RETURN
-%token T_TRY
-%token T_CATCH
-%token T_FINALLY
-%token T_THROW
-%token T_USE
-%token T_INSTEADOF
-%token T_GLOBAL
-%right T_STATIC T_ABSTRACT T_FINAL T_PRIVATE T_PROTECTED T_PUBLIC
-%token T_VAR
-%token T_UNSET
-%token T_ISSET
-%token T_EMPTY
-%token T_HALT_COMPILER
-%token T_CLASS
-%token T_TRAIT
-%token T_INTERFACE
-%token T_EXTENDS
-%token T_IMPLEMENTS
-%token T_OBJECT_OPERATOR
-%token T_DOUBLE_ARROW
-%token T_LIST
-%token T_ARRAY
-%token T_CALLABLE
-%token T_CLASS_C
-%token T_TRAIT_C
-%token T_METHOD_C
-%token T_FUNC_C
-%token T_LINE
-%token T_FILE
-%token T_COMMENT
-%token T_DOC_COMMENT
-%token T_OPEN_TAG
-%token T_OPEN_TAG_WITH_ECHO
-%token T_CLOSE_TAG
-%token T_WHITESPACE
-%token T_START_HEREDOC
-%token T_END_HEREDOC
-%token T_DOLLAR_OPEN_CURLY_BRACES
-%token T_CURLY_OPEN
-%token T_PAAMAYIM_NEKUDOTAYIM
-%token T_NAMESPACE
-%token T_NS_C
-%token T_DIR
-%token T_NS_SEPARATOR
-%token T_ELLIPSIS
+%tokens
 
 %%
 
@@ -115,11 +9,37 @@ start:
     top_statement_list                                      { $$ = $this->handleNamespaces($1); }
 ;
 
-top_statement_list:
-      top_statement_list top_statement                      { pushNormalizing($1, $2); }
+top_statement_list_ex:
+      top_statement_list_ex top_statement                   { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+top_statement_list:
+      top_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
+reserved_non_modifiers:
+      T_INCLUDE | T_INCLUDE_ONCE | T_EVAL | T_REQUIRE | T_REQUIRE_ONCE | T_LOGICAL_OR | T_LOGICAL_XOR | T_LOGICAL_AND
+    | T_INSTANCEOF | T_NEW | T_CLONE | T_EXIT | T_IF | T_ELSEIF | T_ELSE | T_ENDIF | T_ECHO | T_DO | T_WHILE
+    | T_ENDWHILE | T_FOR | T_ENDFOR | T_FOREACH | T_ENDFOREACH | T_DECLARE | T_ENDDECLARE | T_AS | T_TRY | T_CATCH
+    | T_FINALLY | T_THROW | T_USE | T_INSTEADOF | T_GLOBAL | T_VAR | T_UNSET | T_ISSET | T_EMPTY | T_CONTINUE | T_GOTO
+    | T_FUNCTION | T_CONST | T_RETURN | T_PRINT | T_YIELD | T_LIST | T_SWITCH | T_ENDSWITCH | T_CASE | T_DEFAULT
+    | T_BREAK | T_ARRAY | T_CALLABLE | T_EXTENDS | T_IMPLEMENTS | T_NAMESPACE | T_TRAIT | T_INTERFACE | T_CLASS
+    | T_CLASS_C | T_TRAIT_C | T_FUNC_C | T_METHOD_C | T_LINE | T_FILE | T_DIR | T_NS_C | T_HALT_COMPILER
+;
+
+semi_reserved:
+      reserved_non_modifiers
+    | T_STATIC | T_ABSTRACT | T_FINAL | T_PRIVATE | T_PROTECTED | T_PUBLIC
+;
+
+identifier:
+      T_STRING                                              { $$ = $1; }
+    | semi_reserved                                         { $$ = $1; }
+;
+
 namespace_name_parts:
       T_STRING                                              { init($1); }
     | namespace_name_parts T_NS_SEPARATOR T_STRING          { push($1, $3); }
@@ -135,25 +55,66 @@ top_statement:
     | class_declaration_statement                           { $$ = $1; }
     | T_HALT_COMPILER
           { $$ = Stmt\HaltCompiler[$this->lexer->handleHaltCompiler()]; }
-    | T_NAMESPACE namespace_name ';'                        { $$ = Stmt\Namespace_[$2, null]; }
-    | T_NAMESPACE namespace_name '{' top_statement_list '}' { $$ = Stmt\Namespace_[$2, $4]; }
-    | T_NAMESPACE '{' top_statement_list '}'                { $$ = Stmt\Namespace_[null,     $3]; }
+    | T_NAMESPACE namespace_name ';'
+          { $$ = Stmt\Namespace_[$2, null]; $this->checkNamespace($$); }
+    | T_NAMESPACE namespace_name '{' top_statement_list '}'
+          { $$ = Stmt\Namespace_[$2, $4]; $this->checkNamespace($$); }
+    | T_NAMESPACE '{' top_statement_list '}'
+          { $$ = Stmt\Namespace_[null, $3]; $this->checkNamespace($$); }
     | T_USE use_declarations ';'                            { $$ = Stmt\Use_[$2, Stmt\Use_::TYPE_NORMAL]; }
-    | T_USE T_FUNCTION use_declarations ';'                 { $$ = Stmt\Use_[$3, Stmt\Use_::TYPE_FUNCTION]; }
-    | T_USE T_CONST use_declarations ';'                    { $$ = Stmt\Use_[$3, Stmt\Use_::TYPE_CONSTANT]; }
+    | T_USE use_type use_declarations ';'                   { $$ = Stmt\Use_[$3, $2]; }
+    | group_use_declaration ';'                             { $$ = $1; }
     | T_CONST constant_declaration_list ';'                 { $$ = Stmt\Const_[$2]; }
 ;
 
+use_type:
+      T_FUNCTION                                            { $$ = Stmt\Use_::TYPE_FUNCTION; }
+    | T_CONST                                               { $$ = Stmt\Use_::TYPE_CONSTANT; }
+;
+
+/* Using namespace_name_parts here to avoid s/r conflict on T_NS_SEPARATOR */
+group_use_declaration:
+      T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, $2]; }
+    | T_USE use_type T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+          { $$ = Stmt\GroupUse[new Name($4, stackAttributes(#4)), $7, $2]; }
+    | T_USE namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
+          { $$ = Stmt\GroupUse[new Name($2, stackAttributes(#2)), $5, Stmt\Use_::TYPE_UNKNOWN]; }
+    | T_USE T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
+          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, Stmt\Use_::TYPE_UNKNOWN]; }
+;
+
+unprefixed_use_declarations:
+      unprefixed_use_declarations ',' unprefixed_use_declaration
+          { push($1, $3); }
+    | unprefixed_use_declaration                            { init($1); }
+;
+
 use_declarations:
       use_declarations ',' use_declaration                  { push($1, $3); }
     | use_declaration                                       { init($1); }
 ;
 
+inline_use_declarations:
+      inline_use_declarations ',' inline_use_declaration    { push($1, $3); }
+    | inline_use_declaration                                { init($1); }
+;
+
+unprefixed_use_declaration:
+      namespace_name
+          { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #1); }
+    | namespace_name T_AS T_STRING
+          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #3); }
+;
+
 use_declaration:
-      namespace_name                                        { $$ = Stmt\UseUse[$1, null]; }
-    | namespace_name T_AS T_STRING                          { $$ = Stmt\UseUse[$1, $3]; }
-    | T_NS_SEPARATOR namespace_name                         { $$ = Stmt\UseUse[$2, null]; }
-    | T_NS_SEPARATOR namespace_name T_AS T_STRING           { $$ = Stmt\UseUse[$2, $4]; }
+      unprefixed_use_declaration                            { $$ = $1; }
+    | T_NS_SEPARATOR unprefixed_use_declaration             { $$ = $2; }
+;
+
+inline_use_declaration:
+      unprefixed_use_declaration                            { $$ = $1; $$->type = Stmt\Use_::TYPE_NORMAL; }
+    | use_type unprefixed_use_declaration                   { $$ = $2; $$->type = $1; }
 ;
 
 constant_declaration_list:
@@ -162,31 +123,47 @@ constant_declaration_list:
 ;
 
 constant_declaration:
-    T_STRING '=' static_scalar                              { $$ = Const_[$1, $3]; }
+    T_STRING '=' static_scalar                              { $$ = Node\Const_[$1, $3]; }
+;
+
+class_const_list:
+      class_const_list ',' class_const                      { push($1, $3); }
+    | class_const                                           { init($1); }
+;
+
+class_const:
+    identifier '=' static_scalar                            { $$ = Node\Const_[$1, $3]; }
+;
+
+inner_statement_list_ex:
+      inner_statement_list_ex inner_statement               { pushNormalizing($1, $2); }
+    | /* empty */                                           { init(); }
 ;
 
 inner_statement_list:
-      inner_statement_list inner_statement                  { pushNormalizing($1, $2); }
-    | /* empty */                                           { init(); }
+      inner_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
 ;
 
 inner_statement:
       statement                                             { $$ = $1; }
     | function_declaration_statement                        { $$ = $1; }
     | class_declaration_statement                           { $$ = $1; }
-    | T_HALT_COMPILER                                       { error('__HALT_COMPILER() can only be used from the outermost scope'); }
+    | T_HALT_COMPILER
+          { throw new Error('__HALT_COMPILER() can only be used from the outermost scope', attributes()); }
 ;
 
-statement:
-      '{' inner_statement_list '}'                          { $$ = $2; }
+non_empty_statement:
+      '{' inner_statement_list '}'                          { $$ = $2; prependLeadingComments($$); }
     | T_IF parentheses_expr statement elseif_list else_single
-          { $$ = Stmt\If_[$2, [stmts: toArray($3), elseifs: $4, else: $5]]; }
+          { $$ = Stmt\If_[$2, ['stmts' => toArray($3), 'elseifs' => $4, 'else' => $5]]; }
     | T_IF parentheses_expr ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
-          { $$ = Stmt\If_[$2, [stmts: $4, elseifs: $5, else: $6]]; }
+          { $$ = Stmt\If_[$2, ['stmts' => $4, 'elseifs' => $5, 'else' => $6]]; }
     | T_WHILE parentheses_expr while_statement              { $$ = Stmt\While_[$2, $3]; }
     | T_DO statement T_WHILE parentheses_expr ';'           { $$ = Stmt\Do_   [$4, toArray($2)]; }
     | T_FOR '(' for_expr ';'  for_expr ';' for_expr ')' for_statement
-          { $$ = Stmt\For_[[init: $3, cond: $5, loop: $7, stmts: $9]]; }
+          { $$ = Stmt\For_[['init' => $3, 'cond' => $5, 'loop' => $7, 'stmts' => $9]]; }
     | T_SWITCH parentheses_expr switch_case_list            { $$ = Stmt\Switch_[$2, $3]; }
     | T_BREAK ';'                                           { $$ = Stmt\Break_[null]; }
     | T_BREAK expr ';'                                      { $$ = Stmt\Break_[$2]; }
@@ -202,16 +179,24 @@ statement:
     | expr ';'                                              { $$ = $1; }
     | T_UNSET '(' variables_list ')' ';'                    { $$ = Stmt\Unset_[$3]; }
     | T_FOREACH '(' expr T_AS foreach_variable ')' foreach_statement
-          { $$ = Stmt\Foreach_[$3, $5[0], [keyVar: null, byRef: $5[1], stmts: $7]]; }
+          { $$ = Stmt\Foreach_[$3, $5[0], ['keyVar' => null, 'byRef' => $5[1], 'stmts' => $7]]; }
     | T_FOREACH '(' expr T_AS variable T_DOUBLE_ARROW foreach_variable ')' foreach_statement
-          { $$ = Stmt\Foreach_[$3, $7[0], [keyVar: $5, byRef: $7[1], stmts: $9]]; }
+          { $$ = Stmt\Foreach_[$3, $7[0], ['keyVar' => $5, 'byRef' => $7[1], 'stmts' => $9]]; }
     | T_DECLARE '(' declare_list ')' declare_statement      { $$ = Stmt\Declare_[$3, $5]; }
-    | ';'                                                   { $$ = array(); /* means: no statement */ }
     | T_TRY '{' inner_statement_list '}' catches optional_finally
-          { $$ = Stmt\TryCatch[$3, $5, $6]; }
+          { $$ = Stmt\TryCatch[$3, $5, $6]; $this->checkTryCatch($$); }
     | T_THROW expr ';'                                      { $$ = Stmt\Throw_[$2]; }
     | T_GOTO T_STRING ';'                                   { $$ = Stmt\Goto_[$2]; }
     | T_STRING ':'                                          { $$ = Stmt\Label[$1]; }
+    | expr error                                            { $$ = $1; }
+    | error                                                 { $$ = array(); /* means: no statement */ }
+;
+
+statement:
+      non_empty_statement                                   { $$ = $1; }
+    | ';'
+          { makeNop($$, $this->startAttributeStack[#1]);
+            if ($$ === null) $$ = array(); /* means: no statement */ }
 ;
 
 catches:
@@ -221,12 +206,12 @@ catches:
 
 catch:
     T_CATCH '(' name T_VARIABLE ')' '{' inner_statement_list '}'
-        { $$ = Stmt\Catch_[$3, parseVar($4), $7]; }
+        { $$ = Stmt\Catch_[array($3), parseVar($4), $7]; }
 ;
 
 optional_finally:
       /* empty */                                           { $$ = null; }
-    | T_FINALLY '{' inner_statement_list '}'                { $$ = $3; }
+    | T_FINALLY '{' inner_statement_list '}'                { $$ = Stmt\Finally_[$3]; }
 ;
 
 variables_list:
@@ -245,17 +230,19 @@ optional_ellipsis:
 ;
 
 function_declaration_statement:
-    T_FUNCTION optional_ref T_STRING '(' parameter_list ')' '{' inner_statement_list '}'
-        { $$ = Stmt\Function_[$3, [byRef: $2, params: $5, stmts: $8]]; }
+    T_FUNCTION optional_ref T_STRING '(' parameter_list ')' optional_return_type '{' inner_statement_list '}'
+        { $$ = Stmt\Function_[$3, ['byRef' => $2, 'params' => $5, 'returnType' => $7, 'stmts' => $9]]; }
 ;
 
 class_declaration_statement:
       class_entry_type T_STRING extends_from implements_list '{' class_statement_list '}'
-          { $$ = Stmt\Class_[$2, [type: $1, extends: $3, implements: $4, stmts: $6]]; }
+          { $$ = Stmt\Class_[$2, ['type' => $1, 'extends' => $3, 'implements' => $4, 'stmts' => $6]];
+            $this->checkClass($$, #2); }
     | T_INTERFACE T_STRING interface_extends_list '{' class_statement_list '}'
-          { $$ = Stmt\Interface_[$2, [extends: $3, stmts: $5]]; }
+          { $$ = Stmt\Interface_[$2, ['extends' => $3, 'stmts' => $5]];
+            $this->checkInterface($$, #2); }
     | T_TRAIT T_STRING '{' class_statement_list '}'
-          { $$ = Stmt\Trait_[$2, $4]; }
+          { $$ = Stmt\Trait_[$2, ['stmts' => $4]]; }
 ;
 
 class_entry_type:
@@ -266,22 +253,22 @@ class_entry_type:
 
 extends_from:
       /* empty */                                           { $$ = null; }
-    | T_EXTENDS name                                        { $$ = $2; }
+    | T_EXTENDS class_name                                  { $$ = $2; }
 ;
 
 interface_extends_list:
       /* empty */                                           { $$ = array(); }
-    | T_EXTENDS name_list                                   { $$ = $2; }
+    | T_EXTENDS class_name_list                             { $$ = $2; }
 ;
 
 implements_list:
       /* empty */                                           { $$ = array(); }
-    | T_IMPLEMENTS name_list                                { $$ = $2; }
+    | T_IMPLEMENTS class_name_list                          { $$ = $2; }
 ;
 
-name_list:
-      name                                                  { init($1); }
-    | name_list ',' name                                    { push($1, $3); }
+class_name_list:
+      class_name                                            { init($1); }
+    | class_name_list ',' class_name                        { push($1, $3); }
 ;
 
 for_statement:
@@ -295,7 +282,8 @@ foreach_statement:
 ;
 
 declare_statement:
-      statement                                             { $$ = toArray($1); }
+      non_empty_statement                                   { $$ = toArray($1); }
+    | ';'                                                   { $$ = null; }
     | ':' inner_statement_list T_ENDDECLARE ';'             { $$ = $2; }
 ;
 
@@ -380,23 +368,32 @@ non_empty_parameter_list:
 ;
 
 parameter:
-      optional_class_type optional_ref optional_ellipsis T_VARIABLE
-          { $$ = Param[parseVar($4), null, $1, $2, $3]; }
-    | optional_class_type optional_ref optional_ellipsis T_VARIABLE '=' static_scalar
-          { $$ = Param[parseVar($4), $6, $1, $2, $3]; }
+      optional_param_type optional_ref optional_ellipsis T_VARIABLE
+          { $$ = Node\Param[parseVar($4), null, $1, $2, $3]; $this->checkParam($$); }
+    | optional_param_type optional_ref optional_ellipsis T_VARIABLE '=' static_scalar
+          { $$ = Node\Param[parseVar($4), $6, $1, $2, $3]; $this->checkParam($$); }
 ;
 
-optional_class_type:
-      /* empty */                                           { $$ = null; }
-    | name                                                  { $$ = $1; }
+type:
+      name                                                  { $$ = $1; }
     | T_ARRAY                                               { $$ = 'array'; }
     | T_CALLABLE                                            { $$ = 'callable'; }
 ;
 
+optional_param_type:
+      /* empty */                                           { $$ = null; }
+    | type                                                  { $$ = $1; }
+;
+
+optional_return_type:
+      /* empty */                                           { $$ = null; }
+    | ':' type                                              { $$ = $2; }
+;
+
 argument_list:
       '(' ')'                                               { $$ = array(); }
     | '(' non_empty_argument_list ')'                       { $$ = $2; }
-    | '(' yield_expr ')'                                    { $$ = array(Arg[$2, false, false]); }
+    | '(' yield_expr ')'                                    { $$ = array(Node\Arg[$2, false, false]); }
 ;
 
 non_empty_argument_list:
@@ -405,9 +402,9 @@ non_empty_argument_list:
 ;
 
 argument:
-      expr                                                  { $$ = Arg[$1, false, false]; }
-    | '&' variable                                          { $$ = Arg[$2, true, false]; }
-    | T_ELLIPSIS expr                                       { $$ = Arg[$2, false, true]; }
+      expr                                                  { $$ = Node\Arg[$1, false, false]; }
+    | '&' variable                                          { $$ = Node\Arg[$2, true, false]; }
+    | T_ELLIPSIS expr                                       { $$ = Node\Arg[$2, false, true]; }
 ;
 
 global_var_list:
@@ -437,11 +434,13 @@ class_statement_list:
 ;
 
 class_statement:
-      variable_modifiers property_declaration_list ';'      { $$ = Stmt\Property[$1, $2]; }
-    | T_CONST constant_declaration_list ';'                 { $$ = Stmt\ClassConst[$2]; }
-    | method_modifiers T_FUNCTION optional_ref T_STRING '(' parameter_list ')' method_body
-          { $$ = Stmt\ClassMethod[$4, [type: $1, byRef: $3, params: $6, stmts: $8]]; }
-    | T_USE name_list trait_adaptations                     { $$ = Stmt\TraitUse[$2, $3]; }
+      variable_modifiers property_declaration_list ';'
+          { $$ = Stmt\Property[$1, $2]; $this->checkProperty($$, #1); }
+    | T_CONST class_const_list ';'                          { $$ = Stmt\ClassConst[$2, 0]; }
+    | method_modifiers T_FUNCTION optional_ref identifier '(' parameter_list ')' optional_return_type method_body
+          { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]];
+            $this->checkClassMethod($$, #1); }
+    | T_USE class_name_list trait_adaptations               { $$ = Stmt\TraitUse[$2, $3]; }
 ;
 
 trait_adaptations:
@@ -455,22 +454,24 @@ trait_adaptation_list:
 ;
 
 trait_adaptation:
-      trait_method_reference_fully_qualified T_INSTEADOF name_list ';'
+      trait_method_reference_fully_qualified T_INSTEADOF class_name_list ';'
           { $$ = Stmt\TraitUseAdaptation\Precedence[$1[0], $1[1], $3]; }
-    | trait_method_reference T_AS member_modifier T_STRING ';'
+    | trait_method_reference T_AS member_modifier identifier ';'
           { $$ = Stmt\TraitUseAdaptation\Alias[$1[0], $1[1], $3, $4]; }
     | trait_method_reference T_AS member_modifier ';'
           { $$ = Stmt\TraitUseAdaptation\Alias[$1[0], $1[1], $3, null]; }
     | trait_method_reference T_AS T_STRING ';'
           { $$ = Stmt\TraitUseAdaptation\Alias[$1[0], $1[1], null, $3]; }
+    | trait_method_reference T_AS reserved_non_modifiers ';'
+          { $$ = Stmt\TraitUseAdaptation\Alias[$1[0], $1[1], null, $3]; }
 ;
 
 trait_method_reference_fully_qualified:
-      name T_PAAMAYIM_NEKUDOTAYIM T_STRING                  { $$ = array($1, $3); }
+      name T_PAAMAYIM_NEKUDOTAYIM identifier                { $$ = array($1, $3); }
 ;
 trait_method_reference:
       trait_method_reference_fully_qualified                { $$ = $1; }
-    | T_STRING                                              { $$ = array(null, $1); }
+    | identifier                                            { $$ = array(null, $1); }
 ;
 
 method_body:
@@ -480,17 +481,17 @@ method_body:
 
 variable_modifiers:
       non_empty_member_modifiers                            { $$ = $1; }
-    | T_VAR                                                 { $$ = Stmt\Class_::MODIFIER_PUBLIC; }
+    | T_VAR                                                 { $$ = 0; }
 ;
 
 method_modifiers:
-      /* empty */                                           { $$ = Stmt\Class_::MODIFIER_PUBLIC; }
+      /* empty */                                           { $$ = 0; }
     | non_empty_member_modifiers                            { $$ = $1; }
 ;
 
 non_empty_member_modifiers:
       member_modifier                                       { $$ = $1; }
-    | non_empty_member_modifiers member_modifier            { Stmt\Class_::verifyModifier($1, $2); $$ = $1 | $2; }
+    | non_empty_member_modifiers member_modifier            { $this->checkModifier($1, $2, #2); $$ = $1 | $2; }
 ;
 
 member_modifier:
@@ -571,6 +572,7 @@ expr:
     | expr T_IS_NOT_IDENTICAL expr                          { $$ = Expr\BinaryOp\NotIdentical  [$1, $3]; }
     | expr T_IS_EQUAL expr                                  { $$ = Expr\BinaryOp\Equal         [$1, $3]; }
     | expr T_IS_NOT_EQUAL expr                              { $$ = Expr\BinaryOp\NotEqual      [$1, $3]; }
+    | expr T_SPACESHIP expr                                 { $$ = Expr\BinaryOp\Spaceship     [$1, $3]; }
     | expr '<' expr                                         { $$ = Expr\BinaryOp\Smaller       [$1, $3]; }
     | expr T_IS_SMALLER_OR_EQUAL expr                       { $$ = Expr\BinaryOp\SmallerOrEqual[$1, $3]; }
     | expr '>' expr                                         { $$ = Expr\BinaryOp\Greater       [$1, $3]; }
@@ -581,6 +583,7 @@ expr:
     | '(' new_expr ')'                                      { $$ = $2; }
     | expr '?' expr ':' expr                                { $$ = Expr\Ternary[$1, $3,   $5]; }
     | expr '?' ':' expr                                     { $$ = Expr\Ternary[$1, null, $4]; }
+    | expr T_COALESCE expr                                  { $$ = Expr\BinaryOp\Coalesce[$1, $3]; }
     | T_ISSET '(' variables_list ')'                        { $$ = Expr\Isset_[$3]; }
     | T_EMPTY '(' expr ')'                                  { $$ = Expr\Empty_[$3]; }
     | T_INCLUDE expr                                        { $$ = Expr\Include_[$2, Expr\Include_::TYPE_INCLUDE]; }
@@ -588,14 +591,17 @@ expr:
     | T_EVAL parentheses_expr                               { $$ = Expr\Eval_[$2]; }
     | T_REQUIRE expr                                        { $$ = Expr\Include_[$2, Expr\Include_::TYPE_REQUIRE]; }
     | T_REQUIRE_ONCE expr                                   { $$ = Expr\Include_[$2, Expr\Include_::TYPE_REQUIRE_ONCE]; }
-    | T_INT_CAST expr                                       { $$ = Expr\Cast\Int     [$2]; }
+    | T_INT_CAST expr                                       { $$ = Expr\Cast\Int_    [$2]; }
     | T_DOUBLE_CAST expr                                    { $$ = Expr\Cast\Double  [$2]; }
-    | T_STRING_CAST expr                                    { $$ = Expr\Cast\String  [$2]; }
+    | T_STRING_CAST expr                                    { $$ = Expr\Cast\String_ [$2]; }
     | T_ARRAY_CAST expr                                     { $$ = Expr\Cast\Array_  [$2]; }
-    | T_OBJECT_CAST expr                                    { $$ = Expr\Cast\Object  [$2]; }
-    | T_BOOL_CAST expr                                      { $$ = Expr\Cast\Bool    [$2]; }
+    | T_OBJECT_CAST expr                                    { $$ = Expr\Cast\Object_ [$2]; }
+    | T_BOOL_CAST expr                                      { $$ = Expr\Cast\Bool_   [$2]; }
     | T_UNSET_CAST expr                                     { $$ = Expr\Cast\Unset_  [$2]; }
-    | T_EXIT exit_expr                                      { $$ = Expr\Exit_        [$2]; }
+    | T_EXIT exit_expr
+          { $attrs = attributes();
+            $attrs['kind'] = strtolower($1) === 'exit' ? Expr\Exit_::KIND_EXIT : Expr\Exit_::KIND_DIE;
+            $$ = new Expr\Exit_($2, $attrs); }
     | '@' expr                                              { $$ = Expr\ErrorSuppress[$2]; }
     | scalar                                                { $$ = $1; }
     | array_expr                                            { $$ = $1; }
@@ -603,10 +609,13 @@ expr:
     | '`' backticks_expr '`'                                { $$ = Expr\ShellExec[$2]; }
     | T_PRINT expr                                          { $$ = Expr\Print_[$2]; }
     | T_YIELD                                               { $$ = Expr\Yield_[null, null]; }
-    | T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars '{' inner_statement_list '}'
-          { $$ = Expr\Closure[[static: false, byRef: $2, params: $4, uses: $6, stmts: $8]]; }
-    | T_STATIC T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars '{' inner_statement_list '}'
-          { $$ = Expr\Closure[[static: true, byRef: $3, params: $5, uses: $7, stmts: $9]]; }
+    | T_YIELD_FROM expr                                     { $$ = Expr\YieldFrom[$2]; }
+    | T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars optional_return_type
+      '{' inner_statement_list '}'
+          { $$ = Expr\Closure[['static' => false, 'byRef' => $2, 'params' => $4, 'uses' => $6, 'returnType' => $7, 'stmts' => $9]]; }
+    | T_STATIC T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars optional_return_type
+      '{' inner_statement_list '}'
+          { $$ = Expr\Closure[['static' => true, 'byRef' => $3, 'params' => $5, 'uses' => $7, 'returnType' => $8, 'stmts' => $10]]; }
 ;
 
 parentheses_expr:
@@ -620,21 +629,33 @@ yield_expr:
 ;
 
 array_expr:
-      T_ARRAY '(' array_pair_list ')'                       { $$ = Expr\Array_[$3]; }
-    | '[' array_pair_list ']'                               { $$ = Expr\Array_[$2]; }
+      T_ARRAY '(' array_pair_list ')'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_LONG;
+            $$ = new Expr\Array_($3, $attrs); }
+    | '[' array_pair_list ']'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_SHORT;
+            $$ = new Expr\Array_($2, $attrs); }
 ;
 
 scalar_dereference:
       array_expr '[' dim_offset ']'                         { $$ = Expr\ArrayDimFetch[$1, $3]; }
     | T_CONSTANT_ENCAPSED_STRING '[' dim_offset ']'
-          { $$ = Expr\ArrayDimFetch[Scalar\String[Scalar\String::parse($1)], $3]; }
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = Expr\ArrayDimFetch[new Scalar\String_(Scalar\String_::parse($1), $attrs), $3]; }
     | constant '[' dim_offset ']'                           { $$ = Expr\ArrayDimFetch[$1, $3]; }
     | scalar_dereference '[' dim_offset ']'                 { $$ = Expr\ArrayDimFetch[$1, $3]; }
     /* alternative array syntax missing intentionally */
 ;
 
+anonymous_class:
+      T_CLASS ctor_arguments extends_from implements_list '{' class_statement_list '}'
+          { $$ = array(Stmt\Class_[null, ['type' => 0, 'extends' => $3, 'implements' => $4, 'stmts' => $6]], $2);
+            $this->checkClass($$[0], -1); }
+
 new_expr:
       T_NEW class_name_reference ctor_arguments             { $$ = Expr\New_[$2, $3]; }
+    | T_NEW anonymous_class
+          { list($class, $ctorArgs) = $2; $$ = Expr\New_[$class, $ctorArgs]; }
 ;
 
 lexical_vars:
@@ -653,7 +674,7 @@ lexical_var:
 
 function_call:
       name argument_list                                    { $$ = Expr\FuncCall[$1, $2]; }
-    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM T_STRING argument_list
+    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM identifier argument_list
           { $$ = Expr\StaticCall[$1, $3, $4]; }
     | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM '{' expr '}' argument_list
           { $$ = Expr\StaticCall[$1, $4, $6]; }
@@ -721,8 +742,9 @@ exit_expr:
 
 backticks_expr:
       /* empty */                                           { $$ = array(); }
-    | T_ENCAPSED_AND_WHITESPACE                             { $$ = array(Scalar\String::parseEscapeSequences($1, '`')); }
-    | encaps_list                                           { parseEncapsed($1, '`'); $$ = $1; }
+    | T_ENCAPSED_AND_WHITESPACE
+          { $$ = array(Scalar\EncapsedStringPart[Scalar\String_::parseEscapeSequences($1, '`', false)]); }
+    | encaps_list                                           { parseEncapsed($1, '`', false); $$ = $1; }
 ;
 
 ctor_arguments:
@@ -731,9 +753,11 @@ ctor_arguments:
 ;
 
 common_scalar:
-      T_LNUMBER                                             { $$ = Scalar\LNumber[Scalar\LNumber::parse($1)]; }
+      T_LNUMBER                                             { $$ = $this->parseLNumber($1, attributes(), true); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
-    | T_CONSTANT_ENCAPSED_STRING                            { $$ = Scalar\String[Scalar\String::parse($1)]; }
+    | T_CONSTANT_ENCAPSED_STRING
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = new Scalar\String_(Scalar\String_::parse($1, false), $attrs); }
     | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
     | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
     | T_DIR                                                 { $$ = Scalar\MagicConst\Dir[]; }
@@ -743,14 +767,16 @@ common_scalar:
     | T_FUNC_C                                              { $$ = Scalar\MagicConst\Function_[]; }
     | T_NS_C                                                { $$ = Scalar\MagicConst\Namespace_[]; }
     | T_START_HEREDOC T_ENCAPSED_AND_WHITESPACE T_END_HEREDOC
-          { $$ = Scalar\String[Scalar\String::parseDocString($1, $2)]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_(Scalar\String_::parseDocString($1, $2, false), $attrs); }
     | T_START_HEREDOC T_END_HEREDOC
-          { $$ = Scalar\String['']; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_('', $attrs); }
 ;
 
 static_scalar:
       common_scalar                                         { $$ = $1; }
-    | class_name T_PAAMAYIM_NEKUDOTAYIM class_const_name    { $$ = Expr\ClassConstFetch[$1, $3]; }
+    | class_name T_PAAMAYIM_NEKUDOTAYIM identifier          { $$ = Expr\ClassConstFetch[$1, $3]; }
     | name                                                  { $$ = Expr\ConstFetch[$1]; }
     | T_ARRAY '(' static_array_pair_list ')'                { $$ = Expr\Array_[$3]; }
     | '[' static_array_pair_list ']'                        { $$ = Expr\Array_[$2]; }
@@ -795,7 +821,7 @@ static_operation:
 
 constant:
       name                                                  { $$ = Expr\ConstFetch[$1]; }
-    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM class_const_name
+    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM identifier
           { $$ = Expr\ClassConstFetch[$1, $3]; }
 ;
 
@@ -803,14 +829,11 @@ scalar:
       common_scalar                                         { $$ = $1; }
     | constant                                              { $$ = $1; }
     | '"' encaps_list '"'
-          { parseEncapsed($2, '"'); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
+            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
     | T_START_HEREDOC encaps_list T_END_HEREDOC
-          { parseEncapsedDoc($2); $$ = Scalar\Encapsed[$2]; }
-;
-
-class_const_name:
-      T_STRING                                              { $$ = $1; }
-    | T_CLASS                                               { $$ = 'class'; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            parseEncapsedDoc($2, true); $$ = new Scalar\Encapsed($2, $attrs); }
 ;
 
 static_array_pair_list:
@@ -902,6 +925,7 @@ object_property:
       T_STRING                                              { $$ = $1; }
     | '{' expr '}'                                          { $$ = $2; }
     | variable_without_objects                              { $$ = $1; }
+    | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
 ;
 
 list_expr:
@@ -914,8 +938,8 @@ list_expr_elements:
 ;
 
 list_expr_element:
-      variable                                              { $$ = $1; }
-    | list_expr                                             { $$ = $1; }
+      variable                                              { $$ = Expr\ArrayItem[$1, null, false]; }
+    | list_expr                                             { $$ = Expr\ArrayItem[$1, null, false]; }
     | /* empty */                                           { $$ = null; }
 ;
 
@@ -938,15 +962,23 @@ array_pair:
 
 encaps_list:
       encaps_list encaps_var                                { push($1, $2); }
-    | encaps_list T_ENCAPSED_AND_WHITESPACE                 { push($1, $2); }
+    | encaps_list encaps_string_part                        { push($1, $2); }
     | encaps_var                                            { init($1); }
-    | T_ENCAPSED_AND_WHITESPACE encaps_var                  { init($1, $2); }
+    | encaps_string_part encaps_var                         { init($1, $2); }
+;
+
+encaps_string_part:
+      T_ENCAPSED_AND_WHITESPACE                             { $$ = Scalar\EncapsedStringPart[$1]; }
+;
+
+encaps_base_var:
+      T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
 ;
 
 encaps_var:
-      T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
-    | T_VARIABLE '[' encaps_var_offset ']'                  { $$ = Expr\ArrayDimFetch[Expr\Variable[parseVar($1)], $3]; }
-    | T_VARIABLE T_OBJECT_OPERATOR T_STRING                 { $$ = Expr\PropertyFetch[Expr\Variable[parseVar($1)], $3]; }
+      encaps_base_var                                       { $$ = $1; }
+    | encaps_base_var '[' encaps_var_offset ']'             { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | encaps_base_var T_OBJECT_OPERATOR T_STRING            { $$ = Expr\PropertyFetch[$1, $3]; }
     | T_DOLLAR_OPEN_CURLY_BRACES expr '}'                   { $$ = Expr\Variable[$2]; }
     | T_DOLLAR_OPEN_CURLY_BRACES T_STRING_VARNAME '}'       { $$ = Expr\Variable[$2]; }
     | T_DOLLAR_OPEN_CURLY_BRACES T_STRING_VARNAME '[' expr ']' '}'
@@ -955,8 +987,8 @@ encaps_var:
 ;
 
 encaps_var_offset:
-      T_STRING                                              { $$ = Scalar\String[$1]; }
-    | T_NUM_STRING                                          { $$ = Scalar\String[$1]; }
+      T_STRING                                              { $$ = Scalar\String_[$1]; }
+    | T_NUM_STRING                                          { $$ = $this->parseNumString($1, attributes()); }
     | T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
 ;
 

grammar/php7.y

@@ -0,0 +1,951 @@
+%pure_parser
+%expect 2
+
+%tokens
+
+%%
+
+start:
+    top_statement_list                                      { $$ = $this->handleNamespaces($1); }
+;
+
+top_statement_list_ex:
+      top_statement_list_ex top_statement                   { pushNormalizing($1, $2); }
+    | /* empty */                                           { init(); }
+;
+
+top_statement_list:
+      top_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
+reserved_non_modifiers:
+      T_INCLUDE | T_INCLUDE_ONCE | T_EVAL | T_REQUIRE | T_REQUIRE_ONCE | T_LOGICAL_OR | T_LOGICAL_XOR | T_LOGICAL_AND
+    | T_INSTANCEOF | T_NEW | T_CLONE | T_EXIT | T_IF | T_ELSEIF | T_ELSE | T_ENDIF | T_ECHO | T_DO | T_WHILE
+    | T_ENDWHILE | T_FOR | T_ENDFOR | T_FOREACH | T_ENDFOREACH | T_DECLARE | T_ENDDECLARE | T_AS | T_TRY | T_CATCH
+    | T_FINALLY | T_THROW | T_USE | T_INSTEADOF | T_GLOBAL | T_VAR | T_UNSET | T_ISSET | T_EMPTY | T_CONTINUE | T_GOTO
+    | T_FUNCTION | T_CONST | T_RETURN | T_PRINT | T_YIELD | T_LIST | T_SWITCH | T_ENDSWITCH | T_CASE | T_DEFAULT
+    | T_BREAK | T_ARRAY | T_CALLABLE | T_EXTENDS | T_IMPLEMENTS | T_NAMESPACE | T_TRAIT | T_INTERFACE | T_CLASS
+    | T_CLASS_C | T_TRAIT_C | T_FUNC_C | T_METHOD_C | T_LINE | T_FILE | T_DIR | T_NS_C | T_HALT_COMPILER
+;
+
+semi_reserved:
+      reserved_non_modifiers
+    | T_STATIC | T_ABSTRACT | T_FINAL | T_PRIVATE | T_PROTECTED | T_PUBLIC
+;
+
+identifier:
+      T_STRING                                              { $$ = $1; }
+    | semi_reserved                                         { $$ = $1; }
+;
+
+namespace_name_parts:
+      T_STRING                                              { init($1); }
+    | namespace_name_parts T_NS_SEPARATOR T_STRING          { push($1, $3); }
+;
+
+namespace_name:
+      namespace_name_parts                                  { $$ = Name[$1]; }
+;
+
+semi:
+      ';'                                                   { /* nothing */ }
+    | error                                                 { /* nothing */ }
+;
+
+no_comma:
+      /* empty */ { /* nothing */ }
+    | ',' { $this->emitError(new Error('A trailing comma is not allowed here', attributes())); }
+;
+
+optional_comma:
+      /* empty */
+    | ','
+
+top_statement:
+      statement                                             { $$ = $1; }
+    | function_declaration_statement                        { $$ = $1; }
+    | class_declaration_statement                           { $$ = $1; }
+    | T_HALT_COMPILER
+          { $$ = Stmt\HaltCompiler[$this->lexer->handleHaltCompiler()]; }
+    | T_NAMESPACE namespace_name semi
+          { $$ = Stmt\Namespace_[$2, null]; $this->checkNamespace($$); }
+    | T_NAMESPACE namespace_name '{' top_statement_list '}'
+          { $$ = Stmt\Namespace_[$2, $4]; $this->checkNamespace($$); }
+    | T_NAMESPACE '{' top_statement_list '}'
+          { $$ = Stmt\Namespace_[null, $3]; $this->checkNamespace($$); }
+    | T_USE use_declarations semi                           { $$ = Stmt\Use_[$2, Stmt\Use_::TYPE_NORMAL]; }
+    | T_USE use_type use_declarations semi                  { $$ = Stmt\Use_[$3, $2]; }
+    | group_use_declaration semi                            { $$ = $1; }
+    | T_CONST constant_declaration_list semi                { $$ = Stmt\Const_[$2]; }
+;
+
+use_type:
+      T_FUNCTION                                            { $$ = Stmt\Use_::TYPE_FUNCTION; }
+    | T_CONST                                               { $$ = Stmt\Use_::TYPE_CONSTANT; }
+;
+
+/* Using namespace_name_parts here to avoid s/r conflict on T_NS_SEPARATOR */
+group_use_declaration:
+      T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, $2]; }
+    | T_USE use_type T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+          { $$ = Stmt\GroupUse[new Name($4, stackAttributes(#4)), $7, $2]; }
+    | T_USE namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
+          { $$ = Stmt\GroupUse[new Name($2, stackAttributes(#2)), $5, Stmt\Use_::TYPE_UNKNOWN]; }
+    | T_USE T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
+          { $$ = Stmt\GroupUse[new Name($3, stackAttributes(#3)), $6, Stmt\Use_::TYPE_UNKNOWN]; }
+;
+
+unprefixed_use_declarations:
+      non_empty_unprefixed_use_declarations optional_comma  { $$ = $1; }
+;
+
+non_empty_unprefixed_use_declarations:
+      non_empty_unprefixed_use_declarations ',' unprefixed_use_declaration
+          { push($1, $3); }
+    | unprefixed_use_declaration                            { init($1); }
+;
+
+use_declarations:
+      non_empty_use_declarations no_comma                   { $$ = $1; }
+;
+
+non_empty_use_declarations:
+      non_empty_use_declarations ',' use_declaration        { push($1, $3); }
+    | use_declaration                                       { init($1); }
+;
+
+inline_use_declarations:
+      non_empty_inline_use_declarations optional_comma      { $$ = $1; }
+;
+
+non_empty_inline_use_declarations:
+      non_empty_inline_use_declarations ',' inline_use_declaration
+          { push($1, $3); }
+    | inline_use_declaration                                { init($1); }
+;
+
+unprefixed_use_declaration:
+      namespace_name
+          { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #1); }
+    | namespace_name T_AS T_STRING
+          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; $this->checkUseUse($$, #3); }
+;
+
+use_declaration:
+      unprefixed_use_declaration                            { $$ = $1; }
+    | T_NS_SEPARATOR unprefixed_use_declaration             { $$ = $2; }
+;
+
+inline_use_declaration:
+      unprefixed_use_declaration                            { $$ = $1; $$->type = Stmt\Use_::TYPE_NORMAL; }
+    | use_type unprefixed_use_declaration                   { $$ = $2; $$->type = $1; }
+;
+
+constant_declaration_list:
+      non_empty_constant_declaration_list no_comma          { $$ = $1; }
+;
+
+non_empty_constant_declaration_list:
+      non_empty_constant_declaration_list ',' constant_declaration
+          { push($1, $3); }
+    | constant_declaration                                  { init($1); }
+;
+
+constant_declaration:
+    T_STRING '=' expr                                       { $$ = Node\Const_[$1, $3]; }
+;
+
+class_const_list:
+      non_empty_class_const_list no_comma                   { $$ = $1; }
+;
+
+non_empty_class_const_list:
+      non_empty_class_const_list ',' class_const            { push($1, $3); }
+    | class_const                                           { init($1); }
+;
+
+class_const:
+    identifier '=' expr                                     { $$ = Node\Const_[$1, $3]; }
+;
+
+inner_statement_list_ex:
+      inner_statement_list_ex inner_statement               { pushNormalizing($1, $2); }
+    | /* empty */                                           { init(); }
+;
+
+inner_statement_list:
+      inner_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
+inner_statement:
+      statement                                             { $$ = $1; }
+    | function_declaration_statement                        { $$ = $1; }
+    | class_declaration_statement                           { $$ = $1; }
+    | T_HALT_COMPILER
+          { throw new Error('__HALT_COMPILER() can only be used from the outermost scope', attributes()); }
+;
+
+non_empty_statement:
+      '{' inner_statement_list '}'                          { $$ = $2; prependLeadingComments($$); }
+    | T_IF '(' expr ')' statement elseif_list else_single
+          { $$ = Stmt\If_[$3, ['stmts' => toArray($5), 'elseifs' => $6, 'else' => $7]]; }
+    | T_IF '(' expr ')' ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
+          { $$ = Stmt\If_[$3, ['stmts' => $6, 'elseifs' => $7, 'else' => $8]]; }
+    | T_WHILE '(' expr ')' while_statement                  { $$ = Stmt\While_[$3, $5]; }
+    | T_DO statement T_WHILE '(' expr ')' ';'               { $$ = Stmt\Do_   [$5, toArray($2)]; }
+    | T_FOR '(' for_expr ';'  for_expr ';' for_expr ')' for_statement
+          { $$ = Stmt\For_[['init' => $3, 'cond' => $5, 'loop' => $7, 'stmts' => $9]]; }
+    | T_SWITCH '(' expr ')' switch_case_list                { $$ = Stmt\Switch_[$3, $5]; }
+    | T_BREAK optional_expr semi                            { $$ = Stmt\Break_[$2]; }
+    | T_CONTINUE optional_expr semi                         { $$ = Stmt\Continue_[$2]; }
+    | T_RETURN optional_expr semi                           { $$ = Stmt\Return_[$2]; }
+    | T_GLOBAL global_var_list semi                         { $$ = Stmt\Global_[$2]; }
+    | T_STATIC static_var_list semi                         { $$ = Stmt\Static_[$2]; }
+    | T_ECHO expr_list semi                                 { $$ = Stmt\Echo_[$2]; }
+    | T_INLINE_HTML                                         { $$ = Stmt\InlineHTML[$1]; }
+    | expr semi                                             { $$ = $1; }
+    | T_UNSET '(' variables_list ')' semi                   { $$ = Stmt\Unset_[$3]; }
+    | T_FOREACH '(' expr T_AS foreach_variable ')' foreach_statement
+          { $$ = Stmt\Foreach_[$3, $5[0], ['keyVar' => null, 'byRef' => $5[1], 'stmts' => $7]]; }
+    | T_FOREACH '(' expr T_AS variable T_DOUBLE_ARROW foreach_variable ')' foreach_statement
+          { $$ = Stmt\Foreach_[$3, $7[0], ['keyVar' => $5, 'byRef' => $7[1], 'stmts' => $9]]; }
+    | T_DECLARE '(' declare_list ')' declare_statement      { $$ = Stmt\Declare_[$3, $5]; }
+    | T_TRY '{' inner_statement_list '}' catches optional_finally
+          { $$ = Stmt\TryCatch[$3, $5, $6]; $this->checkTryCatch($$); }
+    | T_THROW expr semi                                     { $$ = Stmt\Throw_[$2]; }
+    | T_GOTO T_STRING semi                                  { $$ = Stmt\Goto_[$2]; }
+    | T_STRING ':'                                          { $$ = Stmt\Label[$1]; }
+    | error                                                 { $$ = array(); /* means: no statement */ }
+;
+
+statement:
+      non_empty_statement                                   { $$ = $1; }
+    | ';'
+          { makeNop($$, $this->startAttributeStack[#1]);
+            if ($$ === null) $$ = array(); /* means: no statement */ }
+;
+
+catches:
+      /* empty */                                           { init(); }
+    | catches catch                                         { push($1, $2); }
+;
+
+name_union:
+      name                                                  { init($1); }
+    | name_union '|' name                                   { push($1, $3); }
+;
+
+catch:
+    T_CATCH '(' name_union T_VARIABLE ')' '{' inner_statement_list '}'
+        { $$ = Stmt\Catch_[$3, parseVar($4), $7]; }
+;
+
+optional_finally:
+      /* empty */                                           { $$ = null; }
+    | T_FINALLY '{' inner_statement_list '}'                { $$ = Stmt\Finally_[$3]; }
+;
+
+variables_list:
+      non_empty_variables_list no_comma                     { $$ = $1; }
+;
+
+non_empty_variables_list:
+      variable                                              { init($1); }
+    | non_empty_variables_list ',' variable                 { push($1, $3); }
+;
+
+optional_ref:
+      /* empty */                                           { $$ = false; }
+    | '&'                                                   { $$ = true; }
+;
+
+optional_ellipsis:
+      /* empty */                                           { $$ = false; }
+    | T_ELLIPSIS                                            { $$ = true; }
+;
+
+function_declaration_statement:
+    T_FUNCTION optional_ref T_STRING '(' parameter_list ')' optional_return_type '{' inner_statement_list '}'
+        { $$ = Stmt\Function_[$3, ['byRef' => $2, 'params' => $5, 'returnType' => $7, 'stmts' => $9]]; }
+;
+
+class_declaration_statement:
+      class_entry_type T_STRING extends_from implements_list '{' class_statement_list '}'
+          { $$ = Stmt\Class_[$2, ['type' => $1, 'extends' => $3, 'implements' => $4, 'stmts' => $6]];
+            $this->checkClass($$, #2); }
+    | T_INTERFACE T_STRING interface_extends_list '{' class_statement_list '}'
+          { $$ = Stmt\Interface_[$2, ['extends' => $3, 'stmts' => $5]];
+            $this->checkInterface($$, #2); }
+    | T_TRAIT T_STRING '{' class_statement_list '}'
+          { $$ = Stmt\Trait_[$2, ['stmts' => $4]]; }
+;
+
+class_entry_type:
+      T_CLASS                                               { $$ = 0; }
+    | T_ABSTRACT T_CLASS                                    { $$ = Stmt\Class_::MODIFIER_ABSTRACT; }
+    | T_FINAL T_CLASS                                       { $$ = Stmt\Class_::MODIFIER_FINAL; }
+;
+
+extends_from:
+      /* empty */                                           { $$ = null; }
+    | T_EXTENDS class_name                                  { $$ = $2; }
+;
+
+interface_extends_list:
+      /* empty */                                           { $$ = array(); }
+    | T_EXTENDS class_name_list                             { $$ = $2; }
+;
+
+implements_list:
+      /* empty */                                           { $$ = array(); }
+    | T_IMPLEMENTS class_name_list                          { $$ = $2; }
+;
+
+class_name_list:
+      non_empty_class_name_list no_comma                    { $$ = $1; }
+;
+
+non_empty_class_name_list:
+      class_name                                            { init($1); }
+    | non_empty_class_name_list ',' class_name              { push($1, $3); }
+;
+
+for_statement:
+      statement                                             { $$ = toArray($1); }
+    | ':' inner_statement_list T_ENDFOR ';'                 { $$ = $2; }
+;
+
+foreach_statement:
+      statement                                             { $$ = toArray($1); }
+    | ':' inner_statement_list T_ENDFOREACH ';'             { $$ = $2; }
+;
+
+declare_statement:
+      non_empty_statement                                   { $$ = toArray($1); }
+    | ';'                                                   { $$ = null; }
+    | ':' inner_statement_list T_ENDDECLARE ';'             { $$ = $2; }
+;
+
+declare_list:
+      non_empty_declare_list no_comma                       { $$ = $1; }
+;
+
+non_empty_declare_list:
+      declare_list_element                                  { init($1); }
+    | non_empty_declare_list ',' declare_list_element       { push($1, $3); }
+;
+
+declare_list_element:
+      T_STRING '=' expr                                     { $$ = Stmt\DeclareDeclare[$1, $3]; }
+;
+
+switch_case_list:
+      '{' case_list '}'                                     { $$ = $2; }
+    | '{' ';' case_list '}'                                 { $$ = $3; }
+    | ':' case_list T_ENDSWITCH ';'                         { $$ = $2; }
+    | ':' ';' case_list T_ENDSWITCH ';'                     { $$ = $3; }
+;
+
+case_list:
+      /* empty */                                           { init(); }
+    | case_list case                                        { push($1, $2); }
+;
+
+case:
+      T_CASE expr case_separator inner_statement_list       { $$ = Stmt\Case_[$2, $4]; }
+    | T_DEFAULT case_separator inner_statement_list         { $$ = Stmt\Case_[null, $3]; }
+;
+
+case_separator:
+      ':'
+    | ';'
+;
+
+while_statement:
+      statement                                             { $$ = toArray($1); }
+    | ':' inner_statement_list T_ENDWHILE ';'               { $$ = $2; }
+;
+
+elseif_list:
+      /* empty */                                           { init(); }
+    | elseif_list elseif                                    { push($1, $2); }
+;
+
+elseif:
+      T_ELSEIF '(' expr ')' statement                       { $$ = Stmt\ElseIf_[$3, toArray($5)]; }
+;
+
+new_elseif_list:
+      /* empty */                                           { init(); }
+    | new_elseif_list new_elseif                            { push($1, $2); }
+;
+
+new_elseif:
+     T_ELSEIF '(' expr ')' ':' inner_statement_list         { $$ = Stmt\ElseIf_[$3, $6]; }
+;
+
+else_single:
+      /* empty */                                           { $$ = null; }
+    | T_ELSE statement                                      { $$ = Stmt\Else_[toArray($2)]; }
+;
+
+new_else_single:
+      /* empty */                                           { $$ = null; }
+    | T_ELSE ':' inner_statement_list                       { $$ = Stmt\Else_[$3]; }
+;
+
+foreach_variable:
+      variable                                              { $$ = array($1, false); }
+    | '&' variable                                          { $$ = array($2, true); }
+    | list_expr                                             { $$ = array($1, false); }
+    | array_short_syntax                                    { $$ = array($1, false); }
+;
+
+parameter_list:
+      non_empty_parameter_list no_comma                     { $$ = $1; }
+    | /* empty */                                           { $$ = array(); }
+;
+
+non_empty_parameter_list:
+      parameter                                             { init($1); }
+    | non_empty_parameter_list ',' parameter                { push($1, $3); }
+;
+
+parameter:
+      optional_param_type optional_ref optional_ellipsis T_VARIABLE
+          { $$ = Node\Param[parseVar($4), null, $1, $2, $3]; $this->checkParam($$); }
+    | optional_param_type optional_ref optional_ellipsis T_VARIABLE '=' expr
+          { $$ = Node\Param[parseVar($4), $6, $1, $2, $3]; $this->checkParam($$); }
+;
+
+type_expr:
+      type                                                  { $$ = $1; }
+    | '?' type                                              { $$ = Node\NullableType[$2]; }
+;
+
+type:
+      name                                                  { $$ = $this->handleBuiltinTypes($1); }
+    | T_ARRAY                                               { $$ = 'array'; }
+    | T_CALLABLE                                            { $$ = 'callable'; }
+;
+
+optional_param_type:
+      /* empty */                                           { $$ = null; }
+    | type_expr                                             { $$ = $1; }
+;
+
+optional_return_type:
+      /* empty */                                           { $$ = null; }
+    | ':' type_expr                                         { $$ = $2; }
+;
+
+argument_list:
+      '(' ')'                                               { $$ = array(); }
+    | '(' non_empty_argument_list no_comma ')'              { $$ = $2; }
+;
+
+non_empty_argument_list:
+      argument                                              { init($1); }
+    | non_empty_argument_list ',' argument                  { push($1, $3); }
+;
+
+argument:
+      expr                                                  { $$ = Node\Arg[$1, false, false]; }
+    | '&' variable                                          { $$ = Node\Arg[$2, true, false]; }
+    | T_ELLIPSIS expr                                       { $$ = Node\Arg[$2, false, true]; }
+;
+
+global_var_list:
+      non_empty_global_var_list no_comma                    { $$ = $1; }
+;
+
+non_empty_global_var_list:
+      non_empty_global_var_list ',' global_var              { push($1, $3); }
+    | global_var                                            { init($1); }
+;
+
+global_var:
+      simple_variable                                       { $$ = Expr\Variable[$1]; }
+;
+
+static_var_list:
+      non_empty_static_var_list no_comma                    { $$ = $1; }
+;
+
+non_empty_static_var_list:
+      non_empty_static_var_list ',' static_var              { push($1, $3); }
+    | static_var                                            { init($1); }
+;
+
+static_var:
+      T_VARIABLE                                            { $$ = Stmt\StaticVar[parseVar($1), null]; }
+    | T_VARIABLE '=' expr                                   { $$ = Stmt\StaticVar[parseVar($1), $3]; }
+;
+
+class_statement_list:
+      class_statement_list class_statement                  { push($1, $2); }
+    | /* empty */                                           { init(); }
+;
+
+class_statement:
+      variable_modifiers property_declaration_list ';'
+          { $$ = Stmt\Property[$1, $2]; $this->checkProperty($$, #1); }
+    | method_modifiers T_CONST class_const_list ';'
+          { $$ = Stmt\ClassConst[$3, $1]; $this->checkClassConst($$, #1); }
+    | method_modifiers T_FUNCTION optional_ref identifier '(' parameter_list ')' optional_return_type method_body
+          { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]];
+            $this->checkClassMethod($$, #1); }
+    | T_USE class_name_list trait_adaptations               { $$ = Stmt\TraitUse[$2, $3]; }
+;
+
+trait_adaptations:
+      ';'                                                   { $$ = array(); }
+    | '{' trait_adaptation_list '}'                         { $$ = $2; }
+;
+
+trait_adaptation_list:
+      /* empty */                                           { init(); }
+    | trait_adaptation_list trait_adaptation                { push($1, $2); }
+;
+
+trait_adaptation:
+      trait_method_reference_fully_qualified T_INSTEADOF class_name_list ';'
+          { $$ = Stmt\TraitUseAdaptation\Precedence[$1[0], $1[1], $3]; }
+    | trait_method_reference T_AS member_modifier identifier ';'
+          { $$ = Stmt\TraitUseAdaptation\Alias[$1[0], $1[1], $3, $4]; }
+    | trait_method_reference T_AS member_modifier ';'
+          { $$ = Stmt\TraitUseAdaptation\Alias[$1[0], $1[1], $3, null]; }
+    | trait_method_reference T_AS T_STRING ';'
+          { $$ = Stmt\TraitUseAdaptation\Alias[$1[0], $1[1], null, $3]; }
+    | trait_method_reference T_AS reserved_non_modifiers ';'
+          { $$ = Stmt\TraitUseAdaptation\Alias[$1[0], $1[1], null, $3]; }
+;
+
+trait_method_reference_fully_qualified:
+      name T_PAAMAYIM_NEKUDOTAYIM identifier                { $$ = array($1, $3); }
+;
+trait_method_reference:
+      trait_method_reference_fully_qualified                { $$ = $1; }
+    | identifier                                            { $$ = array(null, $1); }
+;
+
+method_body:
+      ';' /* abstract method */                             { $$ = null; }
+    | '{' inner_statement_list '}'                          { $$ = $2; }
+;
+
+variable_modifiers:
+      non_empty_member_modifiers                            { $$ = $1; }
+    | T_VAR                                                 { $$ = 0; }
+;
+
+method_modifiers:
+      /* empty */                                           { $$ = 0; }
+    | non_empty_member_modifiers                            { $$ = $1; }
+;
+
+non_empty_member_modifiers:
+      member_modifier                                       { $$ = $1; }
+    | non_empty_member_modifiers member_modifier            { $this->checkModifier($1, $2, #2); $$ = $1 | $2; }
+;
+
+member_modifier:
+      T_PUBLIC                                              { $$ = Stmt\Class_::MODIFIER_PUBLIC; }
+    | T_PROTECTED                                           { $$ = Stmt\Class_::MODIFIER_PROTECTED; }
+    | T_PRIVATE                                             { $$ = Stmt\Class_::MODIFIER_PRIVATE; }
+    | T_STATIC                                              { $$ = Stmt\Class_::MODIFIER_STATIC; }
+    | T_ABSTRACT                                            { $$ = Stmt\Class_::MODIFIER_ABSTRACT; }
+    | T_FINAL                                               { $$ = Stmt\Class_::MODIFIER_FINAL; }
+;
+
+property_declaration_list:
+      non_empty_property_declaration_list no_comma          { $$ = $1; }
+;
+
+non_empty_property_declaration_list:
+      property_declaration                                  { init($1); }
+    | non_empty_property_declaration_list ',' property_declaration
+          { push($1, $3); }
+;
+
+property_declaration:
+      T_VARIABLE                                            { $$ = Stmt\PropertyProperty[parseVar($1), null]; }
+    | T_VARIABLE '=' expr                                   { $$ = Stmt\PropertyProperty[parseVar($1), $3]; }
+;
+
+expr_list:
+      non_empty_expr_list no_comma                          { $$ = $1; }
+;
+
+non_empty_expr_list:
+      non_empty_expr_list ',' expr                          { push($1, $3); }
+    | expr                                                  { init($1); }
+;
+
+for_expr:
+      /* empty */                                           { $$ = array(); }
+    | expr_list                                             { $$ = $1; }
+;
+
+expr:
+      variable                                              { $$ = $1; }
+    | list_expr '=' expr                                    { $$ = Expr\Assign[$1, $3]; }
+    | array_short_syntax '=' expr                           { $$ = Expr\Assign[$1, $3]; }
+    | variable '=' expr                                     { $$ = Expr\Assign[$1, $3]; }
+    | variable '=' '&' variable                             { $$ = Expr\AssignRef[$1, $4]; }
+    | new_expr                                              { $$ = $1; }
+    | T_CLONE expr                                          { $$ = Expr\Clone_[$2]; }
+    | variable T_PLUS_EQUAL expr                            { $$ = Expr\AssignOp\Plus      [$1, $3]; }
+    | variable T_MINUS_EQUAL expr                           { $$ = Expr\AssignOp\Minus     [$1, $3]; }
+    | variable T_MUL_EQUAL expr                             { $$ = Expr\AssignOp\Mul       [$1, $3]; }
+    | variable T_DIV_EQUAL expr                             { $$ = Expr\AssignOp\Div       [$1, $3]; }
+    | variable T_CONCAT_EQUAL expr                          { $$ = Expr\AssignOp\Concat    [$1, $3]; }
+    | variable T_MOD_EQUAL expr                             { $$ = Expr\AssignOp\Mod       [$1, $3]; }
+    | variable T_AND_EQUAL expr                             { $$ = Expr\AssignOp\BitwiseAnd[$1, $3]; }
+    | variable T_OR_EQUAL expr                              { $$ = Expr\AssignOp\BitwiseOr [$1, $3]; }
+    | variable T_XOR_EQUAL expr                             { $$ = Expr\AssignOp\BitwiseXor[$1, $3]; }
+    | variable T_SL_EQUAL expr                              { $$ = Expr\AssignOp\ShiftLeft [$1, $3]; }
+    | variable T_SR_EQUAL expr                              { $$ = Expr\AssignOp\ShiftRight[$1, $3]; }
+    | variable T_POW_EQUAL expr                             { $$ = Expr\AssignOp\Pow       [$1, $3]; }
+    | variable T_INC                                        { $$ = Expr\PostInc[$1]; }
+    | T_INC variable                                        { $$ = Expr\PreInc [$2]; }
+    | variable T_DEC                                        { $$ = Expr\PostDec[$1]; }
+    | T_DEC variable                                        { $$ = Expr\PreDec [$2]; }
+    | expr T_BOOLEAN_OR expr                                { $$ = Expr\BinaryOp\BooleanOr [$1, $3]; }
+    | expr T_BOOLEAN_AND expr                               { $$ = Expr\BinaryOp\BooleanAnd[$1, $3]; }
+    | expr T_LOGICAL_OR expr                                { $$ = Expr\BinaryOp\LogicalOr [$1, $3]; }
+    | expr T_LOGICAL_AND expr                               { $$ = Expr\BinaryOp\LogicalAnd[$1, $3]; }
+    | expr T_LOGICAL_XOR expr                               { $$ = Expr\BinaryOp\LogicalXor[$1, $3]; }
+    | expr '|' expr                                         { $$ = Expr\BinaryOp\BitwiseOr [$1, $3]; }
+    | expr '&' expr                                         { $$ = Expr\BinaryOp\BitwiseAnd[$1, $3]; }
+    | expr '^' expr                                         { $$ = Expr\BinaryOp\BitwiseXor[$1, $3]; }
+    | expr '.' expr                                         { $$ = Expr\BinaryOp\Concat    [$1, $3]; }
+    | expr '+' expr                                         { $$ = Expr\BinaryOp\Plus      [$1, $3]; }
+    | expr '-' expr                                         { $$ = Expr\BinaryOp\Minus     [$1, $3]; }
+    | expr '*' expr                                         { $$ = Expr\BinaryOp\Mul       [$1, $3]; }
+    | expr '/' expr                                         { $$ = Expr\BinaryOp\Div       [$1, $3]; }
+    | expr '%' expr                                         { $$ = Expr\BinaryOp\Mod       [$1, $3]; }
+    | expr T_SL expr                                        { $$ = Expr\BinaryOp\ShiftLeft [$1, $3]; }
+    | expr T_SR expr                                        { $$ = Expr\BinaryOp\ShiftRight[$1, $3]; }
+    | expr T_POW expr                                       { $$ = Expr\BinaryOp\Pow       [$1, $3]; }
+    | '+' expr %prec T_INC                                  { $$ = Expr\UnaryPlus [$2]; }
+    | '-' expr %prec T_INC                                  { $$ = Expr\UnaryMinus[$2]; }
+    | '!' expr                                              { $$ = Expr\BooleanNot[$2]; }
+    | '~' expr                                              { $$ = Expr\BitwiseNot[$2]; }
+    | expr T_IS_IDENTICAL expr                              { $$ = Expr\BinaryOp\Identical     [$1, $3]; }
+    | expr T_IS_NOT_IDENTICAL expr                          { $$ = Expr\BinaryOp\NotIdentical  [$1, $3]; }
+    | expr T_IS_EQUAL expr                                  { $$ = Expr\BinaryOp\Equal         [$1, $3]; }
+    | expr T_IS_NOT_EQUAL expr                              { $$ = Expr\BinaryOp\NotEqual      [$1, $3]; }
+    | expr T_SPACESHIP expr                                 { $$ = Expr\BinaryOp\Spaceship     [$1, $3]; }
+    | expr '<' expr                                         { $$ = Expr\BinaryOp\Smaller       [$1, $3]; }
+    | expr T_IS_SMALLER_OR_EQUAL expr                       { $$ = Expr\BinaryOp\SmallerOrEqual[$1, $3]; }
+    | expr '>' expr                                         { $$ = Expr\BinaryOp\Greater       [$1, $3]; }
+    | expr T_IS_GREATER_OR_EQUAL expr                       { $$ = Expr\BinaryOp\GreaterOrEqual[$1, $3]; }
+    | expr T_INSTANCEOF class_name_reference                { $$ = Expr\Instanceof_[$1, $3]; }
+    | '(' expr ')'                                          { $$ = $2; }
+    | expr '?' expr ':' expr                                { $$ = Expr\Ternary[$1, $3,   $5]; }
+    | expr '?' ':' expr                                     { $$ = Expr\Ternary[$1, null, $4]; }
+    | expr T_COALESCE expr                                  { $$ = Expr\BinaryOp\Coalesce[$1, $3]; }
+    | T_ISSET '(' variables_list ')'                        { $$ = Expr\Isset_[$3]; }
+    | T_EMPTY '(' expr ')'                                  { $$ = Expr\Empty_[$3]; }
+    | T_INCLUDE expr                                        { $$ = Expr\Include_[$2, Expr\Include_::TYPE_INCLUDE]; }
+    | T_INCLUDE_ONCE expr                                   { $$ = Expr\Include_[$2, Expr\Include_::TYPE_INCLUDE_ONCE]; }
+    | T_EVAL '(' expr ')'                                   { $$ = Expr\Eval_[$3]; }
+    | T_REQUIRE expr                                        { $$ = Expr\Include_[$2, Expr\Include_::TYPE_REQUIRE]; }
+    | T_REQUIRE_ONCE expr                                   { $$ = Expr\Include_[$2, Expr\Include_::TYPE_REQUIRE_ONCE]; }
+    | T_INT_CAST expr                                       { $$ = Expr\Cast\Int_    [$2]; }
+    | T_DOUBLE_CAST expr                                    { $$ = Expr\Cast\Double  [$2]; }
+    | T_STRING_CAST expr                                    { $$ = Expr\Cast\String_ [$2]; }
+    | T_ARRAY_CAST expr                                     { $$ = Expr\Cast\Array_  [$2]; }
+    | T_OBJECT_CAST expr                                    { $$ = Expr\Cast\Object_ [$2]; }
+    | T_BOOL_CAST expr                                      { $$ = Expr\Cast\Bool_   [$2]; }
+    | T_UNSET_CAST expr                                     { $$ = Expr\Cast\Unset_  [$2]; }
+    | T_EXIT exit_expr
+          { $attrs = attributes();
+            $attrs['kind'] = strtolower($1) === 'exit' ? Expr\Exit_::KIND_EXIT : Expr\Exit_::KIND_DIE;
+            $$ = new Expr\Exit_($2, $attrs); }
+    | '@' expr                                              { $$ = Expr\ErrorSuppress[$2]; }
+    | scalar                                                { $$ = $1; }
+    | '`' backticks_expr '`'                                { $$ = Expr\ShellExec[$2]; }
+    | T_PRINT expr                                          { $$ = Expr\Print_[$2]; }
+    | T_YIELD                                               { $$ = Expr\Yield_[null, null]; }
+    | T_YIELD expr                                          { $$ = Expr\Yield_[$2, null]; }
+    | T_YIELD expr T_DOUBLE_ARROW expr                      { $$ = Expr\Yield_[$4, $2]; }
+    | T_YIELD_FROM expr                                     { $$ = Expr\YieldFrom[$2]; }
+    | T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars optional_return_type
+      '{' inner_statement_list '}'
+          { $$ = Expr\Closure[['static' => false, 'byRef' => $2, 'params' => $4, 'uses' => $6, 'returnType' => $7, 'stmts' => $9]]; }
+    | T_STATIC T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars optional_return_type
+      '{' inner_statement_list '}'
+          { $$ = Expr\Closure[['static' => true, 'byRef' => $3, 'params' => $5, 'uses' => $7, 'returnType' => $8, 'stmts' => $10]]; }
+;
+
+anonymous_class:
+      T_CLASS ctor_arguments extends_from implements_list '{' class_statement_list '}'
+          { $$ = array(Stmt\Class_[null, ['type' => 0, 'extends' => $3, 'implements' => $4, 'stmts' => $6]], $2);
+            $this->checkClass($$[0], -1); }
+
+new_expr:
+      T_NEW class_name_reference ctor_arguments             { $$ = Expr\New_[$2, $3]; }
+    | T_NEW anonymous_class
+          { list($class, $ctorArgs) = $2; $$ = Expr\New_[$class, $ctorArgs]; }
+;
+
+lexical_vars:
+      /* empty */                                           { $$ = array(); }
+    | T_USE '(' lexical_var_list ')'                        { $$ = $3; }
+;
+
+lexical_var_list:
+      non_empty_lexical_var_list no_comma                   { $$ = $1; }
+;
+
+non_empty_lexical_var_list:
+      lexical_var                                           { init($1); }
+    | non_empty_lexical_var_list ',' lexical_var            { push($1, $3); }
+;
+
+lexical_var:
+      optional_ref T_VARIABLE                               { $$ = Expr\ClosureUse[parseVar($2), $1]; }
+;
+
+function_call:
+      name argument_list                                    { $$ = Expr\FuncCall[$1, $2]; }
+    | callable_expr argument_list                           { $$ = Expr\FuncCall[$1, $2]; }
+    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM member_name argument_list
+          { $$ = Expr\StaticCall[$1, $3, $4]; }
+;
+
+class_name:
+      T_STATIC                                              { $$ = Name[$1]; }
+    | name                                                  { $$ = $1; }
+;
+
+name:
+      namespace_name_parts                                  { $$ = Name[$1]; }
+    | T_NS_SEPARATOR namespace_name_parts                   { $$ = Name\FullyQualified[$2]; }
+    | T_NAMESPACE T_NS_SEPARATOR namespace_name_parts       { $$ = Name\Relative[$3]; }
+;
+
+class_name_reference:
+      class_name                                            { $$ = $1; }
+    | new_variable                                          { $$ = $1; }
+    | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
+;
+
+class_name_or_var:
+      class_name                                            { $$ = $1; }
+    | dereferencable                                        { $$ = $1; }
+;
+
+exit_expr:
+      /* empty */                                           { $$ = null; }
+    | '(' optional_expr ')'                                 { $$ = $2; }
+;
+
+backticks_expr:
+      /* empty */                                           { $$ = array(); }
+    | T_ENCAPSED_AND_WHITESPACE
+          { $$ = array(Scalar\EncapsedStringPart[Scalar\String_::parseEscapeSequences($1, '`')]); }
+    | encaps_list                                           { parseEncapsed($1, '`', true); $$ = $1; }
+;
+
+ctor_arguments:
+      /* empty */                                           { $$ = array(); }
+    | argument_list                                         { $$ = $1; }
+;
+
+constant:
+      name                                                  { $$ = Expr\ConstFetch[$1]; }
+    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM identifier
+          { $$ = Expr\ClassConstFetch[$1, $3]; }
+    /* We interpret and isolated FOO:: as an unfinished class constant fetch. It could also be
+       an unfinished static property fetch or unfinished scoped call. */
+    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM error
+          { $$ = Expr\ClassConstFetch[$1, new Expr\Error(stackAttributes(#3))]; $this->errorState = 2; }
+;
+
+array_short_syntax:
+      '[' array_pair_list ']'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_SHORT;
+            $$ = new Expr\Array_($2, $attrs); }
+;
+
+dereferencable_scalar:
+      T_ARRAY '(' array_pair_list ')'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_LONG;
+            $$ = new Expr\Array_($3, $attrs); }
+    | array_short_syntax                                    { $$ = $1; }
+    | T_CONSTANT_ENCAPSED_STRING
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = new Scalar\String_(Scalar\String_::parse($1), $attrs); }
+;
+
+scalar:
+      T_LNUMBER                                             { $$ = $this->parseLNumber($1, attributes()); }
+    | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
+    | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
+    | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
+    | T_DIR                                                 { $$ = Scalar\MagicConst\Dir[]; }
+    | T_CLASS_C                                             { $$ = Scalar\MagicConst\Class_[]; }
+    | T_TRAIT_C                                             { $$ = Scalar\MagicConst\Trait_[]; }
+    | T_METHOD_C                                            { $$ = Scalar\MagicConst\Method[]; }
+    | T_FUNC_C                                              { $$ = Scalar\MagicConst\Function_[]; }
+    | T_NS_C                                                { $$ = Scalar\MagicConst\Namespace_[]; }
+    | dereferencable_scalar                                 { $$ = $1; }
+    | constant                                              { $$ = $1; }
+    | T_START_HEREDOC T_ENCAPSED_AND_WHITESPACE T_END_HEREDOC
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_(Scalar\String_::parseDocString($1, $2), $attrs); }
+    | T_START_HEREDOC T_END_HEREDOC
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_('', $attrs); }
+    | '"' encaps_list '"'
+          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
+            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
+    | T_START_HEREDOC encaps_list T_END_HEREDOC
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            parseEncapsedDoc($2, true); $$ = new Scalar\Encapsed($2, $attrs); }
+;
+
+optional_expr:
+      /* empty */                                           { $$ = null; }
+    | expr                                                  { $$ = $1; }
+;
+
+dereferencable:
+      variable                                              { $$ = $1; }
+    | '(' expr ')'                                          { $$ = $2; }
+    | dereferencable_scalar                                 { $$ = $1; }
+;
+
+callable_expr:
+      callable_variable                                     { $$ = $1; }
+    | '(' expr ')'                                          { $$ = $2; }
+    | dereferencable_scalar                                 { $$ = $1; }
+;
+
+callable_variable:
+      simple_variable                                       { $$ = Expr\Variable[$1]; }
+    | dereferencable '[' optional_expr ']'                  { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | constant '[' optional_expr ']'                        { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | dereferencable '{' expr '}'                           { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | function_call                                         { $$ = $1; }
+    | dereferencable T_OBJECT_OPERATOR property_name argument_list
+          { $$ = Expr\MethodCall[$1, $3, $4]; }
+;
+
+variable:
+      callable_variable                                     { $$ = $1; }
+    | static_member                                         { $$ = $1; }
+    | dereferencable T_OBJECT_OPERATOR property_name        { $$ = Expr\PropertyFetch[$1, $3]; }
+;
+
+simple_variable:
+      T_VARIABLE                                            { $$ = parseVar($1); }
+    | '$' '{' expr '}'                                      { $$ = $3; }
+    | '$' simple_variable                                   { $$ = Expr\Variable[$2]; }
+    | '$' error                                             { $$ = Expr\Error[]; $this->errorState = 2; }
+;
+
+static_member:
+      class_name_or_var T_PAAMAYIM_NEKUDOTAYIM simple_variable
+          { $$ = Expr\StaticPropertyFetch[$1, $3]; }
+;
+
+new_variable:
+      simple_variable                                       { $$ = Expr\Variable[$1]; }
+    | new_variable '[' optional_expr ']'                    { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | new_variable '{' expr '}'                             { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | new_variable T_OBJECT_OPERATOR property_name          { $$ = Expr\PropertyFetch[$1, $3]; }
+    | class_name T_PAAMAYIM_NEKUDOTAYIM simple_variable     { $$ = Expr\StaticPropertyFetch[$1, $3]; }
+    | new_variable T_PAAMAYIM_NEKUDOTAYIM simple_variable   { $$ = Expr\StaticPropertyFetch[$1, $3]; }
+;
+
+member_name:
+      identifier                                            { $$ = $1; }
+    | '{' expr '}'	                                        { $$ = $2; }
+    | simple_variable	                                    { $$ = Expr\Variable[$1]; }
+;
+
+property_name:
+      T_STRING                                              { $$ = $1; }
+    | '{' expr '}'	                                        { $$ = $2; }
+    | simple_variable	                                    { $$ = Expr\Variable[$1]; }
+    | error                                                 { $$ = Expr\Error[]; $this->errorState = 2; }
+;
+
+list_expr:
+      T_LIST '(' list_expr_elements ')'                     { $$ = Expr\List_[$3]; }
+;
+
+list_expr_elements:
+      list_expr_elements ',' list_expr_element              { push($1, $3); }
+    | list_expr_element                                     { init($1); }
+;
+
+list_expr_element:
+      variable                                              { $$ = Expr\ArrayItem[$1, null, false]; }
+    | list_expr                                             { $$ = Expr\ArrayItem[$1, null, false]; }
+    | expr T_DOUBLE_ARROW variable                          { $$ = Expr\ArrayItem[$3, $1, false]; }
+    | expr T_DOUBLE_ARROW list_expr                         { $$ = Expr\ArrayItem[$3, $1, false]; }
+    | /* empty */                                           { $$ = null; }
+;
+
+array_pair_list:
+      inner_array_pair_list
+          { $$ = $1; $end = count($$)-1; if ($$[$end] === null) unset($$[$end]); }
+;
+
+inner_array_pair_list:
+      inner_array_pair_list ',' array_pair                  { push($1, $3); }
+    | array_pair                                            { init($1); }
+;
+
+array_pair:
+      expr T_DOUBLE_ARROW expr                              { $$ = Expr\ArrayItem[$3, $1,   false]; }
+    | expr                                                  { $$ = Expr\ArrayItem[$1, null, false]; }
+    | expr T_DOUBLE_ARROW '&' variable                      { $$ = Expr\ArrayItem[$4, $1,   true]; }
+    | '&' variable                                          { $$ = Expr\ArrayItem[$2, null, true]; }
+    | /* empty */                                           { $$ = null; }
+;
+
+encaps_list:
+      encaps_list encaps_var                                { push($1, $2); }
+    | encaps_list encaps_string_part                        { push($1, $2); }
+    | encaps_var                                            { init($1); }
+    | encaps_string_part encaps_var                         { init($1, $2); }
+;
+
+encaps_string_part:
+      T_ENCAPSED_AND_WHITESPACE                             { $$ = Scalar\EncapsedStringPart[$1]; }
+;
+
+encaps_base_var:
+      T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
+;
+
+encaps_var:
+      encaps_base_var                                       { $$ = $1; }
+    | encaps_base_var '[' encaps_var_offset ']'             { $$ = Expr\ArrayDimFetch[$1, $3]; }
+    | encaps_base_var T_OBJECT_OPERATOR T_STRING            { $$ = Expr\PropertyFetch[$1, $3]; }
+    | T_DOLLAR_OPEN_CURLY_BRACES expr '}'                   { $$ = Expr\Variable[$2]; }
+    | T_DOLLAR_OPEN_CURLY_BRACES T_STRING_VARNAME '}'       { $$ = Expr\Variable[$2]; }
+    | T_DOLLAR_OPEN_CURLY_BRACES T_STRING_VARNAME '[' expr ']' '}'
+          { $$ = Expr\ArrayDimFetch[Expr\Variable[$2], $4]; }
+    | T_CURLY_OPEN variable '}'                             { $$ = $2; }
+;
+
+encaps_var_offset:
+      T_STRING                                              { $$ = Scalar\String_[$1]; }
+    | T_NUM_STRING                                          { $$ = $this->parseNumString($1, attributes()); }
+    | '-' T_NUM_STRING                                      { $$ = $this->parseNumString('-' . $2, attributes()); }
+    | T_VARIABLE                                            { $$ = Expr\Variable[parseVar($1)]; }
+;
+
+%%

grammar/rebuildParser.php

@@ -1,218 +0,0 @@
-<?php
-
-$grammarFile           = __DIR__ . '/zend_language_parser.phpy';
-$skeletonFile          = __DIR__ . '/kmyacc.php.parser';
-$tmpGrammarFile        = __DIR__ . '/tmp_parser.phpy';
-$tmpResultFile         = __DIR__ . '/tmp_parser.php';
-$parserResultFile      = __DIR__ . '/../lib/PhpParser/Parser.php';
-
-// check for kmyacc.exe binary in this directory, otherwise fall back to global name
-$kmyacc = __DIR__ . '/kmyacc.exe';
-if (!file_exists($kmyacc)) {
-    $kmyacc = 'kmyacc';
-}
-
-$options = array_flip($argv);
-$optionDebug = isset($options['--debug']);
-$optionKeepTmpGrammar = isset($options['--keep-tmp-grammar']);
-
-///////////////////////////////
-/// Utility regex constants ///
-///////////////////////////////
-
-const LIB = '(?(DEFINE)
-    (?<singleQuotedString>\'[^\\\\\']*+(?:\\\\.[^\\\\\']*+)*+\')
-    (?<doubleQuotedString>"[^\\\\"]*+(?:\\\\.[^\\\\"]*+)*+")
-    (?<string>(?&singleQuotedString)|(?&doubleQuotedString))
-    (?<comment>/\*[^*]*+(?:\*(?!/)[^*]*+)*+\*/)
-    (?<code>\{[^\'"/{}]*+(?:(?:(?&string)|(?&comment)|(?&code)|/)[^\'"/{}]*+)*+})
-)';
-
-const PARAMS = '\[(?<params>[^[\]]*+(?:\[(?&params)\][^[\]]*+)*+)\]';
-const ARGS   = '\((?<args>[^()]*+(?:\((?&args)\)[^()]*+)*+)\)';
-
-///////////////////
-/// Main script ///
-///////////////////
-
-echo 'Building temporary preproprocessed grammar file.', "\n";
-
-$grammarCode = file_get_contents($grammarFile);
-
-$grammarCode = resolveConstants($grammarCode);
-$grammarCode = resolveNodes($grammarCode);
-$grammarCode = resolveMacros($grammarCode);
-$grammarCode = resolveArrays($grammarCode);
-
-file_put_contents($tmpGrammarFile, $grammarCode);
-
-$additionalArgs = $optionDebug ? '-t -v' : '';
-
-echo "Building parser.\n";
-$output = trim(shell_exec("$kmyacc $additionalArgs -l -m $skeletonFile $tmpGrammarFile 2>&1"));
-echo "Output: \"$output\"\n";
-
-moveFileWithDirCheck($tmpResultFile, $parserResultFile);
-
-if (!$optionKeepTmpGrammar) {
-    unlink($tmpGrammarFile);
-}
-
-///////////////////////////////
-/// Preprocessing functions ///
-///////////////////////////////
-
-function resolveConstants($code) {
-    return preg_replace('~[A-Z][a-zA-Z_\\\\]++::~', 'Node\\\\$0', $code);
-}
-
-function resolveNodes($code) {
-    return preg_replace_callback(
-        '~(?<name>[A-Z][a-zA-Z_\\\\]++)\s*' . PARAMS . '~',
-        function($matches) {
-            // recurse
-            $matches['params'] = resolveNodes($matches['params']);
-
-            $params = magicSplit(
-                '(?:' . PARAMS . '|' . ARGS . ')(*SKIP)(*FAIL)|,',
-                $matches['params']
-            );
-
-            $paramCode = '';
-            foreach ($params as $param) {
-                $paramCode .= $param . ', ';
-            }
-
-            return 'new Node\\' . $matches['name'] . '(' . $paramCode . '$attributes)';
-        },
-        $code
-    );
-}
-
-function resolveMacros($code) {
-    return preg_replace_callback(
-        '~\b(?<!::|->)(?!array\()(?<name>[a-z][A-Za-z]++)' . ARGS . '~',
-        function($matches) {
-            // recurse
-            $matches['args'] = resolveMacros($matches['args']);
-
-            $name = $matches['name'];
-            $args = magicSplit(
-                '(?:' . PARAMS . '|' . ARGS . ')(*SKIP)(*FAIL)|,',
-                $matches['args']
-            );
-
-            if ('error' == $name) {
-                assertArgs(1, $args, $name);
-
-                return 'throw new Error(' . $args[0] . ')';
-            }
-
-            if ('init' == $name) {
-                return '$$ = array(' . implode(', ', $args) . ')';
-            }
-
-            if ('push' == $name) {
-                assertArgs(2, $args, $name);
-
-                return $args[0] . '[] = ' . $args[1] . '; $$ = ' . $args[0];
-            }
-
-            if ('pushNormalizing' == $name) {
-                assertArgs(2, $args, $name);
-
-                return 'if (is_array(' . $args[1] . ')) { $$ = array_merge(' . $args[0] . ', ' . $args[1] . '); } else { ' . $args[0] . '[] = ' . $args[1] . '; $$ = ' . $args[0] . '; }';
-            }
-
-            if ('toArray' == $name) {
-                assertArgs(1, $args, $name);
-
-                return 'is_array(' . $args[0] . ') ? ' . $args[0] . ' : array(' . $args[0] . ')';
-            }
-
-            if ('parseVar' == $name) {
-                assertArgs(1, $args, $name);
-
-                return 'substr(' . $args[0] . ', 1)';
-            }
-
-            if ('parseEncapsed' == $name) {
-                assertArgs(2, $args, $name);
-
-                return 'foreach (' . $args[0] . ' as &$s) { if (is_string($s)) { $s = Node\Scalar\String::parseEscapeSequences($s, ' . $args[1] . '); } }';
-            }
-
-            if ('parseEncapsedDoc' == $name) {
-                assertArgs(1, $args, $name);
-
-                return 'foreach (' . $args[0] . ' as &$s) { if (is_string($s)) { $s = Node\Scalar\String::parseEscapeSequences($s, null); } } $s = preg_replace(\'~(\r\n|\n|\r)$~\', \'\', $s); if (\'\' === $s) array_pop(' . $args[0] . ');';
-            }
-
-            throw new Exception(sprintf('Unknown macro "%s"', $name));
-        },
-        $code
-    );
-}
-
-function assertArgs($num, $args, $name) {
-    if ($num != count($args)) {
-        die('Wrong argument count for ' . $name . '().');
-    }
-}
-
-function resolveArrays($code) {
-    return preg_replace_callback(
-        '~' . PARAMS . '~',
-        function ($matches) {
-            $elements = magicSplit(
-                '(?:' . PARAMS . '|' . ARGS . ')(*SKIP)(*FAIL)|,',
-                $matches['params']
-            );
-
-            // don't convert [] to array, it might have different meaning
-            if (empty($elements)) {
-                return $matches[0];
-            }
-
-            $elementCodes = array();
-            foreach ($elements as $element) {
-                // convert only arrays where all elements have keys
-                if (false === strpos($element, ':')) {
-                    return $matches[0];
-                }
-
-                list($key, $value) = explode(':', $element, 2);
-                $elementCodes[] = "'" . $key . "' =>" . $value;
-            }
-
-            return 'array(' . implode(', ', $elementCodes) . ')';
-        },
-        $code
-    );
-}
-
-function moveFileWithDirCheck($fromPath, $toPath) {
-    $dir = dirname($toPath);
-    if (!is_dir($dir)) {
-        mkdir($dir, 0777, true);
-    }
-    rename($fromPath, $toPath);
-}
-
-//////////////////////////////
-/// Regex helper functions ///
-//////////////////////////////
-
-function regex($regex) {
-    return '~' . LIB . '(?:' . str_replace('~', '\~', $regex) . ')~';
-}
-
-function magicSplit($regex, $string) {
-    $pieces = preg_split(regex('(?:(?&string)|(?&comment)|(?&code))(*SKIP)(*FAIL)|' . $regex), $string);
-
-    foreach ($pieces as &$piece) {
-        $piece = trim($piece);
-    }
-
-    return array_filter($pieces);
-}

grammar/rebuildParsers.php

@@ -0,0 +1,263 @@
+<?php
+
+$grammarFileToName = [
+    __DIR__ . '/php5.y' => 'Php5',
+    __DIR__ . '/php7.y' => 'Php7',
+];
+
+$tokensFile     = __DIR__ . '/tokens.y';
+$tokensTemplate = __DIR__ . '/tokens.template';
+$skeletonFile   = __DIR__ . '/parser.template';
+$tmpGrammarFile = __DIR__ . '/tmp_parser.phpy';
+$tmpResultFile  = __DIR__ . '/tmp_parser.php';
+$resultDir = __DIR__ . '/../lib/PhpParser/Parser';
+$tokensResultsFile = $resultDir . '/Tokens.php';
+
+// check for kmyacc.exe binary in this directory, otherwise fall back to global name
+$kmyacc = __DIR__ . '/kmyacc.exe';
+if (!file_exists($kmyacc)) {
+    $kmyacc = 'kmyacc';
+}
+
+$options = array_flip($argv);
+$optionDebug = isset($options['--debug']);
+$optionKeepTmpGrammar = isset($options['--keep-tmp-grammar']);
+
+///////////////////////////////
+/// Utility regex constants ///
+///////////////////////////////
+
+const LIB = '(?(DEFINE)
+    (?<singleQuotedString>\'[^\\\\\']*+(?:\\\\.[^\\\\\']*+)*+\')
+    (?<doubleQuotedString>"[^\\\\"]*+(?:\\\\.[^\\\\"]*+)*+")
+    (?<string>(?&singleQuotedString)|(?&doubleQuotedString))
+    (?<comment>/\*[^*]*+(?:\*(?!/)[^*]*+)*+\*/)
+    (?<code>\{[^\'"/{}]*+(?:(?:(?&string)|(?&comment)|(?&code)|/)[^\'"/{}]*+)*+})
+)';
+
+const PARAMS = '\[(?<params>[^[\]]*+(?:\[(?&params)\][^[\]]*+)*+)\]';
+const ARGS   = '\((?<args>[^()]*+(?:\((?&args)\)[^()]*+)*+)\)';
+
+///////////////////
+/// Main script ///
+///////////////////
+
+$tokens = file_get_contents($tokensFile);
+
+foreach ($grammarFileToName as $grammarFile => $name) {
+    echo "Building temporary $name grammar file.\n";
+
+    $grammarCode = file_get_contents($grammarFile);
+    $grammarCode = str_replace('%tokens', $tokens, $grammarCode);
+
+    $grammarCode = resolveNodes($grammarCode);
+    $grammarCode = resolveMacros($grammarCode);
+    $grammarCode = resolveStackAccess($grammarCode);
+
+    file_put_contents($tmpGrammarFile, $grammarCode);
+
+    $additionalArgs = $optionDebug ? '-t -v' : '';
+
+    echo "Building $name parser.\n";
+    $output = trim(shell_exec("$kmyacc $additionalArgs -l -m $skeletonFile -p $name $tmpGrammarFile 2>&1"));
+    echo "Output: \"$output\"\n";
+
+    $resultCode = file_get_contents($tmpResultFile);
+    $resultCode = removeTrailingWhitespace($resultCode);
+
+    ensureDirExists($resultDir);
+    file_put_contents("$resultDir/$name.php", $resultCode);
+    unlink($tmpResultFile);
+
+    echo "Building token definition.\n";
+    $output = trim(shell_exec("$kmyacc -l -m $tokensTemplate $tmpGrammarFile 2>&1"));
+    assert($output === '');
+    rename($tmpResultFile, $tokensResultsFile);
+
+    if (!$optionKeepTmpGrammar) {
+        unlink($tmpGrammarFile);
+    }
+}
+
+///////////////////////////////
+/// Preprocessing functions ///
+///////////////////////////////
+
+function resolveNodes($code) {
+    return preg_replace_callback(
+        '~\b(?<name>[A-Z][a-zA-Z_\\\\]++)\s*' . PARAMS . '~',
+        function($matches) {
+            // recurse
+            $matches['params'] = resolveNodes($matches['params']);
+
+            $params = magicSplit(
+                '(?:' . PARAMS . '|' . ARGS . ')(*SKIP)(*FAIL)|,',
+                $matches['params']
+            );
+
+            $paramCode = '';
+            foreach ($params as $param) {
+                $paramCode .= $param . ', ';
+            }
+
+            return 'new ' . $matches['name'] . '(' . $paramCode . 'attributes())';
+        },
+        $code
+    );
+}
+
+function resolveMacros($code) {
+    return preg_replace_callback(
+        '~\b(?<!::|->)(?!array\()(?<name>[a-z][A-Za-z]++)' . ARGS . '~',
+        function($matches) {
+            // recurse
+            $matches['args'] = resolveMacros($matches['args']);
+
+            $name = $matches['name'];
+            $args = magicSplit(
+                '(?:' . PARAMS . '|' . ARGS . ')(*SKIP)(*FAIL)|,',
+                $matches['args']
+            );
+
+            if ('attributes' == $name) {
+                assertArgs(0, $args, $name);
+                return '$this->startAttributeStack[#1] + $this->endAttributes';
+            }
+
+            if ('stackAttributes' == $name) {
+                assertArgs(1, $args, $name);
+                return '$this->startAttributeStack[' . $args[0] . ']'
+                     . ' + $this->endAttributeStack[' . $args[0] . ']';
+            }
+
+            if ('init' == $name) {
+                return '$$ = array(' . implode(', ', $args) . ')';
+            }
+
+            if ('push' == $name) {
+                assertArgs(2, $args, $name);
+
+                return $args[0] . '[] = ' . $args[1] . '; $$ = ' . $args[0];
+            }
+
+            if ('pushNormalizing' == $name) {
+                assertArgs(2, $args, $name);
+
+                return 'if (is_array(' . $args[1] . ')) { $$ = array_merge(' . $args[0] . ', ' . $args[1] . '); }'
+                     . ' else { ' . $args[0] . '[] = ' . $args[1] . '; $$ = ' . $args[0] . '; }';
+            }
+
+            if ('toArray' == $name) {
+                assertArgs(1, $args, $name);
+
+                return 'is_array(' . $args[0] . ') ? ' . $args[0] . ' : array(' . $args[0] . ')';
+            }
+
+            if ('parseVar' == $name) {
+                assertArgs(1, $args, $name);
+
+                return 'substr(' . $args[0] . ', 1)';
+            }
+
+            if ('parseEncapsed' == $name) {
+                assertArgs(3, $args, $name);
+
+                return 'foreach (' . $args[0] . ' as $s) { if ($s instanceof Node\Scalar\EncapsedStringPart) {'
+                     . ' $s->value = Node\Scalar\String_::parseEscapeSequences($s->value, ' . $args[1] . ', ' . $args[2] . '); } }';
+            }
+
+            if ('parseEncapsedDoc' == $name) {
+                assertArgs(2, $args, $name);
+
+                return 'foreach (' . $args[0] . ' as $s) { if ($s instanceof Node\Scalar\EncapsedStringPart) {'
+                     . ' $s->value = Node\Scalar\String_::parseEscapeSequences($s->value, null, ' . $args[1] . '); } }'
+                     . ' $s->value = preg_replace(\'~(\r\n|\n|\r)\z~\', \'\', $s->value);'
+                     . ' if (\'\' === $s->value) array_pop(' . $args[0] . ');';
+            }
+
+            if ('makeNop' == $name) {
+                assertArgs(2, $args, $name);
+
+                return '$startAttributes = ' . $args[1] . ';'
+                . ' if (isset($startAttributes[\'comments\']))'
+                . ' { ' . $args[0] . ' = new Stmt\Nop([\'comments\' => $startAttributes[\'comments\']]); }'
+                . ' else { ' . $args[0] . ' = null; }';
+            }
+
+            if ('strKind' == $name) {
+                assertArgs(1, $args, $name);
+
+                return '(' . $args[0] . '[0] === "\'" || (' . $args[0] . '[1] === "\'" && '
+                     . '(' . $args[0] . '[0] === \'b\' || ' . $args[0] . '[0] === \'B\')) '
+                     . '? Scalar\String_::KIND_SINGLE_QUOTED : Scalar\String_::KIND_DOUBLE_QUOTED)';
+            }
+
+            if ('setDocStringAttrs' == $name) {
+                assertArgs(2, $args, $name);
+
+                return $args[0] . '[\'kind\'] = strpos(' . $args[1] . ', "\'") === false '
+                     . '? Scalar\String_::KIND_HEREDOC : Scalar\String_::KIND_NOWDOC; '
+                     . 'preg_match(\'/\A[bB]?<<<[ \t]*[\\\'"]?([a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*)[\\\'"]?(?:\r\n|\n|\r)\z/\', ' . $args[1] . ', $matches); '
+                     . $args[0] . '[\'docLabel\'] = $matches[1];';
+            }
+
+            if ('prependLeadingComments' == $name) {
+                assertArgs(1, $args, $name);
+
+                return '$attrs = $this->startAttributeStack[#1]; $stmts = ' . $args[0] . '; '
+                . 'if (!empty($attrs[\'comments\']) && isset($stmts[0])) {'
+                . '$stmts[0]->setAttribute(\'comments\', '
+                . 'array_merge($attrs[\'comments\'], $stmts[0]->getAttribute(\'comments\', []))); }';
+            }
+
+            return $matches[0];
+        },
+        $code
+    );
+}
+
+function assertArgs($num, $args, $name) {
+    if ($num != count($args)) {
+        die('Wrong argument count for ' . $name . '().');
+    }
+}
+
+function resolveStackAccess($code) {
+    $code = preg_replace('/\$\d+/', '$this->semStack[$0]', $code);
+    $code = preg_replace('/#(\d+)/', '$$1', $code);
+    return $code;
+}
+
+function removeTrailingWhitespace($code) {
+    $lines = explode("\n", $code);
+    $lines = array_map('rtrim', $lines);
+    return implode("\n", $lines);
+}
+
+function ensureDirExists($dir) {
+    if (!is_dir($dir)) {
+        mkdir($dir, 0777, true);
+    }
+}
+
+//////////////////////////////
+/// Regex helper functions ///
+//////////////////////////////
+
+function regex($regex) {
+    return '~' . LIB . '(?:' . str_replace('~', '\~', $regex) . ')~';
+}
+
+function magicSplit($regex, $string) {
+    $pieces = preg_split(regex('(?:(?&string)|(?&comment)|(?&code))(*SKIP)(*FAIL)|' . $regex), $string);
+
+    foreach ($pieces as &$piece) {
+        $piece = trim($piece);
+    }
+
+    if ($pieces === ['']) {
+        return [];
+    }
+
+    return $pieces;
+}

grammar/tokens.template

@@ -0,0 +1,17 @@
+<?php
+$meta #
+#semval($) $this->semValue
+#semval($,%t) $this->semValue
+#semval(%n) $this->stackPos-(%l-%n)
+#semval(%n,%t) $this->stackPos-(%l-%n)
+
+namespace PhpParser\Parser;
+#include;
+
+/* GENERATED file based on grammar/tokens.y */
+final class Tokens
+{
+#tokenval
+    const %s = %n;
+#endtokenval
+}

grammar/tokens.y

@@ -0,0 +1,113 @@
+/* We currently rely on the token ID mapping to be the same between PHP 5 and PHP 7 - so the same lexer can be used for
+ * both. This is enforced by sharing this token file. */
+
+%left T_INCLUDE T_INCLUDE_ONCE T_EVAL T_REQUIRE T_REQUIRE_ONCE
+%left ','
+%left T_LOGICAL_OR
+%left T_LOGICAL_XOR
+%left T_LOGICAL_AND
+%right T_PRINT
+%right T_YIELD
+%right T_DOUBLE_ARROW
+%right T_YIELD_FROM
+%left '=' T_PLUS_EQUAL T_MINUS_EQUAL T_MUL_EQUAL T_DIV_EQUAL T_CONCAT_EQUAL T_MOD_EQUAL T_AND_EQUAL T_OR_EQUAL T_XOR_EQUAL T_SL_EQUAL T_SR_EQUAL T_POW_EQUAL
+%left '?' ':'
+%right T_COALESCE
+%left T_BOOLEAN_OR
+%left T_BOOLEAN_AND
+%left '|'
+%left '^'
+%left '&'
+%nonassoc T_IS_EQUAL T_IS_NOT_EQUAL T_IS_IDENTICAL T_IS_NOT_IDENTICAL T_SPACESHIP
+%nonassoc '<' T_IS_SMALLER_OR_EQUAL '>' T_IS_GREATER_OR_EQUAL
+%left T_SL T_SR
+%left '+' '-' '.'
+%left '*' '/' '%'
+%right '!'
+%nonassoc T_INSTANCEOF
+%right '~' T_INC T_DEC T_INT_CAST T_DOUBLE_CAST T_STRING_CAST T_ARRAY_CAST T_OBJECT_CAST T_BOOL_CAST T_UNSET_CAST '@'
+%right T_POW
+%right '['
+%nonassoc T_NEW T_CLONE
+%token T_EXIT
+%token T_IF
+%left T_ELSEIF
+%left T_ELSE
+%left T_ENDIF
+%token T_LNUMBER
+%token T_DNUMBER
+%token T_STRING
+%token T_STRING_VARNAME
+%token T_VARIABLE
+%token T_NUM_STRING
+%token T_INLINE_HTML
+%token T_CHARACTER
+%token T_BAD_CHARACTER
+%token T_ENCAPSED_AND_WHITESPACE
+%token T_CONSTANT_ENCAPSED_STRING
+%token T_ECHO
+%token T_DO
+%token T_WHILE
+%token T_ENDWHILE
+%token T_FOR
+%token T_ENDFOR
+%token T_FOREACH
+%token T_ENDFOREACH
+%token T_DECLARE
+%token T_ENDDECLARE
+%token T_AS
+%token T_SWITCH
+%token T_ENDSWITCH
+%token T_CASE
+%token T_DEFAULT
+%token T_BREAK
+%token T_CONTINUE
+%token T_GOTO
+%token T_FUNCTION
+%token T_CONST
+%token T_RETURN
+%token T_TRY
+%token T_CATCH
+%token T_FINALLY
+%token T_THROW
+%token T_USE
+%token T_INSTEADOF
+%token T_GLOBAL
+%right T_STATIC T_ABSTRACT T_FINAL T_PRIVATE T_PROTECTED T_PUBLIC
+%token T_VAR
+%token T_UNSET
+%token T_ISSET
+%token T_EMPTY
+%token T_HALT_COMPILER
+%token T_CLASS
+%token T_TRAIT
+%token T_INTERFACE
+%token T_EXTENDS
+%token T_IMPLEMENTS
+%token T_OBJECT_OPERATOR
+%token T_DOUBLE_ARROW
+%token T_LIST
+%token T_ARRAY
+%token T_CALLABLE
+%token T_CLASS_C
+%token T_TRAIT_C
+%token T_METHOD_C
+%token T_FUNC_C
+%token T_LINE
+%token T_FILE
+%token T_COMMENT
+%token T_DOC_COMMENT
+%token T_OPEN_TAG
+%token T_OPEN_TAG_WITH_ECHO
+%token T_CLOSE_TAG
+%token T_WHITESPACE
+%token T_START_HEREDOC
+%token T_END_HEREDOC
+%token T_DOLLAR_OPEN_CURLY_BRACES
+%token T_CURLY_OPEN
+%token T_PAAMAYIM_NEKUDOTAYIM
+%token T_NAMESPACE
+%token T_NS_C
+%token T_DIR
+%token T_NS_SEPARATOR
+%token T_ELLIPSIS

lib/PhpParser/Autoloader.php

@@ -7,14 +7,21 @@ namespace PhpParser;
  */
 class Autoloader
 {
+    /** @var bool Whether the autoloader has been registered. */
+    private static $registered = false;
+
     /**
      * Registers PhpParser\Autoloader as an SPL autoloader.
      *
      * @param bool $prepend Whether to prepend the autoloader instead of appending
      */
     static public function register($prepend = false) {
-        ini_set('unserialize_callback_func', 'spl_autoload_call');
+        if (self::$registered === true) {
+            return;
+        }
+
         spl_autoload_register(array(__CLASS__, 'autoload'), true, $prepend);
+        self::$registered = true;
     }
 
     /**
@@ -24,207 +31,10 @@ class Autoloader
      */
     static public function autoload($class) {
         if (0 === strpos($class, 'PhpParser\\')) {
-            $fileName = dirname(__DIR__) . '/' . strtr($class, '\\', '/') . '.php';
+            $fileName = __DIR__ . strtr(substr($class, 9), '\\', '/') . '.php';
             if (file_exists($fileName)) {
                 require $fileName;
             }
-        } else if (0 === strpos($class, 'PHPParser_')) {
-            if (isset(self::$oldToNewMap[$class])) {
-                self::registerLegacyAliases();
-            }
         }
     }
-
-    private static function registerLegacyAliases() {
-        foreach (self::$oldToNewMap as $old => $new) {
-            class_alias($new, $old);
-        }
-    }
-
-    private static $oldToNewMap = array(
-        'PHPParser_Builder' => 'PhpParser\Builder',
-        'PHPParser_BuilderAbstract' => 'PhpParser\BuilderAbstract',
-        'PHPParser_BuilderFactory' => 'PhpParser\BuilderFactory',
-        'PHPParser_Comment' => 'PhpParser\Comment',
-        'PHPParser_Comment_Doc' => 'PhpParser\Comment\Doc',
-        'PHPParser_Error' => 'PhpParser\Error',
-        'PHPParser_Lexer' => 'PhpParser\Lexer',
-        'PHPParser_Lexer_Emulative' => 'PhpParser\Lexer\Emulative',
-        'PHPParser_Node' => 'PhpParser\Node',
-        'PHPParser_NodeAbstract' => 'PhpParser\NodeAbstract',
-        'PHPParser_NodeDumper' => 'PhpParser\NodeDumper',
-        'PHPParser_NodeTraverser' => 'PhpParser\NodeTraverser',
-        'PHPParser_NodeTraverserInterface' => 'PhpParser\NodeTraverserInterface',
-        'PHPParser_NodeVisitor' => 'PhpParser\NodeVisitor',
-        'PHPParser_NodeVisitor_NameResolver' => 'PhpParser\NodeVisitor\NameResolver',
-        'PHPParser_NodeVisitorAbstract' => 'PhpParser\NodeVisitorAbstract',
-        'PHPParser_Parser' => 'PhpParser\Parser',
-        'PHPParser_PrettyPrinterAbstract' => 'PhpParser\PrettyPrinterAbstract',
-        'PHPParser_PrettyPrinter_Default' => 'PhpParser\PrettyPrinter\Standard',
-        'PHPParser_PrettyPrinter_Zend' => 'PhpParser\PrettyPrinter\Standard',
-        'PHPParser_Serializer' => 'PhpParser\Serializer',
-        'PHPParser_Serializer_XML' => 'PhpParser\Serializer\XML',
-        'PHPParser_Unserializer' => 'PhpParser\Unserializer',
-        'PHPParser_Unserializer_XML' => 'PhpParser\Unserializer\XML',
-
-        'PHPParser_Builder_Class' => 'PhpParser\Builder\Class_',
-        'PHPParser_Builder_Function' => 'PhpParser\Builder\Function_',
-        'PHPParser_Builder_Interface' => 'PhpParser\Builder\Interface_',
-        'PHPParser_Builder_Method' => 'PhpParser\Builder\Method',
-        'PHPParser_Builder_Param' => 'PhpParser\Builder\Param',
-        'PHPParser_Builder_Property' => 'PhpParser\Builder\Property',
-
-        'PHPParser_Node_Arg' => 'PhpParser\Node\Arg',
-        'PHPParser_Node_Const' => 'PhpParser\Node\Const_',
-        'PHPParser_Node_Expr' => 'PhpParser\Node\Expr',
-        'PHPParser_Node_Name' => 'PhpParser\Node\Name',
-        'PHPParser_Node_Name_FullyQualified' => 'PhpParser\Node\Name\FullyQualified',
-        'PHPParser_Node_Name_Relative' => 'PhpParser\Node\Name\Relative',
-        'PHPParser_Node_Param' => 'PhpParser\Node\Param',
-        'PHPParser_Node_Scalar' => 'PhpParser\Node\Scalar',
-        'PHPParser_Node_Stmt' => 'PhpParser\Node\Stmt',
-
-        'PHPParser_Node_Stmt_Break' => 'PhpParser\Node\Stmt\Break_',
-        'PHPParser_Node_Stmt_Case' => 'PhpParser\Node\Stmt\Case_',
-        'PHPParser_Node_Stmt_Catch' => 'PhpParser\Node\Stmt\Catch_',
-        'PHPParser_Node_Stmt_Class' => 'PhpParser\Node\Stmt\Class_',
-        'PHPParser_Node_Stmt_ClassConst' => 'PhpParser\Node\Stmt\ClassConst',
-        'PHPParser_Node_Stmt_ClassMethod' => 'PhpParser\Node\Stmt\ClassMethod',
-        'PHPParser_Node_Stmt_Const' => 'PhpParser\Node\Stmt\Const_',
-        'PHPParser_Node_Stmt_Continue' => 'PhpParser\Node\Stmt\Continue_',
-        'PHPParser_Node_Stmt_Declare' => 'PhpParser\Node\Stmt\Declare_',
-        'PHPParser_Node_Stmt_DeclareDeclare' => 'PhpParser\Node\Stmt\DeclareDeclare',
-        'PHPParser_Node_Stmt_Do' => 'PhpParser\Node\Stmt\Do_',
-        'PHPParser_Node_Stmt_Echo' => 'PhpParser\Node\Stmt\Echo_',
-        'PHPParser_Node_Stmt_Else' => 'PhpParser\Node\Stmt\Else_',
-        'PHPParser_Node_Stmt_ElseIf' => 'PhpParser\Node\Stmt\ElseIf_',
-        'PHPParser_Node_Stmt_For' => 'PhpParser\Node\Stmt\For_',
-        'PHPParser_Node_Stmt_Foreach' => 'PhpParser\Node\Stmt\Foreach_',
-        'PHPParser_Node_Stmt_Function' => 'PhpParser\Node\Stmt\Function_',
-        'PHPParser_Node_Stmt_Global' => 'PhpParser\Node\Stmt\Global_',
-        'PHPParser_Node_Stmt_Goto' => 'PhpParser\Node\Stmt\Goto_',
-        'PHPParser_Node_Stmt_HaltCompiler' => 'PhpParser\Node\Stmt\HaltCompiler',
-        'PHPParser_Node_Stmt_If' => 'PhpParser\Node\Stmt\If_',
-        'PHPParser_Node_Stmt_InlineHTML' => 'PhpParser\Node\Stmt\InlineHTML',
-        'PHPParser_Node_Stmt_Interface' => 'PhpParser\Node\Stmt\Interface_',
-        'PHPParser_Node_Stmt_Label' => 'PhpParser\Node\Stmt\Label',
-        'PHPParser_Node_Stmt_Namespace' => 'PhpParser\Node\Stmt\Namespace_',
-        'PHPParser_Node_Stmt_Property' => 'PhpParser\Node\Stmt\Property',
-        'PHPParser_Node_Stmt_PropertyProperty' => 'PhpParser\Node\Stmt\PropertyProperty',
-        'PHPParser_Node_Stmt_Return' => 'PhpParser\Node\Stmt\Return_',
-        'PHPParser_Node_Stmt_Static' => 'PhpParser\Node\Stmt\Static_',
-        'PHPParser_Node_Stmt_StaticVar' => 'PhpParser\Node\Stmt\StaticVar',
-        'PHPParser_Node_Stmt_Switch' => 'PhpParser\Node\Stmt\Switch_',
-        'PHPParser_Node_Stmt_Throw' => 'PhpParser\Node\Stmt\Throw_',
-        'PHPParser_Node_Stmt_Trait' => 'PhpParser\Node\Stmt\Trait_',
-        'PHPParser_Node_Stmt_TraitUse' => 'PhpParser\Node\Stmt\TraitUse',
-        'PHPParser_Node_Stmt_TraitUseAdaptation' => 'PhpParser\Node\Stmt\TraitUseAdaptation',
-        'PHPParser_Node_Stmt_TraitUseAdaptation_Alias' => 'PhpParser\Node\Stmt\TraitUseAdaptation\Alias',
-        'PHPParser_Node_Stmt_TraitUseAdaptation_Precedence' => 'PhpParser\Node\Stmt\TraitUseAdaptation\Precedence',
-        'PHPParser_Node_Stmt_TryCatch' => 'PhpParser\Node\Stmt\TryCatch',
-        'PHPParser_Node_Stmt_Unset' => 'PhpParser\Node\Stmt\Unset_',
-        'PHPParser_Node_Stmt_UseUse' => 'PhpParser\Node\Stmt\UseUse',
-        'PHPParser_Node_Stmt_Use' => 'PhpParser\Node\Stmt\Use_',
-        'PHPParser_Node_Stmt_While' => 'PhpParser\Node\Stmt\While_',
-
-        'PHPParser_Node_Expr_AssignBitwiseAnd' => 'PhpParser\Node\Expr\AssignOp\BitwiseAnd',
-        'PHPParser_Node_Expr_AssignBitwiseOr' => 'PhpParser\Node\Expr\AssignOp\BitwiseOr',
-        'PHPParser_Node_Expr_AssignBitwiseXor' => 'PhpParser\Node\Expr\AssignOp\BitwiseXor',
-        'PHPParser_Node_Expr_AssignConcat' => 'PhpParser\Node\Expr\AssignOp\Concat',
-        'PHPParser_Node_Expr_AssignDiv' => 'PhpParser\Node\Expr\AssignOp\Div',
-        'PHPParser_Node_Expr_AssignMinus' => 'PhpParser\Node\Expr\AssignOp\Minus',
-        'PHPParser_Node_Expr_AssignMod' => 'PhpParser\Node\Expr\AssignOp\Mod',
-        'PHPParser_Node_Expr_AssignMul' => 'PhpParser\Node\Expr\AssignOp\Mul',
-        'PHPParser_Node_Expr_AssignPlus' => 'PhpParser\Node\Expr\AssignOp\Plus',
-        'PHPParser_Node_Expr_AssignShiftLeft' => 'PhpParser\Node\Expr\AssignOp\ShiftLeft',
-        'PHPParser_Node_Expr_AssignShiftRight' => 'PhpParser\Node\Expr\AssignOp\ShiftRight',
-
-        'PHPParser_Node_Expr_Cast' => 'PhpParser\Node\Expr\Cast',
-        'PHPParser_Node_Expr_Cast_Array' => 'PhpParser\Node\Expr\Cast\Array_',
-        'PHPParser_Node_Expr_Cast_Bool' => 'PhpParser\Node\Expr\Cast\Bool',
-        'PHPParser_Node_Expr_Cast_Double' => 'PhpParser\Node\Expr\Cast\Double',
-        'PHPParser_Node_Expr_Cast_Int' => 'PhpParser\Node\Expr\Cast\Int',
-        'PHPParser_Node_Expr_Cast_Object' => 'PhpParser\Node\Expr\Cast\Object',
-        'PHPParser_Node_Expr_Cast_String' => 'PhpParser\Node\Expr\Cast\String',
-        'PHPParser_Node_Expr_Cast_Unset' => 'PhpParser\Node\Expr\Cast\Unset_',
-
-        'PHPParser_Node_Expr_BitwiseAnd' => 'PhpParser\Node\Expr\BinaryOp\BitwiseAnd',
-        'PHPParser_Node_Expr_BitwiseOr' => 'PhpParser\Node\Expr\BinaryOp\BitwiseOr',
-        'PHPParser_Node_Expr_BitwiseXor' => 'PhpParser\Node\Expr\BinaryOp\BitwiseXor',
-        'PHPParser_Node_Expr_BooleanAnd' => 'PhpParser\Node\Expr\BinaryOp\BooleanAnd',
-        'PHPParser_Node_Expr_BooleanOr' => 'PhpParser\Node\Expr\BinaryOp\BooleanOr',
-        'PHPParser_Node_Expr_Concat' => 'PhpParser\Node\Expr\BinaryOp\Concat',
-        'PHPParser_Node_Expr_Div' => 'PhpParser\Node\Expr\BinaryOp\Div',
-        'PHPParser_Node_Expr_Equal' => 'PhpParser\Node\Expr\BinaryOp\Equal',
-        'PHPParser_Node_Expr_Greater' => 'PhpParser\Node\Expr\BinaryOp\Greater',
-        'PHPParser_Node_Expr_GreaterOrEqual' => 'PhpParser\Node\Expr\BinaryOp\GreaterOrEqual',
-        'PHPParser_Node_Expr_Identical' => 'PhpParser\Node\Expr\BinaryOp\Identical',
-        'PHPParser_Node_Expr_LogicalAnd' => 'PhpParser\Node\Expr\BinaryOp\LogicalAnd',
-        'PHPParser_Node_Expr_LogicalOr' => 'PhpParser\Node\Expr\BinaryOp\LogicalOr',
-        'PHPParser_Node_Expr_LogicalXor' => 'PhpParser\Node\Expr\BinaryOp\LogicalXor',
-        'PHPParser_Node_Expr_Minus' => 'PhpParser\Node\Expr\BinaryOp\Minus',
-        'PHPParser_Node_Expr_Mod' => 'PhpParser\Node\Expr\BinaryOp\Mod',
-        'PHPParser_Node_Expr_Mul' => 'PhpParser\Node\Expr\BinaryOp\Mul',
-        'PHPParser_Node_Expr_NotEqual' => 'PhpParser\Node\Expr\BinaryOp\NotEqual',
-        'PHPParser_Node_Expr_NotIdentical' => 'PhpParser\Node\Expr\BinaryOp\NotIdentical',
-        'PHPParser_Node_Expr_Plus' => 'PhpParser\Node\Expr\BinaryOp\Plus',
-        'PHPParser_Node_Expr_ShiftLeft' => 'PhpParser\Node\Expr\BinaryOp\ShiftLeft',
-        'PHPParser_Node_Expr_ShiftRight' => 'PhpParser\Node\Expr\BinaryOp\ShiftRight',
-        'PHPParser_Node_Expr_Smaller' => 'PhpParser\Node\Expr\BinaryOp\Smaller',
-        'PHPParser_Node_Expr_SmallerOrEqual' => 'PhpParser\Node\Expr\BinaryOp\SmallerOrEqual',
-
-        'PHPParser_Node_Expr_Array' => 'PhpParser\Node\Expr\Array_',
-        'PHPParser_Node_Expr_ArrayDimFetch' => 'PhpParser\Node\Expr\ArrayDimFetch',
-        'PHPParser_Node_Expr_ArrayItem' => 'PhpParser\Node\Expr\ArrayItem',
-        'PHPParser_Node_Expr_Assign' => 'PhpParser\Node\Expr\Assign',
-        'PHPParser_Node_Expr_AssignRef' => 'PhpParser\Node\Expr\AssignRef',
-        'PHPParser_Node_Expr_BitwiseNot' => 'PhpParser\Node\Expr\BitwiseNot',
-        'PHPParser_Node_Expr_BooleanNot' => 'PhpParser\Node\Expr\BooleanNot',
-        'PHPParser_Node_Expr_ClassConstFetch' => 'PhpParser\Node\Expr\ClassConstFetch',
-        'PHPParser_Node_Expr_Clone' => 'PhpParser\Node\Expr\Clone_',
-        'PHPParser_Node_Expr_Closure' => 'PhpParser\Node\Expr\Closure',
-        'PHPParser_Node_Expr_ClosureUse' => 'PhpParser\Node\Expr\ClosureUse',
-        'PHPParser_Node_Expr_ConstFetch' => 'PhpParser\Node\Expr\ConstFetch',
-        'PHPParser_Node_Expr_Empty' => 'PhpParser\Node\Expr\Empty_',
-        'PHPParser_Node_Expr_ErrorSuppress' => 'PhpParser\Node\Expr\ErrorSuppress',
-        'PHPParser_Node_Expr_Eval' => 'PhpParser\Node\Expr\Eval_',
-        'PHPParser_Node_Expr_Exit' => 'PhpParser\Node\Expr\Exit_',
-        'PHPParser_Node_Expr_FuncCall' => 'PhpParser\Node\Expr\FuncCall',
-        'PHPParser_Node_Expr_Include' => 'PhpParser\Node\Expr\Include_',
-        'PHPParser_Node_Expr_Instanceof' => 'PhpParser\Node\Expr\Instanceof_',
-        'PHPParser_Node_Expr_Isset' => 'PhpParser\Node\Expr\Isset_',
-        'PHPParser_Node_Expr_List' => 'PhpParser\Node\Expr\List_',
-        'PHPParser_Node_Expr_MethodCall' => 'PhpParser\Node\Expr\MethodCall',
-        'PHPParser_Node_Expr_New' => 'PhpParser\Node\Expr\New_',
-        'PHPParser_Node_Expr_PostDec' => 'PhpParser\Node\Expr\PostDec',
-        'PHPParser_Node_Expr_PostInc' => 'PhpParser\Node\Expr\PostInc',
-        'PHPParser_Node_Expr_PreDec' => 'PhpParser\Node\Expr\PreDec',
-        'PHPParser_Node_Expr_PreInc' => 'PhpParser\Node\Expr\PreInc',
-        'PHPParser_Node_Expr_Print' => 'PhpParser\Node\Expr\Print_',
-        'PHPParser_Node_Expr_PropertyFetch' => 'PhpParser\Node\Expr\PropertyFetch',
-        'PHPParser_Node_Expr_ShellExec' => 'PhpParser\Node\Expr\ShellExec',
-        'PHPParser_Node_Expr_StaticCall' => 'PhpParser\Node\Expr\StaticCall',
-        'PHPParser_Node_Expr_StaticPropertyFetch' => 'PhpParser\Node\Expr\StaticPropertyFetch',
-        'PHPParser_Node_Expr_Ternary' => 'PhpParser\Node\Expr\Ternary',
-        'PHPParser_Node_Expr_UnaryMinus' => 'PhpParser\Node\Expr\UnaryMinus',
-        'PHPParser_Node_Expr_UnaryPlus' => 'PhpParser\Node\Expr\UnaryPlus',
-        'PHPParser_Node_Expr_Variable' => 'PhpParser\Node\Expr\Variable',
-        'PHPParser_Node_Expr_Yield' => 'PhpParser\Node\Expr\Yield_',
-
-        'PHPParser_Node_Scalar_ClassConst' => 'PhpParser\Node\Scalar\MagicConst\Class_',
-        'PHPParser_Node_Scalar_DirConst' => 'PhpParser\Node\Scalar\MagicConst\Dir',
-        'PHPParser_Node_Scalar_FileConst' => 'PhpParser\Node\Scalar\MagicConst\File',
-        'PHPParser_Node_Scalar_FuncConst' => 'PhpParser\Node\Scalar\MagicConst\Function_',
-        'PHPParser_Node_Scalar_LineConst' => 'PhpParser\Node\Scalar\MagicConst\Line',
-        'PHPParser_Node_Scalar_MethodConst' => 'PhpParser\Node\Scalar\MagicConst\Method',
-        'PHPParser_Node_Scalar_NSConst' => 'PhpParser\Node\Scalar\MagicConst\Namespace_',
-        'PHPParser_Node_Scalar_TraitConst' => 'PhpParser\Node\Scalar\MagicConst\Trait_',
-
-        'PHPParser_Node_Scalar_DNumber' => 'PhpParser\Node\Scalar\DNumber',
-        'PHPParser_Node_Scalar_Encapsed' => 'PhpParser\Node\Scalar\Encapsed',
-        'PHPParser_Node_Scalar_LNumber' => 'PhpParser\Node\Scalar\LNumber',
-        'PHPParser_Node_Scalar_String' => 'PhpParser\Node\Scalar\String',
-    );
 }
-
-class_alias('PhpParser\Autoloader', 'PHPParser_Autoloader');

lib/PhpParser/Builder/Class_.php

@@ -6,18 +6,18 @@ use PhpParser;
 use PhpParser\Node\Name;
 use PhpParser\Node\Stmt;
 
-class Class_ extends PhpParser\BuilderAbstract
+class Class_ extends Declaration
 {
     protected $name;
 
-    protected $extends;
-    protected $implements;
-    protected $type;
+    protected $extends = null;
+    protected $implements = array();
+    protected $flags = 0;
 
-    protected $uses;
-    protected $constants;
-    protected $properties;
-    protected $methods;
+    protected $uses = array();
+    protected $constants = array();
+    protected $properties = array();
+    protected $methods = array();
 
     /**
      * Creates a class builder.
@@ -26,12 +26,6 @@ class Class_ extends PhpParser\BuilderAbstract
      */
     public function __construct($name) {
         $this->name = $name;
-
-        $this->type = 0;
-        $this->extends = null;
-        $this->implements = array();
-
-        $this->uses = $this->constants = $this->properties = $this->methods = array();
     }
 
     /**
@@ -39,7 +33,7 @@ class Class_ extends PhpParser\BuilderAbstract
      *
      * @param Name|string $class Name of class to extend
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function extend($class) {
         $this->extends = $this->normalizeName($class);
@@ -50,10 +44,9 @@ class Class_ extends PhpParser\BuilderAbstract
     /**
      * Implements one or more interfaces.
      *
-     * @param Name|string $interface Name of interface to implement
-     * @param Name|string $...       More interfaces to implement
+     * @param Name|string ...$interfaces Names of interfaces to implement
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function implement() {
         foreach (func_get_args() as $interface) {
@@ -66,7 +59,7 @@ class Class_ extends PhpParser\BuilderAbstract
     /**
      * Makes the class abstract.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeAbstract() {
         $this->setModifier(Stmt\Class_::MODIFIER_ABSTRACT);
@@ -77,7 +70,7 @@ class Class_ extends PhpParser\BuilderAbstract
     /**
      * Makes the class final.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeFinal() {
         $this->setModifier(Stmt\Class_::MODIFIER_FINAL);
@@ -90,7 +83,7 @@ class Class_ extends PhpParser\BuilderAbstract
      *
      * @param Stmt|PhpParser\Builder $stmt The statement to add
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function addStmt($stmt) {
         $stmt = $this->normalizeNode($stmt);
@@ -112,21 +105,6 @@ class Class_ extends PhpParser\BuilderAbstract
         return $this;
     }
 
-    /**
-     * Adds multiple statements.
-     *
-     * @param array $stmts The statements to add
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function addStmts(array $stmts) {
-        foreach ($stmts as $stmt) {
-            $this->addStmt($stmt);
-        }
-
-        return $this;
-    }
-
     /**
      * Returns the built class node.
      *
@@ -134,10 +112,10 @@ class Class_ extends PhpParser\BuilderAbstract
      */
     public function getNode() {
         return new Stmt\Class_($this->name, array(
-            'type' => $this->type,
+            'flags' => $this->flags,
             'extends' => $this->extends,
             'implements' => $this->implements,
             'stmts' => array_merge($this->uses, $this->constants, $this->properties, $this->methods),
-        ));
+        ), $this->attributes);
     }
 }

lib/PhpParser/Builder/Declaration.php

@@ -0,0 +1,42 @@
+<?php
+
+namespace PhpParser\Builder;
+
+use PhpParser;
+
+abstract class Declaration extends PhpParser\BuilderAbstract
+{
+    protected $attributes = array();
+
+    abstract public function addStmt($stmt);
+
+    /**
+     * Adds multiple statements.
+     *
+     * @param array $stmts The statements to add
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function addStmts(array $stmts) {
+        foreach ($stmts as $stmt) {
+            $this->addStmt($stmt);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Sets doc comment for the declaration.
+     *
+     * @param PhpParser\Comment\Doc|string $docComment Doc comment to set
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function setDocComment($docComment) {
+        $this->attributes['comments'] = array(
+            $this->normalizeDocComment($docComment)
+        );
+
+        return $this;
+    }
+}

lib/PhpParser/Builder/FunctionLike.php

@@ -0,0 +1,75 @@
+<?php
+
+namespace PhpParser\Builder;
+
+use PhpParser;
+use PhpParser\Node;
+
+abstract class FunctionLike extends Declaration
+{
+    protected $returnByRef = false;
+    protected $params = array();
+
+    /** @var string|Node\Name|Node\NullableType|null */
+    protected $returnType = null;
+
+    /**
+     * Make the function return by reference.
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function makeReturnByRef() {
+        $this->returnByRef = true;
+
+        return $this;
+    }
+
+    /**
+     * Adds a parameter.
+     *
+     * @param Node\Param|Param $param The parameter to add
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function addParam($param) {
+        $param = $this->normalizeNode($param);
+
+        if (!$param instanceof Node\Param) {
+            throw new \LogicException(sprintf('Expected parameter node, got "%s"', $param->getType()));
+        }
+
+        $this->params[] = $param;
+
+        return $this;
+    }
+
+    /**
+     * Adds multiple parameters.
+     *
+     * @param array $params The parameters to add
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function addParams(array $params) {
+        foreach ($params as $param) {
+            $this->addParam($param);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Sets the return type for PHP 7.
+     *
+     * @param string|Node\Name|Node\NullableType $type One of array, callable, string, int, float, bool, iterable,
+     *                               or a class/interface name.
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function setReturnType($type)
+    {
+        $this->returnType = $this->normalizeType($type);
+
+        return $this;
+    }
+}

lib/PhpParser/Builder/Function_.php

@@ -6,13 +6,10 @@ use PhpParser;
 use PhpParser\Node;
 use PhpParser\Node\Stmt;
 
-class Function_ extends PhpParser\BuilderAbstract
+class Function_ extends FunctionLike
 {
     protected $name;
-
-    protected $returnByRef;
-    protected $params;
-    protected $stmts;
+    protected $stmts = array();
 
     /**
      * Creates a function builder.
@@ -21,55 +18,6 @@ class Function_ extends PhpParser\BuilderAbstract
      */
     public function __construct($name) {
         $this->name = $name;
-
-        $this->returnByRef = false;
-        $this->params = array();
-        $this->stmts = array();
-    }
-
-    /**
-     * Make the function return by reference.
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function makeReturnByRef() {
-        $this->returnByRef = true;
-
-        return $this;
-    }
-
-    /**
-     * Adds a parameter.
-     *
-     * @param Node\Param|Param $param The parameter to add
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function addParam($param) {
-        $param = $this->normalizeNode($param);
-
-        if (!$param instanceof Node\Param) {
-            throw new \LogicException(sprintf('Expected parameter node, got "%s"', $param->getType()));
-        }
-
-        $this->params[] = $param;
-
-        return $this;
-    }
-
-    /**
-     * Adds multiple parameters.
-     *
-     * @param array $params The parameters to add
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function addParams(array $params) {
-        foreach ($params as $param) {
-            $this->addParam($param);
-        }
-
-        return $this;
     }
 
     /**
@@ -77,7 +25,7 @@ class Function_ extends PhpParser\BuilderAbstract
      *
      * @param Node|PhpParser\Builder $stmt The statement to add
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function addStmt($stmt) {
         $this->stmts[] = $this->normalizeNode($stmt);
@@ -85,21 +33,6 @@ class Function_ extends PhpParser\BuilderAbstract
         return $this;
     }
 
-    /**
-     * Adds multiple statements.
-     *
-     * @param array $stmts The statements to add
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function addStmts(array $stmts) {
-        foreach ($stmts as $stmt) {
-            $this->addStmt($stmt);
-        }
-
-        return $this;
-    }
-
     /**
      * Returns the built function node.
      *
@@ -107,9 +40,10 @@ class Function_ extends PhpParser\BuilderAbstract
      */
     public function getNode() {
         return new Stmt\Function_($this->name, array(
-            'byRef'  => $this->returnByRef,
-            'params' => $this->params,
-            'stmts'  => $this->stmts,
-        ));
+            'byRef'      => $this->returnByRef,
+            'params'     => $this->params,
+            'returnType' => $this->returnType,
+            'stmts'      => $this->stmts,
+        ), $this->attributes);
     }
-}
+}

lib/PhpParser/Builder/Interface_.php

@@ -6,12 +6,12 @@ use PhpParser;
 use PhpParser\Node\Name;
 use PhpParser\Node\Stmt;
 
-class Interface_ extends PhpParser\BuilderAbstract
+class Interface_ extends Declaration
 {
     protected $name;
-    protected $extends;
-    protected $constants;
-    protected $methods;
+    protected $extends = array();
+    protected $constants = array();
+    protected $methods = array();
 
     /**
      * Creates an interface builder.
@@ -20,17 +20,14 @@ class Interface_ extends PhpParser\BuilderAbstract
      */
     public function __construct($name) {
         $this->name = $name;
-        $this->extends = array();
-        $this->constants = $this->methods = array();
     }
 
     /**
      * Extends one or more interfaces.
      *
-     * @param Name|string $interface Name of interface to extend
-     * @param Name|string $...       More interfaces to extend
+     * @param Name|string ...$interfaces Names of interfaces to extend
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function extend() {
         foreach (func_get_args() as $interface) {
@@ -45,7 +42,7 @@ class Interface_ extends PhpParser\BuilderAbstract
      *
      * @param Stmt|PhpParser\Builder $stmt The statement to add
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function addStmt($stmt) {
         $stmt = $this->normalizeNode($stmt);
@@ -70,22 +67,7 @@ class Interface_ extends PhpParser\BuilderAbstract
     }
 
     /**
-     * Adds multiple statements.
-     *
-     * @param array $stmts The statements to add
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function addStmts(array $stmts) {
-        foreach ($stmts as $stmt) {
-            $this->addStmt($stmt);
-        }
-
-        return $this;
-    }
-
-    /**
-     * Returns the built class node.
+     * Returns the built interface node.
      *
      * @return Stmt\Interface_ The built interface node
      */
@@ -93,6 +75,6 @@ class Interface_ extends PhpParser\BuilderAbstract
         return new Stmt\Interface_($this->name, array(
             'extends' => $this->extends,
             'stmts' => array_merge($this->constants, $this->methods),
-        ));
+        ), $this->attributes);
     }
 }

lib/PhpParser/Builder/Method.php

@@ -6,14 +6,13 @@ use PhpParser;
 use PhpParser\Node;
 use PhpParser\Node\Stmt;
 
-class Method extends PhpParser\BuilderAbstract
+class Method extends FunctionLike
 {
     protected $name;
+    protected $flags = 0;
 
-    protected $type;
-    protected $returnByRef;
-    protected $params;
-    protected $stmts;
+    /** @var array|null */
+    protected $stmts = array();
 
     /**
      * Creates a method builder.
@@ -22,17 +21,12 @@ class Method extends PhpParser\BuilderAbstract
      */
     public function __construct($name) {
         $this->name = $name;
-
-        $this->type = 0;
-        $this->returnByRef = false;
-        $this->params = array();
-        $this->stmts = array();
     }
 
     /**
      * Makes the method public.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makePublic() {
         $this->setModifier(Stmt\Class_::MODIFIER_PUBLIC);
@@ -43,7 +37,7 @@ class Method extends PhpParser\BuilderAbstract
     /**
      * Makes the method protected.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeProtected() {
         $this->setModifier(Stmt\Class_::MODIFIER_PROTECTED);
@@ -54,7 +48,7 @@ class Method extends PhpParser\BuilderAbstract
     /**
      * Makes the method private.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makePrivate() {
         $this->setModifier(Stmt\Class_::MODIFIER_PRIVATE);
@@ -65,7 +59,7 @@ class Method extends PhpParser\BuilderAbstract
     /**
      * Makes the method static.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeStatic() {
         $this->setModifier(Stmt\Class_::MODIFIER_STATIC);
@@ -76,7 +70,7 @@ class Method extends PhpParser\BuilderAbstract
     /**
      * Makes the method abstract.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeAbstract() {
         if (!empty($this->stmts)) {
@@ -92,7 +86,7 @@ class Method extends PhpParser\BuilderAbstract
     /**
      * Makes the method final.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeFinal() {
         $this->setModifier(Stmt\Class_::MODIFIER_FINAL);
@@ -100,57 +94,12 @@ class Method extends PhpParser\BuilderAbstract
         return $this;
     }
 
-    /**
-     * Make the method return by reference.
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function makeReturnByRef() {
-        $this->returnByRef = true;
-
-        return $this;
-    }
-
-    /**
-     * Adds a parameter.
-     *
-     * @param Node\Param|Param $param The parameter to add
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function addParam($param) {
-        $param = $this->normalizeNode($param);
-
-        if (!$param instanceof Node\Param) {
-            throw new \LogicException(sprintf('Expected parameter node, got "%s"', $param->getType()));
-        }
-
-        $this->params[] = $param;
-
-        return $this;
-    }
-
-    /**
-     * Adds multiple parameters.
-     *
-     * @param array $params The parameters to add
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function addParams(array $params) {
-        foreach ($params as $param) {
-            $this->addParam($param);
-        }
-
-        return $this;
-    }
-
     /**
      * Adds a statement.
      *
      * @param Node|PhpParser\Builder $stmt The statement to add
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function addStmt($stmt) {
         if (null === $this->stmts) {
@@ -162,21 +111,6 @@ class Method extends PhpParser\BuilderAbstract
         return $this;
     }
 
-    /**
-     * Adds multiple statements.
-     *
-     * @param array $stmts The statements to add
-     *
-     * @return self The builder instance (for fluid interface)
-     */
-    public function addStmts(array $stmts) {
-        foreach ($stmts as $stmt) {
-            $this->addStmt($stmt);
-        }
-
-        return $this;
-    }
-
     /**
      * Returns the built method node.
      *
@@ -184,10 +118,11 @@ class Method extends PhpParser\BuilderAbstract
      */
     public function getNode() {
         return new Stmt\ClassMethod($this->name, array(
-            'type'   => $this->type !== 0 ? $this->type : Stmt\Class_::MODIFIER_PUBLIC,
-            'byRef'  => $this->returnByRef,
-            'params' => $this->params,
-            'stmts'  => $this->stmts,
-        ));
+            'flags'      => $this->flags,
+            'byRef'      => $this->returnByRef,
+            'params'     => $this->params,
+            'returnType' => $this->returnType,
+            'stmts'      => $this->stmts,
+        ), $this->attributes);
     }
-}
+}

lib/PhpParser/Builder/Namespace_.php

@@ -0,0 +1,59 @@
+<?php
+
+namespace PhpParser\Builder;
+
+use PhpParser;
+use PhpParser\Node;
+use PhpParser\Node\Stmt;
+
+class Namespace_ extends PhpParser\BuilderAbstract
+{
+    private $name;
+    private $stmts = array();
+
+    /**
+     * Creates a namespace builder.
+     *
+     * @param Node\Name|string|null $name Name of the namespace
+     */
+    public function __construct($name) {
+        $this->name = null !== $name ? $this->normalizeName($name) : null;
+    }
+
+    /**
+     * Adds a statement.
+     *
+     * @param Node|PhpParser\Builder $stmt The statement to add
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function addStmt($stmt) {
+        $this->stmts[] = $this->normalizeNode($stmt);
+
+        return $this;
+    }
+
+    /**
+     * Adds multiple statements.
+     *
+     * @param array $stmts The statements to add
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function addStmts(array $stmts) {
+        foreach ($stmts as $stmt) {
+            $this->addStmt($stmt);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Returns the built node.
+     *
+     * @return Node The built node
+     */
+    public function getNode() {
+        return new Stmt\Namespace_($this->name, $this->stmts);
+    }
+}

lib/PhpParser/Builder/Param.php

@@ -9,9 +9,14 @@ class Param extends PhpParser\BuilderAbstract
 {
     protected $name;
 
-    protected $default;
-    protected $type;
-    protected $byRef;
+    protected $default = null;
+
+    /** @var string|Node\Name|Node\NullableType|null */
+    protected $type = null;
+
+    protected $byRef = false;
+
+    protected $variadic = false;
 
     /**
      * Creates a parameter builder.
@@ -20,10 +25,6 @@ class Param extends PhpParser\BuilderAbstract
      */
     public function __construct($name) {
         $this->name = $name;
-
-        $this->default = null;
-        $this->type = null;
-        $this->byRef = false;
     }
 
     /**
@@ -31,7 +32,7 @@ class Param extends PhpParser\BuilderAbstract
      *
      * @param mixed $value Default value to use
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function setDefault($value) {
         $this->default = $this->normalizeValue($value);
@@ -42,15 +43,14 @@ class Param extends PhpParser\BuilderAbstract
     /**
      * Sets type hint for the parameter.
      *
-     * @param string|Node\Name $type Type hint to use
+     * @param string|Node\Name|Node\NullableType $type Type hint to use
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function setTypeHint($type) {
-        if ($type === 'array' || $type === 'callable') {
-            $this->type = $type;
-        } else {
-            $this->type = $this->normalizeName($type);
+        $this->type = $this->normalizeType($type);
+        if ($this->type === 'void') {
+            throw new \LogicException('Parameter type cannot be void');
         }
 
         return $this;
@@ -59,7 +59,7 @@ class Param extends PhpParser\BuilderAbstract
     /**
      * Make the parameter accept the value by reference.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeByRef() {
         $this->byRef = true;
@@ -67,6 +67,17 @@ class Param extends PhpParser\BuilderAbstract
         return $this;
     }
 
+    /**
+     * Make the parameter variadic
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function makeVariadic() {
+        $this->variadic = true;
+
+        return $this;
+    }
+
     /**
      * Returns the built parameter node.
      *
@@ -74,7 +85,7 @@ class Param extends PhpParser\BuilderAbstract
      */
     public function getNode() {
         return new Node\Param(
-            $this->name, $this->default, $this->type, $this->byRef
+            $this->name, $this->default, $this->type, $this->byRef, $this->variadic
         );
     }
-}
+}

lib/PhpParser/Builder/Property.php

@@ -9,8 +9,9 @@ class Property extends PhpParser\BuilderAbstract
 {
     protected $name;
 
-    protected $type;
-    protected $default;
+    protected $flags = 0;
+    protected $default = null;
+    protected $attributes = array();
 
     /**
      * Creates a property builder.
@@ -19,15 +20,12 @@ class Property extends PhpParser\BuilderAbstract
      */
     public function __construct($name) {
         $this->name = $name;
-
-        $this->type = 0;
-        $this->default = null;
     }
 
     /**
      * Makes the property public.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makePublic() {
         $this->setModifier(Stmt\Class_::MODIFIER_PUBLIC);
@@ -38,7 +36,7 @@ class Property extends PhpParser\BuilderAbstract
     /**
      * Makes the property protected.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeProtected() {
         $this->setModifier(Stmt\Class_::MODIFIER_PROTECTED);
@@ -49,7 +47,7 @@ class Property extends PhpParser\BuilderAbstract
     /**
      * Makes the property private.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makePrivate() {
         $this->setModifier(Stmt\Class_::MODIFIER_PRIVATE);
@@ -60,7 +58,7 @@ class Property extends PhpParser\BuilderAbstract
     /**
      * Makes the property static.
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function makeStatic() {
         $this->setModifier(Stmt\Class_::MODIFIER_STATIC);
@@ -73,7 +71,7 @@ class Property extends PhpParser\BuilderAbstract
      *
      * @param mixed $value Default value to use
      *
-     * @return self The builder instance (for fluid interface)
+     * @return $this The builder instance (for fluid interface)
      */
     public function setDefault($value) {
         $this->default = $this->normalizeValue($value);
@@ -81,6 +79,21 @@ class Property extends PhpParser\BuilderAbstract
         return $this;
     }
 
+    /**
+     * Sets doc comment for the property.
+     *
+     * @param PhpParser\Comment\Doc|string $docComment Doc comment to set
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function setDocComment($docComment) {
+        $this->attributes = array(
+            'comments' => array($this->normalizeDocComment($docComment))
+        );
+
+        return $this;
+    }
+
     /**
      * Returns the built class node.
      *
@@ -88,10 +101,11 @@ class Property extends PhpParser\BuilderAbstract
      */
     public function getNode() {
         return new Stmt\Property(
-            $this->type !== 0 ? $this->type : Stmt\Class_::MODIFIER_PUBLIC,
+            $this->flags !== 0 ? $this->flags : Stmt\Class_::MODIFIER_PUBLIC,
             array(
                 new Stmt\PropertyProperty($this->name, $this->default)
-            )
+            ),
+            $this->attributes
         );
     }
 }

lib/PhpParser/Builder/Trait_.php

@@ -0,0 +1,56 @@
+<?php
+
+namespace PhpParser\Builder;
+
+use PhpParser;
+use PhpParser\Node\Stmt;
+
+class Trait_ extends Declaration
+{
+    protected $name;
+    protected $properties = array();
+    protected $methods = array();
+
+    /**
+     * Creates an interface builder.
+     *
+     * @param string $name Name of the interface
+     */
+    public function __construct($name) {
+        $this->name = $name;
+    }
+
+    /**
+     * Adds a statement.
+     *
+     * @param Stmt|PhpParser\Builder $stmt The statement to add
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function addStmt($stmt) {
+        $stmt = $this->normalizeNode($stmt);
+
+        if ($stmt instanceof Stmt\Property) {
+            $this->properties[] = $stmt;
+        } else if ($stmt instanceof Stmt\ClassMethod) {
+            $this->methods[] = $stmt;
+        } else {
+            throw new \LogicException(sprintf('Unexpected node of type "%s"', $stmt->getType()));
+        }
+
+        return $this;
+    }
+
+    /**
+     * Returns the built trait node.
+     *
+     * @return Stmt\Trait_ The built interface node
+     */
+    public function getNode() {
+        return new Stmt\Trait_(
+            $this->name, array(
+                'stmts' => array_merge($this->properties, $this->methods)
+            ), $this->attributes
+        );
+    }
+}

lib/PhpParser/Builder/Use_.php

@@ -0,0 +1,58 @@
+<?php
+
+namespace PhpParser\Builder;
+
+use PhpParser\BuilderAbstract;
+use PhpParser\Node;
+use PhpParser\Node\Stmt;
+
+/**
+ * @method $this as(string $alias) Sets alias for used name.
+ */
+class Use_ extends BuilderAbstract {
+    protected $name;
+    protected $type;
+    protected $alias = null;
+
+    /**
+     * Creates a name use (alias) builder.
+     *
+     * @param Node\Name|string $name Name of the entity (namespace, class, function, constant) to alias
+     * @param int              $type One of the Stmt\Use_::TYPE_* constants
+     */
+    public function __construct($name, $type) {
+        $this->name = $this->normalizeName($name);
+        $this->type = $type;
+    }
+
+    /**
+     * Sets alias for used name.
+     *
+     * @param string $alias Alias to use (last component of full name by default)
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    protected function as_($alias) {
+        $this->alias = $alias;
+        return $this;
+    }
+    public function __call($name, $args) {
+        if (method_exists($this, $name . '_')) {
+            return call_user_func_array(array($this, $name . '_'), $args);
+        }
+
+        throw new \LogicException(sprintf('Method "%s" does not exist', $name));
+    }
+
+    /**
+     * Returns the built node.
+     *
+     * @return Node The built node
+     */
+    public function getNode() {
+        $alias = null !== $this->alias ? $this->alias : $this->name->getLast();
+        return new Stmt\Use_(array(
+            new Stmt\UseUse($this->name, $alias)
+        ), $this->type);
+    }
+}

lib/PhpParser/BuilderAbstract.php

@@ -2,10 +2,12 @@
 
 namespace PhpParser;
 
-use PhpParser\Node\Name;
+use PhpParser\Comment;
 use PhpParser\Node\Expr;
-use PhpParser\Node\Stmt;
+use PhpParser\Node\Name;
+use PhpParser\Node\NullableType;
 use PhpParser\Node\Scalar;
+use PhpParser\Node\Stmt;
 
 abstract class BuilderAbstract implements Builder {
     /**
@@ -35,9 +37,64 @@ abstract class BuilderAbstract implements Builder {
     protected function normalizeName($name) {
         if ($name instanceof Name) {
             return $name;
-        } else {
-            return new Name($name);
+        } elseif (is_string($name)) {
+            if (!$name) {
+                throw new \LogicException('Name cannot be empty');
+            }
+
+            if ($name[0] == '\\') {
+                return new Name\FullyQualified(substr($name, 1));
+            } elseif (0 === strpos($name, 'namespace\\')) {
+                return new Name\Relative(substr($name, strlen('namespace\\')));
+            } else {
+                return new Name($name);
+            }
         }
+
+        throw new \LogicException('Name must be a string or an instance of PhpParser\Node\Name');
+    }
+
+    /**
+     * Normalizes a type: Converts plain-text type names into proper AST representation.
+     *
+     * In particular, builtin types are left as strings, custom types become Names and nullables
+     * are wrapped in NullableType nodes.
+     *
+     * @param Name|string|NullableType $type The type to normalize
+     *
+     * @return Name|string|NullableType The normalized type
+     */
+    protected function normalizeType($type) {
+        if (!is_string($type)) {
+            if (!$type instanceof Name && !$type instanceof NullableType) {
+                throw new \LogicException(
+                    'Type must be a string, or an instance of Name or NullableType');
+            }
+            return $type;
+        }
+
+        $nullable = false;
+        if (strlen($type) > 0 && $type[0] === '?') {
+            $nullable = true;
+            $type = substr($type, 1);
+        }
+
+        $builtinTypes = array(
+            'array', 'callable', 'string', 'int', 'float', 'bool', 'iterable', 'void', 'object'
+        );
+
+        $lowerType = strtolower($type);
+        if (in_array($lowerType, $builtinTypes)) {
+            $type = $lowerType;
+        } else {
+            $type = $this->normalizeName($type);
+        }
+
+        if ($nullable && $type === 'void') {
+            throw new \LogicException('void type cannot be nullable');
+        }
+
+        return $nullable ? new Node\NullableType($type) : $type;
     }
 
     /**
@@ -64,7 +121,7 @@ abstract class BuilderAbstract implements Builder {
         } elseif (is_float($value)) {
             return new Scalar\DNumber($value);
         } elseif (is_string($value)) {
-            return new Scalar\String($value);
+            return new Scalar\String_($value);
         } elseif (is_array($value)) {
             $items = array();
             $lastKey = -1;
@@ -89,13 +146,30 @@ abstract class BuilderAbstract implements Builder {
         }
     }
 
+    /**
+     * Normalizes a doc comment: Converts plain strings to PhpParser\Comment\Doc.
+     *
+     * @param Comment\Doc|string $docComment The doc comment to normalize
+     *
+     * @return Comment\Doc The normalized doc comment
+     */
+    protected function normalizeDocComment($docComment) {
+        if ($docComment instanceof Comment\Doc) {
+            return $docComment;
+        } else if (is_string($docComment)) {
+            return new Comment\Doc($docComment);
+        } else {
+            throw new \LogicException('Doc comment must be a string or an instance of PhpParser\Comment\Doc');
+        }
+    }
+
     /**
      * Sets a modifier in the $this->type property.
      *
      * @param int $modifier Modifier to set
      */
     protected function setModifier($modifier) {
-        Stmt\Class_::verifyModifier($this->type, $modifier);
-        $this->type |= $modifier;
+        Stmt\Class_::verifyModifier($this->flags, $modifier);
+        $this->flags |= $modifier;
     }
-}
+}

lib/PhpParser/BuilderFactory.php

@@ -3,18 +3,32 @@
 namespace PhpParser;
 
 use PhpParser\Builder;
+use PhpParser\Node\Stmt\Use_;
 
 /**
- * "class", "interface" and "function" are reserved keywords, so the methods are defined as _class(),
- * _interface() and _function() in the class and are made available as class(), interface() and function()
- * through __call() magic.
+ * The following methods use reserved keywords, so their implementation is defined with an underscore and made available
+ * with the reserved name through __call() magic.
  *
+ * @method Builder\Namespace_ namespace(string $name) Creates a namespace builder.
  * @method Builder\Class_     class(string $name)     Creates a class builder.
- * @method Builder\Function_  function(string $name)  Creates a function builder
  * @method Builder\Interface_ interface(string $name) Creates an interface builder.
+ * @method Builder\Trait_     trait(string $name)     Creates a trait builder.
+ * @method Builder\Function_  function(string $name)  Creates a function builder.
+ * @method Builder\Use_       use(string $name)       Creates a namespace/class use builder.
  */
 class BuilderFactory
 {
+    /**
+     * Creates a namespace builder.
+     *
+     * @param null|string|Node\Name $name Name of the namespace
+     *
+     * @return Builder\Namespace_ The created namespace builder
+     */
+    protected function _namespace($name) {
+        return new Builder\Namespace_($name);
+    }
+
     /**
      * Creates a class builder.
      *
@@ -27,7 +41,7 @@ class BuilderFactory
     }
 
     /**
-     * Creates a interface builder.
+     * Creates an interface builder.
      *
      * @param string $name Name of the interface
      *
@@ -37,6 +51,17 @@ class BuilderFactory
         return new Builder\Interface_($name);
     }
 
+    /**
+     * Creates a trait builder.
+     *
+     * @param string $name Name of the trait
+     *
+     * @return Builder\Trait_ The created trait builder
+     */
+    protected function _trait($name) {
+        return new Builder\Trait_($name);
+    }
+
     /**
      * Creates a method builder.
      *
@@ -81,6 +106,17 @@ class BuilderFactory
         return new Builder\Function_($name);
     }
 
+    /**
+     * Creates a namespace/class use builder.
+     *
+     * @param string|Node\Name Name to alias
+     *
+     * @return Builder\Use_ The create use builder
+     */
+    protected function _use($name) {
+        return new Builder\Use_($name, Use_::TYPE_NORMAL);
+    }
+
     public function __call($name, array $args) {
         if (method_exists($this, '_' . $name)) {
             return call_user_func_array(array($this, '_' . $name), $args);
@@ -88,4 +124,4 @@ class BuilderFactory
 
         throw new \LogicException(sprintf('Method "%s" does not exist', $name));
     }
-}
+}

lib/PhpParser/Comment.php

@@ -2,20 +2,23 @@
 
 namespace PhpParser;
 
-class Comment
+class Comment implements \JsonSerializable
 {
     protected $text;
     protected $line;
+    protected $filePos;
 
     /**
      * Constructs a comment node.
      *
-     * @param string $text Comment text (including comment delimiters like /*)
-     * @param int    $line Line number the comment started on
+     * @param string $text         Comment text (including comment delimiters like /*)
+     * @param int    $startLine    Line number the comment started on
+     * @param int    $startFilePos File offset the comment started on
      */
-    public function __construct($text, $line = -1) {
+    public function __construct($text, $startLine = -1, $startFilePos = -1) {
         $this->text = $text;
-        $this->line = $line;
+        $this->line = $startLine;
+        $this->filePos = $startFilePos;
     }
 
     /**
@@ -27,15 +30,6 @@ class Comment
         return $this->text;
     }
 
-    /**
-     * Sets the comment text.
-     *
-     * @param string $text The comment text (including comment delimiters like /*)
-     */
-    public function setText($text) {
-        $this->text = $text;
-    }
-
     /**
      * Gets the line number the comment started on.
      *
@@ -46,12 +40,12 @@ class Comment
     }
 
     /**
-     * Sets the line number the comment started on.
+     * Gets the file offset the comment started on.
      *
-     * @param int $line Line number
+     * @return int File offset
      */
-    public function setLine($line) {
-        $this->line = $line;
+    public function getFilePos() {
+        return $this->filePos;
     }
 
     /**
@@ -75,7 +69,8 @@ class Comment
      */
     public function getReformattedText() {
         $text = trim($this->text);
-        if (false === strpos($text, "\n")) {
+        $newlinePos = strpos($text, "\n");
+        if (false === $newlinePos) {
             // Single line comments don't need further processing
             return $text;
         } elseif (preg_match('((*BSR_ANYCRLF)(*ANYCRLF)^.*(?:\R\s+\*.*)+$)', $text)) {
@@ -105,15 +100,41 @@ class Comment
             //
             //     /* Some text.
             //        Some more text.
+            //          Indented text.
             //        Even more text. */
             //
-            // is handled by taking the length of the "/* " segment and leaving only that
-            // many space characters before the lines. Thus in the above example only three
-            // space characters are left at the start of every line.
-            return preg_replace('(^\s*(?= {' . strlen($matches[0]) . '}(?!\s)))m', '', $text);
+            // is handled by removing the difference between the shortest whitespace prefix on all
+            // lines and the length of the "/* " opening sequence.
+            $prefixLen = $this->getShortestWhitespacePrefixLen(substr($text, $newlinePos + 1));
+            $removeLen = $prefixLen - strlen($matches[0]);
+            return preg_replace('(^\s{' . $removeLen . '})m', '', $text);
         }
 
         // No idea how to format this comment, so simply return as is
         return $text;
     }
+
+    private function getShortestWhitespacePrefixLen($str) {
+        $lines = explode("\n", $str);
+        $shortestPrefixLen = INF;
+        foreach ($lines as $line) {
+            preg_match('(^\s*)', $line, $matches);
+            $prefixLen = strlen($matches[0]);
+            if ($prefixLen < $shortestPrefixLen) {
+                $shortestPrefixLen = $prefixLen;
+            }
+        }
+        return $shortestPrefixLen;
+    }
+
+    public function jsonSerialize() {
+        // Technically not a node, but we make it look like one anyway
+        $type = $this instanceof Comment\Doc ? 'Comment_Doc' : 'Comment';
+        return [
+            'nodeType' => $type,
+            'text' => $this->text,
+            'line' => $this->line,
+            'filePos' => $this->filePos,
+        ];
+    }
 }

lib/PhpParser/Error.php

@@ -5,17 +5,22 @@ namespace PhpParser;
 class Error extends \RuntimeException
 {
     protected $rawMessage;
-    protected $rawLine;
+    protected $attributes;
 
     /**
      * Creates an Exception signifying a parse error.
      *
-     * @param string $message Error message
-     * @param int    $line    Error line in PHP file
+     * @param string    $message    Error message
+     * @param array|int $attributes Attributes of node/token where error occurred
+     *                              (or start line of error -- deprecated)
      */
-    public function __construct($message, $line = -1) {
+    public function __construct($message, $attributes = array()) {
         $this->rawMessage = (string) $message;
-        $this->rawLine    = (int) $line;
+        if (is_array($attributes)) {
+            $this->attributes = $attributes;
+        } else {
+            $this->attributes = array('startLine' => $attributes);
+        }
         $this->updateMessage();
     }
 
@@ -28,6 +33,44 @@ class Error extends \RuntimeException
         return $this->rawMessage;
     }
 
+    /**
+     * Gets the line the error starts in.
+     *
+     * @return int Error start line
+     */
+    public function getStartLine() {
+        return isset($this->attributes['startLine']) ? $this->attributes['startLine'] : -1;
+    }
+
+    /**
+     * Gets the line the error ends in.
+     *
+     * @return int Error end line
+     */
+    public function getEndLine() {
+        return isset($this->attributes['endLine']) ? $this->attributes['endLine'] : -1;
+    }
+
+
+    /**
+     * Gets the attributes of the node/token the error occurred at.
+     *
+     * @return array
+     */
+    public function getAttributes() {
+        return $this->attributes;
+    }
+
+    /**
+     * Sets the attributes of the node/token the error occured at.
+     *
+     * @param array $attributes
+     */
+    public function setAttributes(array $attributes) {
+        $this->attributes = $attributes;
+        $this->updateMessage();
+    }
+
     /**
      * Sets the line of the PHP file the error occurred in.
      *
@@ -39,22 +82,73 @@ class Error extends \RuntimeException
     }
 
     /**
-     * Gets the error line in the PHP file.
+     * Sets the line the error starts in.
      *
-     * @return int Error line in the PHP file
+     * @param int $line Error start line
      */
-    public function getRawLine() {
-        return $this->rawLine;
+    public function setStartLine($line) {
+        $this->attributes['startLine'] = (int) $line;
+        $this->updateMessage();
     }
 
     /**
-     * Sets the line of the PHP file the error occurred in.
+     * Returns whether the error has start and end column information.
      *
-     * @param int $line Error line in the PHP file
+     * For column information enable the startFilePos and endFilePos in the lexer options.
+     *
+     * @return bool
      */
-    public function setRawLine($line) {
-        $this->rawLine = (int) $line;
-        $this->updateMessage();
+    public function hasColumnInfo() {
+        return isset($this->attributes['startFilePos']) && isset($this->attributes['endFilePos']);
+    }
+
+    /**
+     * Gets the start column (1-based) into the line where the error started.
+     *
+     * @param string $code Source code of the file
+     * @return int
+     */
+    public function getStartColumn($code) {
+        if (!$this->hasColumnInfo()) {
+            throw new \RuntimeException('Error does not have column information');
+        }
+
+        return $this->toColumn($code, $this->attributes['startFilePos']);
+    }
+
+    /**
+     * Gets the end column (1-based) into the line where the error ended.
+     *
+     * @param string $code Source code of the file
+     * @return int
+     */
+    public function getEndColumn($code) {
+        if (!$this->hasColumnInfo()) {
+            throw new \RuntimeException('Error does not have column information');
+        }
+
+        return $this->toColumn($code, $this->attributes['endFilePos']);
+    }
+
+    public function getMessageWithColumnInfo($code) {
+        return sprintf(
+            '%s from %d:%d to %d:%d', $this->getRawMessage(),
+            $this->getStartLine(), $this->getStartColumn($code),
+            $this->getEndLine(), $this->getEndColumn($code)
+        );
+    }
+
+    private function toColumn($code, $pos) {
+        if ($pos > strlen($code)) {
+            throw new \RuntimeException('Invalid position information');
+        }
+
+        $lineStartPos = strrpos($code, "\n", $pos - strlen($code));
+        if (false === $lineStartPos) {
+            $lineStartPos = -1;
+        }
+
+        return $pos - $lineStartPos;
     }
 
     /**
@@ -63,10 +157,10 @@ class Error extends \RuntimeException
     protected function updateMessage() {
         $this->message = $this->rawMessage;
 
-        if (-1 === $this->rawLine) {
+        if (-1 === $this->getStartLine()) {
             $this->message .= ' on unknown line';
         } else {
-            $this->message .= ' on line ' . $this->rawLine;
+            $this->message .= ' on line ' . $this->getStartLine();
         }
     }
-}
+}

lib/PhpParser/ErrorHandler.php

@@ -0,0 +1,13 @@
+<?php
+
+namespace PhpParser;
+
+interface ErrorHandler
+{
+    /**
+     * Handle an error generated during lexing, parsing or some other operation.
+     *
+     * @param Error $error The error that needs to be handled
+     */
+    public function handleError(Error $error);
+}

lib/PhpParser/ErrorHandler/Collecting.php

@@ -0,0 +1,46 @@
+<?php
+
+namespace PhpParser\ErrorHandler;
+
+use PhpParser\Error;
+use PhpParser\ErrorHandler;
+
+/**
+ * Error handler that collects all errors into an array.
+ *
+ * This allows graceful handling of errors.
+ */
+class Collecting implements ErrorHandler
+{
+    /** @var Error[] Collected errors */
+    private $errors = [];
+
+    public function handleError(Error $error) {
+        $this->errors[] = $error;
+    }
+
+    /**
+     * Get collected errors.
+     *
+     * @return Error[]
+     */
+    public function getErrors() {
+        return $this->errors;
+    }
+
+    /**
+     * Check whether there are any errors.
+     *
+     * @return bool
+     */
+    public function hasErrors() {
+        return !empty($this->errors);
+    }
+
+    /**
+     * Reset/clear collected errors.
+     */
+    public function clearErrors() {
+        $this->errors = [];
+    }
+}

lib/PhpParser/ErrorHandler/Throwing.php

@@ -0,0 +1,18 @@
+<?php
+
+namespace PhpParser\ErrorHandler;
+
+use PhpParser\Error;
+use PhpParser\ErrorHandler;
+
+/**
+ * Error handler that handles all errors by throwing them.
+ *
+ * This is the default strategy used by all components.
+ */
+class Throwing implements ErrorHandler
+{
+    public function handleError(Error $error) {
+        throw $error;
+    }
+}

lib/PhpParser/Lexer.php

@@ -2,85 +2,219 @@
 
 namespace PhpParser;
 
+use PhpParser\Parser\Tokens;
+
 class Lexer
 {
     protected $code;
     protected $tokens;
     protected $pos;
     protected $line;
+    protected $filePos;
+    protected $prevCloseTagHasNewline;
 
     protected $tokenMap;
     protected $dropTokens;
 
+    protected $usedAttributes;
+
     /**
      * Creates a Lexer.
+     *
+     * @param array $options Options array. Currently only the 'usedAttributes' option is supported,
+     *                       which is an array of attributes to add to the AST nodes. Possible
+     *                       attributes are: 'comments', 'startLine', 'endLine', 'startTokenPos',
+     *                       'endTokenPos', 'startFilePos', 'endFilePos'. The option defaults to the
+     *                       first three. For more info see getNextToken() docs.
      */
-    public function __construct() {
+    public function __construct(array $options = array()) {
         // map from internal tokens to PhpParser tokens
         $this->tokenMap = $this->createTokenMap();
 
         // map of tokens to drop while lexing (the map is only used for isset lookup,
         // that's why the value is simply set to 1; the value is never actually used.)
-        $this->dropTokens = array_fill_keys(array(T_WHITESPACE, T_OPEN_TAG), 1);
+        $this->dropTokens = array_fill_keys(
+            array(T_WHITESPACE, T_OPEN_TAG, T_COMMENT, T_DOC_COMMENT), 1
+        );
+
+        // the usedAttributes member is a map of the used attribute names to a dummy
+        // value (here "true")
+        $options += array(
+            'usedAttributes' => array('comments', 'startLine', 'endLine'),
+        );
+        $this->usedAttributes = array_fill_keys($options['usedAttributes'], true);
     }
 
     /**
      * Initializes the lexer for lexing the provided source code.
      *
-     * @param string $code The source code to lex
+     * This function does not throw if lexing errors occur. Instead, errors may be retrieved using
+     * the getErrors() method.
      *
-     * @throws Error on lexing errors (unterminated comment or unexpected character)
+     * @param string $code The source code to lex
+     * @param ErrorHandler|null $errorHandler Error handler to use for lexing errors. Defaults to
+     *                                        ErrorHandler\Throwing
      */
-    public function startLexing($code) {
-        $scream = ini_set('xdebug.scream', 0);
-
-        $this->resetErrors();
-        $this->tokens = @token_get_all($code);
-        $this->handleErrors();
-
-        ini_set('xdebug.scream', $scream);
+    public function startLexing($code, ErrorHandler $errorHandler = null) {
+        if (null === $errorHandler) {
+            $errorHandler = new ErrorHandler\Throwing();
+        }
 
         $this->code = $code; // keep the code around for __halt_compiler() handling
         $this->pos  = -1;
         $this->line =  1;
+        $this->filePos = 0;
+
+        // If inline HTML occurs without preceding code, treat it as if it had a leading newline.
+        // This ensures proper composability, because having a newline is the "safe" assumption.
+        $this->prevCloseTagHasNewline = true;
+
+        $scream = ini_set('xdebug.scream', '0');
+
+        $this->resetErrors();
+        $this->tokens = @token_get_all($code);
+        $this->handleErrors($errorHandler);
+
+        if (false !== $scream) {
+            ini_set('xdebug.scream', $scream);
+        }
     }
 
     protected function resetErrors() {
-        // set error_get_last() to defined state by forcing an undefined variable error
-        set_error_handler(function() { return false; }, 0);
-        @$undefinedVariable;
-        restore_error_handler();
+        if (function_exists('error_clear_last')) {
+            error_clear_last();
+        } else {
+            // set error_get_last() to defined state by forcing an undefined variable error
+            set_error_handler(function() { return false; }, 0);
+            @$undefinedVariable;
+            restore_error_handler();
+        }
     }
 
-    protected function handleErrors() {
+    private function handleInvalidCharacterRange($start, $end, $line, ErrorHandler $errorHandler) {
+        for ($i = $start; $i < $end; $i++) {
+            $chr = $this->code[$i];
+            if ($chr === 'b' || $chr === 'B') {
+                // HHVM does not treat b" tokens correctly, so ignore these
+                continue;
+            }
+
+            if ($chr === "\0") {
+                // PHP cuts error message after null byte, so need special case
+                $errorMsg = 'Unexpected null byte';
+            } else {
+                $errorMsg = sprintf(
+                    'Unexpected character "%s" (ASCII %d)', $chr, ord($chr)
+                );
+            }
+
+            $errorHandler->handleError(new Error($errorMsg, [
+                'startLine' => $line,
+                'endLine' => $line,
+                'startFilePos' => $i,
+                'endFilePos' => $i,
+            ]));
+        }
+    }
+
+    private function isUnterminatedComment($token) {
+        return ($token[0] === T_COMMENT || $token[0] === T_DOC_COMMENT)
+            && substr($token[1], 0, 2) === '/*'
+            && substr($token[1], -2) !== '*/';
+    }
+
+    private function errorMayHaveOccurred() {
+        if (defined('HHVM_VERSION')) {
+            // In HHVM token_get_all() does not throw warnings, so we need to conservatively
+            // assume that an error occurred
+            return true;
+        }
+
         $error = error_get_last();
+        return null !== $error
+            && false === strpos($error['message'], 'Undefined variable');
+    }
 
-        if (preg_match(
-            '~^Unterminated comment starting line ([0-9]+)$~',
-            $error['message'], $matches
-        )) {
-            throw new Error('Unterminated comment', $matches[1]);
+    protected function handleErrors(ErrorHandler $errorHandler) {
+        if (!$this->errorMayHaveOccurred()) {
+            return;
         }
 
-        if (preg_match(
-            '~^Unexpected character in input:  \'(.)\' \(ASCII=([0-9]+)\)~s',
-            $error['message'], $matches
-        )) {
-            throw new Error(sprintf(
-                'Unexpected character "%s" (ASCII %d)',
-                $matches[1], $matches[2]
-            ));
+        // PHP's error handling for token_get_all() is rather bad, so if we want detailed
+        // error information we need to compute it ourselves. Invalid character errors are
+        // detected by finding "gaps" in the token array. Unterminated comments are detected
+        // by checking if a trailing comment has a "*/" at the end.
+
+        $filePos = 0;
+        $line = 1;
+        foreach ($this->tokens as $i => $token) {
+            $tokenValue = \is_string($token) ? $token : $token[1];
+            $tokenLen = \strlen($tokenValue);
+
+            if (substr($this->code, $filePos, $tokenLen) !== $tokenValue) {
+                // Something is missing, must be an invalid character
+                $nextFilePos = strpos($this->code, $tokenValue, $filePos);
+                $this->handleInvalidCharacterRange(
+                    $filePos, $nextFilePos, $line, $errorHandler);
+                $filePos = $nextFilePos;
+            }
+
+            $filePos += $tokenLen;
+            $line += substr_count($tokenValue, "\n");
         }
 
-        // PHP cuts error message after null byte, so need special case
-        if (preg_match('~^Unexpected character in input:  \'$~', $error['message'])) {
-            throw new Error('Unexpected null byte');
+        if ($filePos !== \strlen($this->code)) {
+            if (substr($this->code, $filePos, 2) === '/*') {
+                // Unlike PHP, HHVM will drop unterminated comments entirely
+                $comment = substr($this->code, $filePos);
+                $errorHandler->handleError(new Error('Unterminated comment', [
+                    'startLine' => $line,
+                    'endLine' => $line + substr_count($comment, "\n"),
+                    'startFilePos' => $filePos,
+                    'endFilePos' => $filePos + \strlen($comment),
+                ]));
+
+                // Emulate the PHP behavior
+                $isDocComment = isset($comment[3]) && $comment[3] === '*';
+                $this->tokens[] = [$isDocComment ? T_DOC_COMMENT : T_COMMENT, $comment, $line];
+            } else {
+                // Invalid characters at the end of the input
+                $this->handleInvalidCharacterRange(
+                    $filePos, \strlen($this->code), $line, $errorHandler);
+            }
+            return;
+        }
+
+        if (count($this->tokens) > 0) {
+            // Check for unterminated comment
+            $lastToken = $this->tokens[count($this->tokens) - 1];
+            if ($this->isUnterminatedComment($lastToken)) {
+                $errorHandler->handleError(new Error('Unterminated comment', [
+                    'startLine' => $line - substr_count($lastToken[1], "\n"),
+                    'endLine' => $line,
+                    'startFilePos' => $filePos - \strlen($lastToken[1]),
+                    'endFilePos' => $filePos,
+                ]));
+            }
         }
     }
 
     /**
      * Fetches the next token.
      *
+     * The available attributes are determined by the 'usedAttributes' option, which can
+     * be specified in the constructor. The following attributes are supported:
+     *
+     *  * 'comments'      => Array of PhpParser\Comment or PhpParser\Comment\Doc instances,
+     *                       representing all comments that occurred between the previous
+     *                       non-discarded token and the current one.
+     *  * 'startLine'     => Line in which the node starts.
+     *  * 'endLine'       => Line in which the node ends.
+     *  * 'startTokenPos' => Offset into the token array of the first token in the node.
+     *  * 'endTokenPos'   => Offset into the token array of the last token in the node.
+     *  * 'startFilePos'  => Offset into the code string of the first character that is part of the node.
+     *  * 'endFilePos'    => Offset into the code string of the last character that is part of the node.
+     *
      * @param mixed $value           Variable to store token content in
      * @param mixed $startAttributes Variable to store start attributes in
      * @param mixed $endAttributes   Variable to store end attributes in
@@ -91,42 +225,88 @@ class Lexer
         $startAttributes = array();
         $endAttributes   = array();
 
-        while (isset($this->tokens[++$this->pos])) {
-            $token = $this->tokens[$this->pos];
-
-            if (is_string($token)) {
-                $startAttributes['startLine'] = $this->line;
-                $endAttributes['endLine']     = $this->line;
-
-                // bug in token_get_all
-                if ('b"' === $token) {
-                    $value = 'b"';
-                    return ord('"');
-                } else {
-                    $value = $token;
-                    return ord($token);
-                }
+        while (1) {
+            if (isset($this->tokens[++$this->pos])) {
+                $token = $this->tokens[$this->pos];
             } else {
-                $this->line += substr_count($token[1], "\n");
-
-                if (T_COMMENT === $token[0]) {
-                    $startAttributes['comments'][] = new Comment($token[1], $token[2]);
-                } elseif (T_DOC_COMMENT === $token[0]) {
-                    $startAttributes['comments'][] = new Comment\Doc($token[1], $token[2]);
-                } elseif (!isset($this->dropTokens[$token[0]])) {
-                    $value = $token[1];
-                    $startAttributes['startLine'] = $token[2];
-                    $endAttributes['endLine']     = $this->line;
-
-                    return $this->tokenMap[$token[0]];
-                }
+                // EOF token with ID 0
+                $token = "\0";
             }
+
+            if (isset($this->usedAttributes['startLine'])) {
+                $startAttributes['startLine'] = $this->line;
+            }
+            if (isset($this->usedAttributes['startTokenPos'])) {
+                $startAttributes['startTokenPos'] = $this->pos;
+            }
+            if (isset($this->usedAttributes['startFilePos'])) {
+                $startAttributes['startFilePos'] = $this->filePos;
+            }
+
+            if (\is_string($token)) {
+                $value = $token;
+                if (isset($token[1])) {
+                    // bug in token_get_all
+                    $this->filePos += 2;
+                    $id = ord('"');
+                } else {
+                    $this->filePos += 1;
+                    $id = ord($token);
+                }
+            } elseif (!isset($this->dropTokens[$token[0]])) {
+                $value = $token[1];
+                $id = $this->tokenMap[$token[0]];
+                if (T_CLOSE_TAG === $token[0]) {
+                    $this->prevCloseTagHasNewline = false !== strpos($token[1], "\n");
+                } else if (T_INLINE_HTML === $token[0]) {
+                    $startAttributes['hasLeadingNewline'] = $this->prevCloseTagHasNewline;
+                }
+
+                $this->line += substr_count($value, "\n");
+                $this->filePos += \strlen($value);
+            } else {
+                if (T_COMMENT === $token[0] || T_DOC_COMMENT === $token[0]) {
+                    if (isset($this->usedAttributes['comments'])) {
+                        $comment = T_DOC_COMMENT === $token[0]
+                            ? new Comment\Doc($token[1], $this->line, $this->filePos)
+                            : new Comment($token[1], $this->line, $this->filePos);
+                        $startAttributes['comments'][] = $comment;
+                    }
+                }
+
+                $this->line += substr_count($token[1], "\n");
+                $this->filePos += \strlen($token[1]);
+                continue;
+            }
+
+            if (isset($this->usedAttributes['endLine'])) {
+                $endAttributes['endLine'] = $this->line;
+            }
+            if (isset($this->usedAttributes['endTokenPos'])) {
+                $endAttributes['endTokenPos'] = $this->pos;
+            }
+            if (isset($this->usedAttributes['endFilePos'])) {
+                $endAttributes['endFilePos'] = $this->filePos - 1;
+            }
+
+            return $id;
         }
 
-        $startAttributes['startLine'] = $this->line;
+        throw new \RuntimeException('Reached end of lexer loop');
+    }
 
-        // 0 is the EOF token
-        return 0;
+    /**
+     * Returns the token array for current code.
+     *
+     * The token array is in the same format as provided by the
+     * token_get_all() function and does not discard tokens (i.e.
+     * whitespace and comments are included). The token position
+     * attributes are against this token array.
+     *
+     * @return array Array of tokens in token_get_all() format
+     */
+    public function getTokens() {
+        return $this->tokens;
     }
 
     /**
@@ -135,23 +315,13 @@ class Lexer
      * @return string Remaining text
      */
     public function handleHaltCompiler() {
-        // get the length of the text before the T_HALT_COMPILER token
-        $textBefore = '';
-        for ($i = 0; $i <= $this->pos; ++$i) {
-            if (is_string($this->tokens[$i])) {
-                $textBefore .= $this->tokens[$i];
-            } else {
-                $textBefore .= $this->tokens[$i][1];
-            }
-        }
-
         // text after T_HALT_COMPILER, still including ();
-        $textAfter = substr($this->code, strlen($textBefore));
+        $textAfter = substr($this->code, $this->filePos);
 
         // ensure that it is followed by ();
         // this simplifies the situation, by not allowing any comments
         // in between of the tokens.
-        if (!preg_match('~\s*\(\s*\)\s*(?:;|\?>\r?\n?)~', $textAfter, $matches)) {
+        if (!preg_match('~^\s*\(\s*\)\s*(?:;|\?>\r?\n?)~', $textAfter, $matches)) {
             throw new Error('__HALT_COMPILER must be followed by "();"');
         }
 
@@ -177,26 +347,33 @@ class Lexer
         // 256 is the minimum possible token number, as everything below
         // it is an ASCII value
         for ($i = 256; $i < 1000; ++$i) {
-            // T_DOUBLE_COLON is equivalent to T_PAAMAYIM_NEKUDOTAYIM
             if (T_DOUBLE_COLON === $i) {
-                $tokenMap[$i] = Parser::T_PAAMAYIM_NEKUDOTAYIM;
-            // T_OPEN_TAG_WITH_ECHO with dropped T_OPEN_TAG results in T_ECHO
+                // T_DOUBLE_COLON is equivalent to T_PAAMAYIM_NEKUDOTAYIM
+                $tokenMap[$i] = Tokens::T_PAAMAYIM_NEKUDOTAYIM;
             } elseif(T_OPEN_TAG_WITH_ECHO === $i) {
-                $tokenMap[$i] = Parser::T_ECHO;
-            // T_CLOSE_TAG is equivalent to ';'
+                // T_OPEN_TAG_WITH_ECHO with dropped T_OPEN_TAG results in T_ECHO
+                $tokenMap[$i] = Tokens::T_ECHO;
             } elseif(T_CLOSE_TAG === $i) {
+                // T_CLOSE_TAG is equivalent to ';'
                 $tokenMap[$i] = ord(';');
-            // and the others can be mapped directly
-            } elseif ('UNKNOWN' !== ($name = token_name($i))
-                      && defined($name = 'PhpParser\Parser::' . $name)
-            ) {
-                $tokenMap[$i] = constant($name);
+            } elseif ('UNKNOWN' !== $name = token_name($i)) {
+                if ('T_HASHBANG' === $name) {
+                    // HHVM uses a special token for #! hashbang lines
+                    $tokenMap[$i] = Tokens::T_INLINE_HTML;
+                } else if (defined($name = 'PhpParser\Parser\Tokens::' . $name)) {
+                    // Other tokens can be mapped directly
+                    $tokenMap[$i] = constant($name);
+                }
             }
         }
 
         // HHVM uses a special token for numbers that overflow to double
         if (defined('T_ONUMBER')) {
-            $tokenMap[T_ONUMBER] = Parser::T_DNUMBER;
+            $tokenMap[T_ONUMBER] = Tokens::T_DNUMBER;
+        }
+        // HHVM also has a separate token for the __COMPILER_HALT_OFFSET__ constant
+        if (defined('T_COMPILER_HALT_OFFSET')) {
+            $tokenMap[T_COMPILER_HALT_OFFSET] = Tokens::T_STRING;
         }
 
         return $tokenMap;

lib/PhpParser/Lexer/Emulative.php

@@ -2,38 +2,29 @@
 
 namespace PhpParser\Lexer;
 
-use PhpParser\Parser;
+use PhpParser\ErrorHandler;
+use PhpParser\Parser\Tokens;
 
-/**
- * ATTENTION: This code is WRITE-ONLY. Do not try to read it.
- */
 class Emulative extends \PhpParser\Lexer
 {
     protected $newKeywords;
     protected $inObjectAccess;
 
-    const T_ELLIPSIS  = 1001;
-    const T_POW       = 1002;
-    const T_POW_EQUAL = 1003;
+    const T_ELLIPSIS   = 1001;
+    const T_POW        = 1002;
+    const T_POW_EQUAL  = 1003;
+    const T_COALESCE   = 1004;
+    const T_SPACESHIP  = 1005;
+    const T_YIELD_FROM = 1006;
 
+    const PHP_7_0 = '7.0.0dev';
     const PHP_5_6 = '5.6.0rc1';
-    const PHP_5_5 = '5.5.0beta1';
-    const PHP_5_4 = '5.4.0beta1';
 
-    public function __construct() {
-        parent::__construct();
+    public function __construct(array $options = array()) {
+        parent::__construct($options);
 
         $newKeywordsPerVersion = array(
-            self::PHP_5_5 => array(
-                'finally'       => Parser::T_FINALLY,
-                'yield'         => Parser::T_YIELD,
-            ),
-            self::PHP_5_4 => array(
-                'callable'      => Parser::T_CALLABLE,
-                'insteadof'     => Parser::T_INSTEADOF,
-                'trait'         => Parser::T_TRAIT,
-                '__trait__'     => Parser::T_TRAIT_C,
-            ),
+            // No new keywords since PHP 5.5
         );
 
         $this->newKeywords = array();
@@ -45,118 +36,121 @@ class Emulative extends \PhpParser\Lexer
             $this->newKeywords += $newKeywords;
         }
 
-        if (version_compare(PHP_VERSION, self::PHP_5_6, '<')) {
-            $this->tokenMap[self::T_ELLIPSIS]  = Parser::T_ELLIPSIS;
-            $this->tokenMap[self::T_POW]       = Parser::T_POW;
-            $this->tokenMap[self::T_POW_EQUAL] = Parser::T_POW_EQUAL;
+        if (version_compare(PHP_VERSION, self::PHP_7_0, '>=')) {
+            return;
         }
+        $this->tokenMap[self::T_COALESCE]   = Tokens::T_COALESCE;
+        $this->tokenMap[self::T_SPACESHIP]  = Tokens::T_SPACESHIP;
+        $this->tokenMap[self::T_YIELD_FROM] = Tokens::T_YIELD_FROM;
+
+        if (version_compare(PHP_VERSION, self::PHP_5_6, '>=')) {
+            return;
+        }
+        $this->tokenMap[self::T_ELLIPSIS]  = Tokens::T_ELLIPSIS;
+        $this->tokenMap[self::T_POW]       = Tokens::T_POW;
+        $this->tokenMap[self::T_POW_EQUAL] = Tokens::T_POW_EQUAL;
     }
 
-    public function startLexing($code) {
+    public function startLexing($code, ErrorHandler $errorHandler = null) {
         $this->inObjectAccess = false;
 
-        $preprocessedCode = $this->preprocessCode($code);
-        parent::startLexing($preprocessedCode);
-        if ($preprocessedCode !== $code) {
-            $this->postprocessTokens();
+        parent::startLexing($code, $errorHandler);
+        if ($this->requiresEmulation($code)) {
+            $this->emulateTokens();
         }
     }
 
     /*
-     * Replaces new features in the code by ~__EMU__{NAME}__{DATA}__~ sequences.
-     * ~LABEL~ is never valid PHP code, that's why we can (to some degree) safely
-     * use it here.
-     * Later when preprocessing the tokens these sequences will either be replaced
-     * by real tokens or replaced with their original content (e.g. if they occurred
-     * inside a string, i.e. a place where they don't have a special meaning).
+     * Checks if the code is potentially using features that require emulation.
      */
-    protected function preprocessCode($code) {
+    protected function requiresEmulation($code) {
+        if (version_compare(PHP_VERSION, self::PHP_7_0, '>=')) {
+            return false;
+        }
+
+        if (preg_match('(\?\?|<=>|yield[ \n\r\t]+from)', $code)) {
+            return true;
+        }
+
         if (version_compare(PHP_VERSION, self::PHP_5_6, '>=')) {
-            return $code;
+            return false;
         }
 
-        $code = str_replace('...', '~__EMU__ELLIPSIS__~', $code);
-        $code = preg_replace('((?<!/)\*\*=)', '~__EMU__POWEQUAL__~', $code);
-        $code = preg_replace('((?<!/)\*\*(?!/))', '~__EMU__POW__~', $code);
-
-        if (version_compare(PHP_VERSION, self::PHP_5_4, '>=')) {
-            return $code;
-        }
-
-        // binary notation (0b010101101001...)
-        return preg_replace('(\b0b[01]+\b)', '~__EMU__BINARY__$0__~', $code);
+        return preg_match('(\.\.\.|(?<!/)\*\*(?!/))', $code);
     }
 
     /*
-     * Replaces the ~__EMU__...~ sequences with real tokens or their original
-     * value.
+     * Emulates tokens for newer PHP versions.
      */
-    protected function postprocessTokens() {
-        // we need to manually iterate and manage a count because we'll change
+    protected function emulateTokens() {
+        // We need to manually iterate and manage a count because we'll change
         // the tokens array on the way
+        $line = 1;
         for ($i = 0, $c = count($this->tokens); $i < $c; ++$i) {
-            // first check that the following tokens are of form ~LABEL~,
-            // then match the __EMU__... sequence.
-            if ('~' === $this->tokens[$i]
-                && isset($this->tokens[$i + 2])
-                && '~' === $this->tokens[$i + 2]
-                && T_STRING === $this->tokens[$i + 1][0]
-                && preg_match('(^__EMU__([A-Z]++)__(?:([A-Za-z0-9]++)__)?$)', $this->tokens[$i + 1][1], $matches)
-            ) {
-                if ('BINARY' === $matches[1]) {
-                    // the binary number can either be an integer or a double, so return a LNUMBER
-                    // or DNUMBER respectively
-                    $replace = array(
-                        array(is_int(bindec($matches[2])) ? T_LNUMBER : T_DNUMBER, $matches[2], $this->tokens[$i + 1][2])
-                    );
-                } else if ('ELLIPSIS' === $matches[1]) {
-                    $replace = array(
-                        array(self::T_ELLIPSIS, '...', $this->tokens[$i + 1][2])
-                    );
-                } else if ('POW' === $matches[1]) {
-                    $replace = array(
-                        array(self::T_POW, '**', $this->tokens[$i + 1][2])
-                    );
-                } else if ('POWEQUAL' === $matches[1]) {
-                    $replace = array(
-                        array(self::T_POW_EQUAL, '**=', $this->tokens[$i + 1][2])
-                    );
-                } else {
-                    // just ignore all other __EMU__ sequences
+            $replace = null;
+            if (isset($this->tokens[$i + 1])) {
+                if ($this->tokens[$i] === '?' && $this->tokens[$i + 1] === '?') {
+                    array_splice($this->tokens, $i, 2, array(
+                        array(self::T_COALESCE, '??', $line)
+                    ));
+                    $c--;
+                    continue;
+                }
+                if ($this->tokens[$i][0] === T_IS_SMALLER_OR_EQUAL
+                    && $this->tokens[$i + 1] === '>'
+                ) {
+                    array_splice($this->tokens, $i, 2, array(
+                        array(self::T_SPACESHIP, '<=>', $line)
+                    ));
+                    $c--;
+                    continue;
+                }
+                if ($this->tokens[$i] === '*' && $this->tokens[$i + 1] === '*') {
+                    array_splice($this->tokens, $i, 2, array(
+                        array(self::T_POW, '**', $line)
+                    ));
+                    $c--;
+                    continue;
+                }
+                if ($this->tokens[$i] === '*' && $this->tokens[$i + 1][0] === T_MUL_EQUAL) {
+                    array_splice($this->tokens, $i, 2, array(
+                        array(self::T_POW_EQUAL, '**=', $line)
+                    ));
+                    $c--;
                     continue;
                 }
-
-                array_splice($this->tokens, $i, 3, $replace);
-                $c -= 3 - count($replace);
-            // for multichar tokens (e.g. strings) replace any ~__EMU__...~ sequences
-            // in their content with the original character sequence
-            } elseif (is_array($this->tokens[$i])
-                      && 0 !== strpos($this->tokens[$i][1], '__EMU__')
-            ) {
-                $this->tokens[$i][1] = preg_replace_callback(
-                    '(~__EMU__([A-Z]++)__(?:([A-Za-z0-9]++)__)?~)',
-                    array($this, 'restoreContentCallback'),
-                    $this->tokens[$i][1]
-                );
             }
-        }
-    }
 
-    /*
-     * This method is a callback for restoring EMU sequences in
-     * multichar tokens (like strings) to their original value.
-     */
-    public function restoreContentCallback(array $matches) {
-        if ('BINARY' === $matches[1]) {
-            return $matches[2];
-        } else if ('ELLIPSIS' === $matches[1]) {
-            return '...';
-        } else if ('POW' === $matches[1]) {
-            return '**';
-        } else if ('POWEQUAL' === $matches[1]) {
-            return '**=';
-        } else {
-            return $matches[0];
+            if (isset($this->tokens[$i + 2])) {
+                if ($this->tokens[$i][0] === T_YIELD && $this->tokens[$i + 1][0] === T_WHITESPACE
+                    && $this->tokens[$i + 2][0] === T_STRING
+                    && !strcasecmp($this->tokens[$i + 2][1], 'from')
+                ) {
+                    array_splice($this->tokens, $i, 3, array(
+                        array(
+                            self::T_YIELD_FROM,
+                            $this->tokens[$i][1] . $this->tokens[$i + 1][1] . $this->tokens[$i + 2][1],
+                            $line
+                        )
+                    ));
+                    $c -= 2;
+                    $line += substr_count($this->tokens[$i][1], "\n");
+                    continue;
+                }
+                if ($this->tokens[$i] === '.' && $this->tokens[$i + 1] === '.'
+                    && $this->tokens[$i + 2] === '.'
+                ) {
+                    array_splice($this->tokens, $i, 3, array(
+                        array(self::T_ELLIPSIS, '...', $line)
+                    ));
+                    $c -= 2;
+                    continue;
+                }
+            }
+
+            if (\is_array($this->tokens[$i])) {
+                $line += substr_count($this->tokens[$i][1], "\n");
+            }
         }
     }
 
@@ -166,17 +160,15 @@ class Emulative extends \PhpParser\Lexer
         // replace new keywords by their respective tokens. This is not done
         // if we currently are in an object access (e.g. in $obj->namespace
         // "namespace" stays a T_STRING tokens and isn't converted to T_NAMESPACE)
-        if (Parser::T_STRING === $token && !$this->inObjectAccess) {
+        if (Tokens::T_STRING === $token && !$this->inObjectAccess) {
             if (isset($this->newKeywords[strtolower($value)])) {
                 return $this->newKeywords[strtolower($value)];
             }
-        // keep track of whether we currently are in an object access (after ->)
-        } elseif (Parser::T_OBJECT_OPERATOR === $token) {
-            $this->inObjectAccess = true;
         } else {
-            $this->inObjectAccess = false;
+            // keep track of whether we currently are in an object access (after ->)
+            $this->inObjectAccess = Tokens::T_OBJECT_OPERATOR === $token;
         }
 
         return $token;
     }
-}
+}

lib/PhpParser/Node.php

@@ -29,6 +29,8 @@ interface Node
      * Sets line the node started in.
      *
      * @param int $line Line
+     *
+     * @deprecated
      */
     public function setLine($line);
 
@@ -41,6 +43,15 @@ interface Node
      */
     public function getDocComment();
 
+    /**
+     * Sets the doc comment of the node.
+     *
+     * This will either replace an existing doc comment or add it to the comments array.
+     *
+     * @param Comment\Doc $docComment Doc comment to set
+     */
+    public function setDocComment(Comment\Doc $docComment);
+
     /**
      * Sets an attribute on a node.
      *

lib/PhpParser/Node/Arg.php

@@ -4,13 +4,15 @@ namespace PhpParser\Node;
 
 use PhpParser\NodeAbstract;
 
-/**
- * @property Expr $value  Value to pass
- * @property bool $byRef  Whether to pass by ref
- * @property bool $unpack Whether to unpack the argument
- */
 class Arg extends NodeAbstract
 {
+    /** @var Expr Value to pass */
+    public $value;
+    /** @var bool Whether to pass by ref */
+    public $byRef;
+    /** @var bool Whether to unpack the argument */
+    public $unpack;
+
     /**
      * Constructs a function call argument node.
      *
@@ -20,13 +22,13 @@ class Arg extends NodeAbstract
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $value, $byRef = false, $unpack = false, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'value'  => $value,
-                'byRef'  => $byRef,
-                'unpack' => $unpack,
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->value = $value;
+        $this->byRef = $byRef;
+        $this->unpack = $unpack;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('value', 'byRef', 'unpack');
+    }
+}

lib/PhpParser/Node/Const_.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node;
 
 use PhpParser\NodeAbstract;
 
-/**
- * @property string              $name  Name
- * @property Expr $value Value
- */
 class Const_ extends NodeAbstract
 {
+    /** @var string Name */
+    public $name;
+    /** @var Expr Value */
+    public $value;
+
     /**
      * Constructs a const node for use in class const and const statements.
      *
@@ -18,12 +19,12 @@ class Const_ extends NodeAbstract
      * @param array   $attributes Additional attributes
      */
     public function __construct($name, Expr $value, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'name'  => $name,
-                'value' => $value,
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->name = $name;
+        $this->value = $value;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('name', 'value');
+    }
+}

lib/PhpParser/Node/Expr/ArrayDimFetch.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr      $var Variable
- * @property null|Expr $dim Array index / dim
- */
 class ArrayDimFetch extends Expr
 {
+    /** @var Expr Variable */
+    public $var;
+    /** @var null|Expr Array index / dim */
+    public $dim;
+
     /**
      * Constructs an array index fetch node.
      *
@@ -18,12 +19,12 @@ class ArrayDimFetch extends Expr
      * @param array     $attributes Additional attributes
      */
     public function __construct(Expr $var, Expr $dim = null, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var' => $var,
-                'dim' => $dim
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
+        $this->dim = $dim;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var', 'dim');
+    }
+}

lib/PhpParser/Node/Expr/ArrayItem.php

@@ -4,13 +4,15 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr      $value Value
- * @property null|Expr $key   Key
- * @property bool                     $byRef Whether to assign by reference
- */
 class ArrayItem extends Expr
 {
+    /** @var null|Expr Key */
+    public $key;
+    /** @var Expr Value */
+    public $value;
+    /** @var bool Whether to assign by reference */
+    public $byRef;
+
     /**
      * Constructs an array item node.
      *
@@ -20,13 +22,13 @@ class ArrayItem extends Expr
      * @param array     $attributes Additional attributes
      */
     public function __construct(Expr $value, Expr $key = null, $byRef = false, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'key'   => $key,
-                'value' => $value,
-                'byRef' => $byRef
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->key = $key;
+        $this->value = $value;
+        $this->byRef = $byRef;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('key', 'value', 'byRef');
+    }
+}

lib/PhpParser/Node/Expr/Array_.php

@@ -4,11 +4,15 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property ArrayItem[] $items Items
- */
 class Array_ extends Expr
 {
+    // For use in "kind" attribute
+    const KIND_LONG = 1;  // array() syntax
+    const KIND_SHORT = 2; // [] syntax
+
+    /** @var ArrayItem[] Items */
+    public $items;
+
     /**
      * Constructs an array node.
      *
@@ -16,11 +20,11 @@ class Array_ extends Expr
      * @param array       $attributes Additional attributes
      */
     public function __construct(array $items = array(), array $attributes = array()) {
-        parent::__construct(
-            array(
-                'items' => $items
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->items = $items;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('items');
+    }
+}

lib/PhpParser/Node/Expr/Assign.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var  Variable
- * @property Expr $expr Expression
- */
 class Assign extends Expr
 {
+    /** @var Expr Variable */
+    public $var;
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs an assignment node.
      *
@@ -18,12 +19,12 @@ class Assign extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $var, Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var'  => $var,
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var', 'expr');
+    }
+}

lib/PhpParser/Node/Expr/AssignOp.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var  Variable
- * @property Expr $expr Expression
- */
 abstract class AssignOp extends Expr
 {
+    /** @var Expr Variable */
+    public $var;
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs a compound assignment operation node.
      *
@@ -18,12 +19,12 @@ abstract class AssignOp extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $var, Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var'  => $var,
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var', 'expr');
+    }
+}

lib/PhpParser/Node/Expr/AssignRef.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var  Variable reference is assigned to
- * @property Expr $expr Variable which is referenced
- */
 class AssignRef extends Expr
 {
+    /** @var Expr Variable reference is assigned to */
+    public $var;
+    /** @var Expr Variable which is referenced */
+    public $expr;
+
     /**
      * Constructs an assignment node.
      *
@@ -18,12 +19,12 @@ class AssignRef extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $var, Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var'  => $var,
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var', 'expr');
+    }
+}

lib/PhpParser/Node/Expr/BinaryOp.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $left  The left hand side expression
- * @property Expr $right The right hand side expression
- */
-class BinaryOp extends Expr
+abstract class BinaryOp extends Expr
 {
+    /** @var Expr The left hand side expression */
+    public $left;
+    /** @var Expr The right hand side expression */
+    public $right;
+
     /**
      * Constructs a bitwise and node.
      *
@@ -18,12 +19,12 @@ class BinaryOp extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $left, Expr $right, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'left'  => $left,
-                'right' => $right
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->left = $left;
+        $this->right = $right;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('left', 'right');
+    }
+}

lib/PhpParser/Node/Expr/BinaryOp/Coalesce.php

@@ -0,0 +1,9 @@
+<?php
+
+namespace PhpParser\Node\Expr\BinaryOp;
+
+use PhpParser\Node\Expr\BinaryOp;
+
+class Coalesce extends BinaryOp
+{
+}

lib/PhpParser/Node/Expr/BinaryOp/Spaceship.php

@@ -0,0 +1,9 @@
+<?php
+
+namespace PhpParser\Node\Expr\BinaryOp;
+
+use PhpParser\Node\Expr\BinaryOp;
+
+class Spaceship extends BinaryOp
+{
+}

lib/PhpParser/Node/Expr/BitwiseNot.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class BitwiseNot extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs a bitwise not node.
      *
@@ -16,11 +16,11 @@ class BitwiseNot extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/BooleanNot.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class BooleanNot extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs a boolean not node.
      *
@@ -16,11 +16,11 @@ class BooleanNot extends Expr
      * @param array               $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/Cast.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 abstract class Cast extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs a cast node.
      *
@@ -16,11 +16,11 @@ abstract class Cast extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/Cast/Bool_.php

@@ -4,6 +4,6 @@ namespace PhpParser\Node\Expr\Cast;
 
 use PhpParser\Node\Expr\Cast;
 
-class Int extends Cast
+class Bool_ extends Cast
 {
-}
+}

lib/PhpParser/Node/Expr/Cast/Double.php

@@ -6,4 +6,4 @@ use PhpParser\Node\Expr\Cast;
 
 class Double extends Cast
 {
-}
+}

lib/PhpParser/Node/Expr/Cast/Int_.php

@@ -4,6 +4,6 @@ namespace PhpParser\Node\Expr\Cast;
 
 use PhpParser\Node\Expr\Cast;
 
-class Bool extends Cast
+class Int_ extends Cast
 {
-}
+}

lib/PhpParser/Node/Expr/Cast/Object_.php

@@ -4,6 +4,6 @@ namespace PhpParser\Node\Expr\Cast;
 
 use PhpParser\Node\Expr\Cast;
 
-class Object extends Cast
+class Object_ extends Cast
 {
-}
+}

lib/PhpParser/Node/Expr/Cast/String_.php

@@ -4,6 +4,6 @@ namespace PhpParser\Node\Expr\Cast;
 
 use PhpParser\Node\Expr\Cast;
 
-class String extends Cast
+class String_ extends Cast
 {
-}
+}

lib/PhpParser/Node/Expr/ClassConstFetch.php

@@ -2,29 +2,30 @@
 
 namespace PhpParser\Node\Expr;
 
-use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
 
-/**
- * @property Name|Expr $class Class name
- * @property string    $name  Constant name
- */
 class ClassConstFetch extends Expr
 {
+    /** @var Name|Expr Class name */
+    public $class;
+    /** @var string|Error Constant name */
+    public $name;
+
     /**
      * Constructs a class const fetch node.
      *
-     * @param Name|Expr $class      Class name
-     * @param string    $name       Constant name
-     * @param array     $attributes Additional attributes
+     * @param Name|Expr    $class      Class name
+     * @param string|Error $name       Constant name
+     * @param array        $attributes Additional attributes
      */
     public function __construct($class, $name, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'class' => $class,
-                'name'  => $name
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->class = $class;
+        $this->name = $name;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('class', 'name');
+    }
+}

lib/PhpParser/Node/Expr/Clone_.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class Clone_ extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs a clone node.
      *
@@ -16,11 +16,11 @@ class Clone_ extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/Closure.php

@@ -4,37 +4,62 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node;
 use PhpParser\Node\Expr;
+use PhpParser\Node\FunctionLike;
 
-/**
- * @property Node[]       $stmts  Statements
- * @property Node\Param[] $params Parameters
- * @property ClosureUse[] $uses   use()s
- * @property bool         $byRef  Whether to return by reference
- * @property bool         $static Whether the closure is static
- */
-class Closure extends Expr
+class Closure extends Expr implements FunctionLike
 {
+    /** @var bool Whether the closure is static */
+    public $static;
+    /** @var bool Whether to return by reference */
+    public $byRef;
+    /** @var Node\Param[] Parameters */
+    public $params;
+    /** @var ClosureUse[] use()s */
+    public $uses;
+    /** @var null|string|Node\Name|Node\NullableType Return type */
+    public $returnType;
+    /** @var Node[] Statements */
+    public $stmts;
+
     /**
      * Constructs a lambda function node.
      *
      * @param array $subNodes   Array of the following optional subnodes:
-     *                          'static' => false  : Whether the closure is static
-     *                          'byRef'  => false  : Whether to return by reference
-     *                          'params' => array(): Parameters
-     *                          'uses'   => array(): use()s
-     *                          'stmts'  => array(): Statements
+     *                          'static'     => false  : Whether the closure is static
+     *                          'byRef'      => false  : Whether to return by reference
+     *                          'params'     => array(): Parameters
+     *                          'uses'       => array(): use()s
+     *                          'returnType' => null   : Return type
+     *                          'stmts'      => array(): Statements
      * @param array $attributes Additional attributes
      */
     public function __construct(array $subNodes = array(), array $attributes = array()) {
-        parent::__construct(
-            array(
-                'static' => isset($subNodes['static']) ? $subNodes['static'] : false,
-                'byRef'  => isset($subNodes['byRef'])  ? $subNodes['byRef']  : false,
-                'params' => isset($subNodes['params']) ? $subNodes['params'] : array(),
-                'uses'   => isset($subNodes['uses'])   ? $subNodes['uses']   : array(),
-                'stmts'  => isset($subNodes['stmts'])  ? $subNodes['stmts']  : array(),
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->static = isset($subNodes['static']) ? $subNodes['static'] : false;
+        $this->byRef = isset($subNodes['byRef']) ? $subNodes['byRef'] : false;
+        $this->params = isset($subNodes['params']) ? $subNodes['params'] : array();
+        $this->uses = isset($subNodes['uses']) ? $subNodes['uses'] : array();
+        $this->returnType = isset($subNodes['returnType']) ? $subNodes['returnType'] : null;
+        $this->stmts = isset($subNodes['stmts']) ? $subNodes['stmts'] : array();
     }
-}
+
+    public function getSubNodeNames() {
+        return array('static', 'byRef', 'params', 'uses', 'returnType', 'stmts');
+    }
+
+    public function returnsByRef() {
+        return $this->byRef;
+    }
+
+    public function getParams() {
+        return $this->params;
+    }
+
+    public function getReturnType() {
+        return $this->returnType;
+    }
+
+    public function getStmts() {
+        return $this->stmts;
+    }
+}

lib/PhpParser/Node/Expr/ClosureUse.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property string $var   Name of variable
- * @property bool   $byRef Whether to use by reference
- */
 class ClosureUse extends Expr
 {
+    /** @var string Name of variable */
+    public $var;
+    /** @var bool Whether to use by reference */
+    public $byRef;
+
     /**
      * Constructs a closure use node.
      *
@@ -18,12 +19,12 @@ class ClosureUse extends Expr
      * @param array       $attributes Additional attributes
      */
     public function __construct($var, $byRef = false, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var'   => $var,
-                'byRef' => $byRef
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
+        $this->byRef = $byRef;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var', 'byRef');
+    }
+}

lib/PhpParser/Node/Expr/ConstFetch.php

@@ -2,14 +2,14 @@
 
 namespace PhpParser\Node\Expr;
 
-use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
 
-/**
- * @property Name $name Constant name
- */
 class ConstFetch extends Expr
 {
+    /** @var Name Constant name */
+    public $name;
+
     /**
      * Constructs a const fetch node.
      *
@@ -17,11 +17,11 @@ class ConstFetch extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Name $name, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'name'  => $name
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->name = $name;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('name');
+    }
+}

lib/PhpParser/Node/Expr/Empty_.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class Empty_ extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs an empty() node.
      *
@@ -16,11 +16,11 @@ class Empty_ extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/Error.php

@@ -0,0 +1,27 @@
+<?php
+
+namespace PhpParser\Node\Expr;
+
+use PhpParser\Node\Expr;
+
+/**
+ * Error node used during parsing with error recovery.
+ *
+ * An error node may be placed at a position where an expression is required, but an error occurred.
+ * Error nodes will not be present if the parser is run in throwOnError mode (the default).
+ */
+class Error extends Expr
+{
+    /**
+     * Constructs an error node.
+     *
+     * @param array $attributes Additional attributes
+     */
+    public function __construct(array $attributes = array()) {
+        parent::__construct($attributes);
+    }
+
+    public function getSubNodeNames() {
+        return array();
+    }
+}

lib/PhpParser/Node/Expr/ErrorSuppress.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class ErrorSuppress extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs an error suppress node.
      *
@@ -16,11 +16,11 @@ class ErrorSuppress extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/Eval_.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class Eval_ extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs an eval() node.
      *
@@ -16,11 +16,11 @@ class Eval_ extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/Exit_.php

@@ -4,11 +4,15 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property null|Expr $expr Expression
- */
 class Exit_ extends Expr
 {
+    /* For use in "kind" attribute */
+    const KIND_EXIT = 1;
+    const KIND_DIE = 2;
+
+    /** @var null|Expr Expression */
+    public $expr;
+
     /**
      * Constructs an exit() node.
      *
@@ -16,11 +20,11 @@ class Exit_ extends Expr
      * @param array                    $attributes Additional attributes
      */
     public function __construct(Expr $expr = null, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/FuncCall.php

@@ -5,12 +5,13 @@ namespace PhpParser\Node\Expr;
 use PhpParser\Node;
 use PhpParser\Node\Expr;
 
-/**
- * @property Node\Name|Expr $name Function name
- * @property Node\Arg[]                    $args Arguments
- */
 class FuncCall extends Expr
 {
+    /** @var Node\Name|Expr Function name */
+    public $name;
+    /** @var Node\Arg[] Arguments */
+    public $args;
+
     /**
      * Constructs a function call node.
      *
@@ -19,12 +20,12 @@ class FuncCall extends Expr
      * @param array                                   $attributes Additional attributes
      */
     public function __construct($name, array $args = array(), array $attributes = array()) {
-        parent::__construct(
-            array(
-                'name' => $name,
-                'args' => $args
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->name = $name;
+        $this->args = $args;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('name', 'args');
+    }
+}

lib/PhpParser/Node/Expr/Include_.php

@@ -4,10 +4,6 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- * @property int  $type Type of include
- */
 class Include_ extends Expr
 {
     const TYPE_INCLUDE      = 1;
@@ -15,6 +11,11 @@ class Include_ extends Expr
     const TYPE_REQUIRE      = 3;
     const TYPE_REQUIRE_ONCE = 4;
 
+    /** @var Expr Expression */
+    public $expr;
+    /** @var int Type of include */
+    public $type;
+
     /**
      * Constructs an include node.
      *
@@ -23,12 +24,12 @@ class Include_ extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, $type, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr,
-                'type' => $type
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
+        $this->type = $type;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr', 'type');
+    }
+}

lib/PhpParser/Node/Expr/Instanceof_.php

@@ -2,15 +2,16 @@
 
 namespace PhpParser\Node\Expr;
 
-use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
 
-/**
- * @property Expr $expr  Expression
- * @property Name|Expr $class Class name
- */
 class Instanceof_ extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+    /** @var Name|Expr Class name */
+    public $class;
+
     /**
      * Constructs an instanceof check node.
      *
@@ -19,12 +20,12 @@ class Instanceof_ extends Expr
      * @param array     $attributes Additional attributes
      */
     public function __construct(Expr $expr, $class, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr'  => $expr,
-                'class' => $class
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
+        $this->class = $class;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr', 'class');
+    }
+}

lib/PhpParser/Node/Expr/Isset_.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr[] $vars Variables
- */
 class Isset_ extends Expr
 {
+    /** @var Expr[] Variables */
+    public $vars;
+
     /**
      * Constructs an array node.
      *
@@ -16,11 +16,11 @@ class Isset_ extends Expr
      * @param array  $attributes Additional attributes
      */
     public function __construct(array $vars, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'vars' => $vars
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->vars = $vars;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('vars');
+    }
+}

lib/PhpParser/Node/Expr/List_.php

@@ -4,23 +4,23 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr[] $vars List of variables to assign to
- */
 class List_ extends Expr
 {
+    /** @var ArrayItem[] List of items to assign to */
+    public $items;
+
     /**
      * Constructs a list() destructuring node.
      *
-     * @param Expr[] $vars       List of variables to assign to
-     * @param array  $attributes Additional attributes
+     * @param ArrayItem[] $items      List of items to assign to
+     * @param array       $attributes Additional attributes
      */
-    public function __construct(array $vars, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'vars' => $vars,
-            ),
-            $attributes
-        );
+    public function __construct(array $items, array $attributes = array()) {
+        parent::__construct($attributes);
+        $this->items = $items;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('items');
+    }
+}

lib/PhpParser/Node/Expr/MethodCall.php

@@ -5,13 +5,15 @@ namespace PhpParser\Node\Expr;
 use PhpParser\Node\Arg;
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr        $var  Variable holding object
- * @property string|Expr $name Method name
- * @property Arg[]       $args Arguments
- */
 class MethodCall extends Expr
 {
+    /** @var Expr Variable holding object */
+    public $var;
+    /** @var string|Expr Method name */
+    public $name;
+    /** @var Arg[] Arguments */
+    public $args;
+
     /**
      * Constructs a function call node.
      *
@@ -21,13 +23,13 @@ class MethodCall extends Expr
      * @param array       $attributes Additional attributes
      */
     public function __construct(Expr $var, $name, array $args = array(), array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var'  => $var,
-                'name' => $name,
-                'args' => $args
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
+        $this->name = $name;
+        $this->args = $args;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var', 'name', 'args');
+    }
+}

lib/PhpParser/Node/Expr/New_.php

@@ -5,26 +5,27 @@ namespace PhpParser\Node\Expr;
 use PhpParser\Node;
 use PhpParser\Node\Expr;
 
-/**
- * @property Node\Name|Expr $class Class name
- * @property Node\Arg[]     $args  Arguments
- */
 class New_ extends Expr
 {
+    /** @var Node\Name|Expr|Node\Stmt\Class_ Class name */
+    public $class;
+    /** @var Node\Arg[] Arguments */
+    public $args;
+
     /**
      * Constructs a function call node.
      *
-     * @param Node\Name|Expr $class      Class name
-     * @param Node\Arg[]     $args       Arguments
-     * @param array          $attributes Additional attributes
+     * @param Node\Name|Expr|Node\Stmt\Class_ $class      Class name (or class node for anonymous classes)
+     * @param Node\Arg[]                      $args       Arguments
+     * @param array                           $attributes Additional attributes
      */
     public function __construct($class, array $args = array(), array $attributes = array()) {
-        parent::__construct(
-            array(
-                'class' => $class,
-                'args'  => $args
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->class = $class;
+        $this->args = $args;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('class', 'args');
+    }
+}

lib/PhpParser/Node/Expr/PostDec.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var Variable
- */
 class PostDec extends Expr
 {
+    /** @var Expr Variable */
+    public $var;
+
     /**
      * Constructs a post decrement node.
      *
@@ -16,11 +16,11 @@ class PostDec extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $var, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var' => $var
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var');
+    }
+}

lib/PhpParser/Node/Expr/PostInc.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var Variable
- */
 class PostInc extends Expr
 {
+    /** @var Expr Variable */
+    public $var;
+
     /**
      * Constructs a post increment node.
      *
@@ -16,11 +16,11 @@ class PostInc extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $var, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var' => $var
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var');
+    }
+}

lib/PhpParser/Node/Expr/PreDec.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var Variable
- */
 class PreDec extends Expr
 {
+    /** @var Expr Variable */
+    public $var;
+
     /**
      * Constructs a pre decrement node.
      *
@@ -16,11 +16,11 @@ class PreDec extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $var, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var' => $var
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var');
+    }
+}

lib/PhpParser/Node/Expr/PreInc.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var Variable
- */
 class PreInc extends Expr
 {
+    /** @var Expr Variable */
+    public $var;
+
     /**
      * Constructs a pre increment node.
      *
@@ -16,11 +16,11 @@ class PreInc extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $var, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var' => $var
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var');
+    }
+}

lib/PhpParser/Node/Expr/Print_.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class Print_ extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs an print() node.
      *
@@ -16,11 +16,11 @@ class Print_ extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/PropertyFetch.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr        $var  Variable holding object
- * @property string|Expr $name Property Name
- */
 class PropertyFetch extends Expr
 {
+    /** @var Expr Variable holding object */
+    public $var;
+    /** @var string|Expr Property name */
+    public $name;
+
     /**
      * Constructs a function call node.
      *
@@ -18,12 +19,12 @@ class PropertyFetch extends Expr
      * @param array       $attributes Additional attributes
      */
     public function __construct(Expr $var, $name, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'var'  => $var,
-                'name' => $name
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->var = $var;
+        $this->name = $name;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('var', 'name');
+    }
+}

lib/PhpParser/Node/Expr/ShellExec.php

@@ -4,23 +4,23 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property array $parts Encapsed string array
- */
 class ShellExec extends Expr
 {
+    /** @var array Encapsed string array */
+    public $parts;
+
     /**
      * Constructs a shell exec (backtick) node.
      *
      * @param array $parts      Encapsed string array
      * @param array $attributes Additional attributes
      */
-    public function __construct($parts, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'parts' => $parts
-            ),
-            $attributes
-        );
+    public function __construct(array $parts, array $attributes = array()) {
+        parent::__construct($attributes);
+        $this->parts = $parts;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('parts');
+    }
+}

lib/PhpParser/Node/Expr/StaticCall.php

@@ -5,13 +5,15 @@ namespace PhpParser\Node\Expr;
 use PhpParser\Node;
 use PhpParser\Node\Expr;
 
-/**
- * @property Node\Name|Expr $class Class name
- * @property string|Expr    $name  Method name
- * @property Node\Arg[]     $args  Arguments
- */
 class StaticCall extends Expr
 {
+    /** @var Node\Name|Expr Class name */
+    public $class;
+    /** @var string|Expr Method name */
+    public $name;
+    /** @var Node\Arg[] Arguments */
+    public $args;
+
     /**
      * Constructs a static method call node.
      *
@@ -21,13 +23,13 @@ class StaticCall extends Expr
      * @param array          $attributes Additional attributes
      */
     public function __construct($class, $name, array $args = array(), array $attributes = array()) {
-        parent::__construct(
-            array(
-                'class' => $class,
-                'name'  => $name,
-                'args'  => $args
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->class = $class;
+        $this->name = $name;
+        $this->args = $args;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('class', 'name', 'args');
+    }
+}

lib/PhpParser/Node/Expr/StaticPropertyFetch.php

@@ -2,15 +2,16 @@
 
 namespace PhpParser\Node\Expr;
 
-use PhpParser\Node\Name;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Name;
 
-/**
- * @property Name|Expr   $class Class name
- * @property string|Expr $name  Property name
- */
 class StaticPropertyFetch extends Expr
 {
+    /** @var Name|Expr Class name */
+    public $class;
+    /** @var string|Expr Property name */
+    public $name;
+
     /**
      * Constructs a static property fetch node.
      *
@@ -19,12 +20,12 @@ class StaticPropertyFetch extends Expr
      * @param array       $attributes Additional attributes
      */
     public function __construct($class, $name, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'class' => $class,
-                'name'  => $name
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->class = $class;
+        $this->name = $name;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('class', 'name');
+    }
+}

lib/PhpParser/Node/Expr/Ternary.php

@@ -4,13 +4,15 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr      $cond Condition
- * @property null|Expr $if   Expression for true
- * @property Expr      $else Expression for false
- */
 class Ternary extends Expr
 {
+    /** @var Expr Condition */
+    public $cond;
+    /** @var null|Expr Expression for true */
+    public $if;
+    /** @var Expr Expression for false */
+    public $else;
+
     /**
      * Constructs a ternary operator node.
      *
@@ -20,13 +22,13 @@ class Ternary extends Expr
      * @param array                    $attributes Additional attributes
      */
     public function __construct(Expr $cond, $if, Expr $else, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'cond' => $cond,
-                'if'   => $if,
-                'else' => $else
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->cond = $cond;
+        $this->if = $if;
+        $this->else = $else;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('cond', 'if', 'else');
+    }
+}

lib/PhpParser/Node/Expr/UnaryMinus.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class UnaryMinus extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs a unary minus node.
      *
@@ -16,11 +16,11 @@ class UnaryMinus extends Expr
      * @param array $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/UnaryPlus.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $expr Expression
- */
 class UnaryPlus extends Expr
 {
+    /** @var Expr Expression */
+    public $expr;
+
     /**
      * Constructs a unary plus node.
      *
@@ -16,11 +16,11 @@ class UnaryPlus extends Expr
      * @param array               $attributes Additional attributes
      */
     public function __construct(Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'expr' => $expr
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->expr = $expr;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/Variable.php

@@ -4,11 +4,11 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property string|Expr $name Name
- */
 class Variable extends Expr
 {
+    /** @var string|Expr Name */
+    public $name;
+
     /**
      * Constructs a variable node.
      *
@@ -16,11 +16,11 @@ class Variable extends Expr
      * @param array                      $attributes Additional attributes
      */
     public function __construct($name, array $attributes = array()) {
-        parent::__construct(
-            array(
-                 'name' => $name
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->name = $name;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('name');
+    }
+}

lib/PhpParser/Node/Expr/YieldFrom.php

@@ -0,0 +1,26 @@
+<?php
+
+namespace PhpParser\Node\Expr;
+
+use PhpParser\Node\Expr;
+
+class YieldFrom extends Expr
+{
+    /** @var Expr Expression to yield from */
+    public $expr;
+
+    /**
+     * Constructs an "yield from" node.
+     *
+     * @param Expr  $expr       Expression
+     * @param array $attributes Additional attributes
+     */
+    public function __construct(Expr $expr, array $attributes = array()) {
+        parent::__construct($attributes);
+        $this->expr = $expr;
+    }
+
+    public function getSubNodeNames() {
+        return array('expr');
+    }
+}

lib/PhpParser/Node/Expr/Yield_.php

@@ -4,12 +4,13 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property null|Expr $value Value expression
- * @property null|Expr $key   Key expression
- */
 class Yield_ extends Expr
 {
+    /** @var null|Expr Key expression */
+    public $key;
+    /** @var null|Expr Value expression */
+    public $value;
+
     /**
      * Constructs a yield expression node.
      *
@@ -18,12 +19,12 @@ class Yield_ extends Expr
      * @param array     $attributes Additional attributes
      */
     public function __construct(Expr $value = null, Expr $key = null, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'key'   => $key,
-                'value' => $value,
-            ),
-            $attributes
-        );
+        parent::__construct($attributes);
+        $this->key = $key;
+        $this->value = $value;
     }
-}
+
+    public function getSubNodeNames() {
+        return array('key', 'value');
+    }
+}

lib/PhpParser/Node/FunctionLike.php

@@ -0,0 +1,36 @@
+<?php
+
+namespace PhpParser\Node;
+
+use PhpParser\Node;
+
+interface FunctionLike extends Node
+{
+    /**
+     * Whether to return by reference
+     *
+     * @return bool
+     */
+    public function returnsByRef();
+
+    /**
+     * List of parameters
+     *
+     * @return Node\Param[]
+     */
+    public function getParams();
+
+    /**
+     * Get the declared return type or null
+     * 
+     * @return null|string|Node\Name|Node\NullableType
+     */
+    public function getReturnType();
+
+    /**
+     * The function body
+     *
+     * @return Node\Stmt[]
+     */
+    public function getStmts();
+}

lib/PhpParser/Node/Name.php

@@ -4,28 +4,26 @@ namespace PhpParser\Node;
 
 use PhpParser\NodeAbstract;
 
-/**
- * @property array $parts Parts of the name
- */
 class Name extends NodeAbstract
 {
+    /**
+     * @var string[] Parts of the name
+     */
+    public $parts;
+
     /**
      * Constructs a name node.
      *
-     * @param string|array $parts      Parts of the name (or name as string)
-     * @param array        $attributes Additional attributes
+     * @param string|array|self $name       Name as string, part array or Name instance (copy ctor)
+     * @param array             $attributes Additional attributes
      */
-    public function __construct($parts, array $attributes = array()) {
-        if (!is_array($parts)) {
-            $parts = explode('\\', $parts);
-        }
+    public function __construct($name, array $attributes = array()) {
+        parent::__construct($attributes);
+        $this->parts = self::prepareName($name);
+    }
 
-        parent::__construct(
-            array(
-                'parts' => $parts,
-            ),
-            $attributes
-        );
+    public function getSubNodeNames() {
+        return array('parts');
     }
 
     /**
@@ -83,14 +81,13 @@ class Name extends NodeAbstract
     }
 
     /**
-     * Returns a string representation of the name by imploding the namespace parts with a separator.
-     *
-     * @param string $separator The separator to use (defaults to the namespace separator \)
+     * Returns a string representation of the name by imploding the namespace parts with the
+     * namespace separator.
      *
      * @return string String representation
      */
-    public function toString($separator = '\\') {
-        return implode($separator, $this->parts);
+    public function toString() {
+        return implode('\\', $this->parts);
     }
 
     /**
@@ -104,48 +101,75 @@ class Name extends NodeAbstract
     }
 
     /**
-     * Sets the whole name.
+     * Gets a slice of a name (similar to array_slice).
      *
-     * @param string|array|self $name The name to set the whole name to
+     * This method returns a new instance of the same type as the original and with the same
+     * attributes.
+     *
+     * If the slice is empty, null is returned. The null value will be correctly handled in
+     * concatenations using concat().
+     *
+     * Offset and length have the same meaning as in array_slice().
+     *
+     * @param int      $offset Offset to start the slice at (may be negative)
+     * @param int|null $length Length of the slice (may be negative)
+     *
+     * @return static|null Sliced name
      */
-    public function set($name) {
-        $this->parts = $this->prepareName($name);
+    public function slice($offset, $length = null) {
+        $numParts = count($this->parts);
+
+        $realOffset = $offset < 0 ? $offset + $numParts : $offset;
+        if ($realOffset < 0 || $realOffset > $numParts) {
+            throw new \OutOfBoundsException(sprintf('Offset %d is out of bounds', $offset));
+        }
+
+        if (null === $length) {
+            $realLength = $numParts - $realOffset;
+        } else {
+            $realLength = $length < 0 ? $length + $numParts - $realOffset : $length;
+            if ($realLength < 0 || $realLength > $numParts) {
+                throw new \OutOfBoundsException(sprintf('Length %d is out of bounds', $length));
+            }
+        }
+
+        if ($realLength === 0) {
+            // Empty slice is represented as null
+            return null;
+        }
+
+        return new static(array_slice($this->parts, $realOffset, $realLength), $this->attributes);
     }
 
     /**
-     * Prepends a name to this name.
+     * Concatenate two names, yielding a new Name instance.
      *
-     * @param string|array|self $name Name to prepend
-     */
-    public function prepend($name) {
-        $this->parts = array_merge($this->prepareName($name), $this->parts);
-    }
-
-    /**
-     * Appends a name to this name.
+     * The type of the generated instance depends on which class this method is called on, for
+     * example Name\FullyQualified::concat() will yield a Name\FullyQualified instance.
      *
-     * @param string|array|self $name Name to append
-     */
-    public function append($name) {
-        $this->parts = array_merge($this->parts, $this->prepareName($name));
-    }
-
-    /**
-     * Sets the first part of the name.
+     * If one of the arguments is null, a new instance of the other name will be returned. If both
+     * arguments are null, null will be returned. As such, writing
+     *     Name::concat($namespace, $shortName)
+     * where $namespace is a Name node or null will work as expected.
      *
-     * @param string|array|self $name The name to set the first part to
-     */
-    public function setFirst($name) {
-        array_splice($this->parts, 0, 1, $this->prepareName($name));
-    }
-
-    /**
-     * Sets the last part of the name.
+     * @param string|array|self|null $name1      The first name
+     * @param string|array|self|null $name2      The second name
+     * @param array                  $attributes Attributes to assign to concatenated name
      *
-     * @param string|array|self $name The name to set the last part to
+     * @return static|null Concatenated name
      */
-    public function setLast($name) {
-        array_splice($this->parts, -1, 1, $this->prepareName($name));
+    public static function concat($name1, $name2, array $attributes = []) {
+        if (null === $name1 && null === $name2) {
+            return null;
+        } elseif (null === $name1) {
+            return new static(self::prepareName($name2), $attributes);
+        } else if (null === $name2) {
+            return new static(self::prepareName($name1), $attributes);
+        } else {
+            return new static(
+                array_merge(self::prepareName($name1), self::prepareName($name2)), $attributes
+            );
+        }
     }
 
     /**
@@ -156,17 +180,17 @@ class Name extends NodeAbstract
      *
      * @return array Prepared name
      */
-    protected function prepareName($name) {
-        if (is_string($name)) {
+    private static function prepareName($name) {
+        if (\is_string($name)) {
             return explode('\\', $name);
-        } elseif (is_array($name)) {
+        } elseif (\is_array($name)) {
             return $name;
         } elseif ($name instanceof self) {
             return $name->parts;
         }
 
         throw new \InvalidArgumentException(
-            'When changing a name you need to pass either a string, an array or a Name node'
+            'Expected string, array of parts or Name instance'
         );
     }
-}
+}