Release PHP-Parser 2.1.1

In particular, support recovering from a missing trailing semicolon,
while keeping the rest of the expression.

.gitignore

@@ -0,0 +1,4 @@
+vendor/
+composer.lock
+grammar/kmyacc.exe
+grammar/y.output

.travis.yml

@@ -1,13 +1,32 @@
 language: php
 
+sudo: false
+
+cache:
+  directories:
+    - $HOME/.composer/cache
+
 php:
   - 5.4
   - 5.5
   - 5.6
   - 7.0
+  - nightly
   - hhvm
 
+install:
+  - if [ $TRAVIS_PHP_VERSION = '5.6' ]; then composer require satooshi/php-coveralls '~1.0'; fi
+  - composer install --prefer-dist
+
 matrix:
   allow_failures:
-    - php: 7.0
+    - php: nightly
   fast_finish: true
+
+script:
+  - if [ $TRAVIS_PHP_VERSION = '5.6' ]; then vendor/bin/phpunit --coverage-clover build/logs/clover.xml; else vendor/bin/phpunit; fi
+  - if [ $TRAVIS_PHP_VERSION = '7.0' ]; then test_old/run-php-src.sh; fi
+
+after_success:
+  if [ $TRAVIS_PHP_VERSION = '5.6' ]; then php vendor/bin/coveralls; fi
+

CHANGELOG.md

@@ -1,8 +1,109 @@
-Version 2.0.0-dev-dev
----------------------
+Version 2.1.2-dev
+-----------------
 
 Nothing yet.
 
+Version 2.1.1 (2016-09-16)
+--------------------------
+
+### Changed
+
+* The pretty printer will now escape all control characters in the range `\x00-\x1F` inside double
+  quoted strings. If no special escape sequence is available, an octal escape will be used.
+* The quality of the error recovery has been improved. In particular unterminated expressions should
+  be handled more gracefully.
+* The PHP 7 parser will now generate a parse error for `$var =& new Obj` assignments.
+* Comments on free-standing code blocks will no be retained as comments on the first statement in
+  the code block.
+
+Version 2.1.0 (2016-04-19)
+--------------------------
+
+### Fixed
+
+* Properly support `B""` strings (with uppercase `B`) in a number of places.
+* Fixed reformatting of indented parts in a certain non-standard comment style.
+
+### Added
+
+* Added `dumpComments` option to node dumper, to enable dumping of comments associated with nodes.
+* Added `Stmt\Nop` node, that is used to collect comments located at the end of a block or at the
+  end of a file (without a following node with which they could otherwise be associated).
+* Added `kind` attribute to `Expr\Exit` to distinguish between `exit` and `die`.
+* Added `kind` attribute to `Scalar\LNumber` to distinguish between decimal, binary, octal and
+  hexadecimal numbers.
+* Added `kind` attribtue to `Expr\Array` to distinguish between `array()` and `[]`.
+* Added `kind` attribute to `Scalar\String` and `Scalar\Encapsed` to distinguish between
+  single-quoted, double-quoted, heredoc and nowdoc string.
+* Added `docLabel` attribute to `Scalar\String` and `Scalar\Encapsed`, if it is a heredoc or
+  nowdoc string.
+* Added start file offset information to `Comment` nodes.
+* Added `setReturnType()` method to function and method builders.
+* Added `-h` and `--help` options to `php-parse` script.
+
+### Changed
+
+* Invalid octal literals now throw a parse error in PHP 7 mode.
+* The pretty printer takes all the new attributes mentioned in the previous section into account.
+* The protected `AbstractPrettyPrinter::pComments()` method no longer returns a trailing newline.
+* The bundled autoloader supports library files being stored in a different directory than
+  `PhpParser` for easier downstream distribution.
+
+### Deprecated
+
+* The `Comment::setLine()` and `Comment::setText()` methods have been deprecated. Construct new
+  objects instead.
+
+### Removed
+
+* The internal (but public) method `Scalar\LNumber::parse()` has been removed. A non-internal
+  `LNumber::fromString()` method has been added instead.
+
+Version 2.0.1 (2016-02-28)
+--------------------------
+
+### Fixed
+
+* `declare() {}` and `declare();` are not semantically equivalent and will now result in different
+  ASTs. The format case will have an empty `stmts` array, while the latter will set `stmts` to
+  `null`.
+* Magic constants are now supported as semi-reserved keywords.
+* A shebang line like `#!/usr/bin/env php` is now allowed at the start of a namespaced file.
+  Previously this generated an exception.
+* The `prettyPrintFile()` method will not strip a trailing `?>` from the raw data that follows a
+  `__halt_compiler()` statement.
+* The `prettyPrintFile()` method will not strip an opening `<?php` if the file starts with a
+  comment followed by InlineHTML.
+
+Version 2.0.0 (2015-12-04)
+--------------------------
+
+### Changed
+
+* String parts of encapsed strings are now represented using `Scalar\EncapsStringPart` nodes.
+  Previously raw strings were used. This affects the `parts` child of `Scalar\Encaps` and
+  `Expr\ShellExec`. The change has been done to allow assignment of attributes to encapsed string
+  parts.
+
+Version 2.0.0-beta1 (2015-10-21)
+--------------------------------
+
+### Fixed
+
+* Fixed issue with too many newlines being stripped at the end of heredoc/nowdoc strings in some
+  cases. (#227)
+
+### Changed
+
+* Update group use support to be in line with recent PHP 7.0 builds.
+* Renamed `php-parse.php` to `php-parse` and registered it as a composer bin.
+* Use composer PSR-4 autoloader instead of custom autoloader.
+* Specify phpunit as a dev dependency.
+
+### Added
+
+* Added `shortArraySyntax` option to pretty printer, to print all arrays using short syntax.
+
 Version 2.0.0-alpha1 (2015-07-14)
 ---------------------------------
 
@@ -43,242 +144,8 @@ A more detailed description of backwards incompatible changes can be found in th
 * The constructor for `Scalar` nodes no longer has a default value. E.g. `new LNumber()` should now
   be written as `new LNumber(0)`.
 
-Version 1.4.0 (2015-07-14)
---------------------------
-
-### Added
-
-* Added interface `PhpParser\Node\FunctionLike`, which is implemented by `Stmt\ClassMethod`,
-  `Stmt\Function_` and `Expr\Closure` nodes. This interface provides getters for their common
-  subnodes.
-* Added `Node\Stmt\ClassLike::getMethod()` to look up a specific method on a class/interface/trait.
-
-### Fixed
-
-* Fixed `isPublic()` return value for implicitly public properties and methods that define and
-  additional modifier like `static` or `abstract`.
-* Properties are now accepted by the trait builder.
-* Fixed `__HALT_COMPILER_OFFSET__` support on HHVM.
-
-Version 1.3.0 (2015-05-02)
---------------------------
-
-### Added
-
-* Errors can now store the attributes of the node/token where the error occurred. Previously only the start line was
-  stored.
-* If file positions are enabled in the lexer, errors can now provide column information if it is available. See
-  [documentation](https://github.com/nikic/PHP-Parser/blob/master/doc/component/Error.markdown#column-information).
-* The parser now provides an experimental error recovery mode, which can be enabled by disabling the `throwOnError`
-  parser option. In this mode the parser will try to construct a partial AST even if the code is not valid PHP. See
-  [documentation](https://github.com/nikic/PHP-Parser/blob/master/doc/component/Error.markdown#error-recovery).
-* Added support for PHP 7 `yield from` expression. It is represented by `Expr\YieldFrom`.
-* Added support for PHP 7 anonymous classes. These are represented by ordinary `Stmt\Class_` nodes with the name set to
-  `null`. Furthermore this implies that `Expr\New_` can now contain a `Stmt\Class_` in its `class` subnode.
-
-### Fixed
-
-* Fixed registration of PHP 7 aliases, for the case where the old name was used before the new name.
-* Fixed handling of precedence when pretty-printing `print` expressions.
-* Floating point numbers are now pretty-printed with a higher precision.
-* Checks for special class names like `self` are now case-insensitive.
-
-Version 1.2.2 (2015-04-03)
---------------------------
-
-* The `NameResolver` now resolves parameter type hints when entering the function/method/closure node. As such other
-  visitors running after it will be able to make use of the resolved names at that point already.
-* The autoloader no longer sets the `unserialize_callback_func` ini option on registration - this is not necessary and
-  may cause issues when running PhpUnit tests with process isolation.
-
-Version 1.2.1 (2015-03-24)
---------------------------
-
-* Fixed registration of the aliases introduced in 1.2.0. Previously the old class names could not be used in
-  `instanceof` checks under some circumstances.
-
-Version 1.2.0 (2015-03-22)
---------------------------
-
-### Changed
-
-* To ensure compatibility with PHP 7, the following node classes have been renamed:
-
-        OLD                             => NEW
-        PhpParser\Node\Expr\Cast\Bool   => PhpParser\Node\Expr\Cast\Bool_
-        PhpParser\Node\Expr\Cast\Int    => PhpParser\Node\Expr\Cast\Int_
-        PhpParser\Node\Expr\Cast\Object => PhpParser\Node\Expr\Cast\Object_
-        PhpParser\Node\Expr\Cast\String => PhpParser\Node\Expr\Cast\String_
-        PhpParser\Node\Scalar\String    => PhpParser\Node\Scalar\String_
-
-  **The previous class names are still supported as aliases.** However it is strongly encouraged to use the new names
-  in order to make your code compatible with PHP 7.
-
-* Subnodes are now stored using real properties instead of an array. This improves performance and memory usage of the
-  initial parse and subsequent node tree operations. The `NodeAbstract` class still supports the old way of specifying
-  subnodes, however this is *deprecated*. In any case properties that are assigned to a node after creation will no
-  longer be considered as subnodes.
-
-* Methods and property declarations will no longer set the `Stmt\Class_::MODIFIER_PUBLIC` flag if no visibility is
-  explicitly given. However the `isPublic()` method will continue to return true. This allows you to distinguish whether
-  a method/property is explicitly or implicitly public and control the pretty printer output more precisely.
-
-* The `Stmt\Class_`, `Stmt\Interface_` and `Stmt\Trait_` nodes now inherit from `Stmt\ClassLike`, which provides a
-  `getMethods()` method. Previously this method was only available on `Stmt\Class_`.
-
-* Support including the `bootstrap.php` file multiple times.
-
-* Make documentation and tests part of the release tarball again.
-
-* Improve support for HHVM and PHP 7.
-
-### Added
-
-* Added support for PHP 7 return type declarations. This adds an additional `returnType` subnode to `Stmt\Function_`,
-  `Stmt\ClassMethod` and `Expr\Closure`.
-
-* Added support for the PHP 7 null coalesce operator `??`. The operator is represented by `Expr\BinaryOp\Coalesce`.
-
-* Added support for the PHP 7 spaceship operator `<=>`. The operator is represented by `Expr\BinaryOp\Spaceship`.
-
-* Added use builder.
-
-* Added global namespace support to the namespace builder.
-
-* Added a constructor flag to `NodeTraverser`, which disables cloning of nodes.
-
-Version 1.1.0 (2015-01-18)
---------------------------
-
-* Methods that do not specify an explicit visibility (e.g. `function method()`) will now have the `MODIFIER_PUBLIC`
-  flag set. This also means that their `isPublic()` method will return true.
-
-* Declaring a property as abstract or final is now an error.
-
-* The `Lexer` and `Lexer\Emulative` classes now accept an `$options` array in their constructors. Currently only the
-  `usedAttributes` option is supported, which determines which attributes will be added to AST nodes. In particular
-  it is now possible to add information on the token and file positions corresponding to a node. For more details see
-  the [Lexer component](https://github.com/nikic/PHP-Parser/blob/master/doc/component/Lexer.markdown) documentation.
-
-* Node visitors can now return `NodeTraverser::DONT_TRAVERSE_CHILDREN` from `enterNode()` in order to skip all children
-  of the current node, for all visitors.
-
-* Added builders for traits and namespaces.
-
-* The class, interface, trait, function, method and property builders now support adding doc comments using the
-  `setDocComment()` method.
-
-* Added support for fully-qualified and namespace-relative names in builders. No longer allow use of name component
-  arrays.
-
-* Do not add documentation and tests to distribution archive files.
-
-Version 1.0.2 (2014-11-04)
---------------------------
-
-* The `NameResolver` visitor now also resolves names in trait adaptations (aliases and precedence declarations).
-
-* Remove stray whitespace when pretty-printing trait adaptations that only change visibility.
-
-Version 1.0.1 (2014-10-14)
---------------------------
-
-* Disallow `new` expressions without a class name. Previously `new;` was accidentally considered to be valid code.
-
-* Support T_ONUMBER token used by HHVM.
-
-* Add ability to directly pass code to the `php-parse.php` script.
-
-* Prevent truncation of `var_dump()` output in the `php-parse.php` script if XDebug is used.
-
-Version 1.0.0 (2014-09-12)
---------------------------
-
-* [BC] Removed deprecated `Template` and `TemplateLoader` classes.
-
-* Fixed XML unserializer to properly work with new namespaced node names.
-
-Version 1.0.0-beta2 (2014-08-31)
---------------------------------
-
-* [PHP 5.6] Updated support for constant scalar expressions to comply with latest changes. This means that arrays
-  and array dimension fetches are now supported as well.
-
-* [PHP 5.6] Direct array dereferencing of constants is supported now, i.e. both `FOO[0]` and `Foo::BAR[0]` are valid
-  now.
-
-* Fixed handling of special class names (`self`, `parent` and `static`) in the name resolver to be case insensitive.
-  Additionally the name resolver now enforces that special class names are only used as unqualified names, e.g. `\self`
-  is considered invalid.
-
-* The case of references to the `static` class name is now preserved. Previously `static` was always lowercased,
-  regardless of the case used in the source code.
-
-* The autoloader now only requires a file if it exists. This allows usages like
-  `class_exists('PhpParser\NotExistingClass')`.
-
-* Added experimental `bin/php-parse.php` script, which is intended to help exploring and debugging the node tree.
-
-* Separated the parser implemention (in `lib/PhpParser/ParserAbstract.php`) and the generated data (in
-  `lib/PhpParser/Parser.php`). Furthermore the parser now uses meaningful variable names and contains comments
-  explaining their usage.
-
-Version 1.0.0-beta1 (2014-03-27)
---------------------------------
-
-* [BC] PHP-Parser now requires PHP 5.3 or newer to run. It is however still possible to *parse* PHP 5.2 source code,
-  while running on a newer version.
-
-* [BC] The library has been moved to use namespaces with the `PhpParser` vendor prefix. However, the old names using
-  underscores are still available as aliases, as such most code should continue running on the new version without
-  further changes.
-
-  However, code performing dispatch operations on `Node::getType()` may be affected by some of the name changes. For
-  example a `+` node will now return type `Expr_BinaryOp_Plus` instead of `Expr_Plus`. In particular this may affect
-  custom pretty printers.
-
-  Due to conflicts with reserved keywords, some class names now end with an underscore, e.g. `PHPParser_Node_Stmt_Class`
-  is now `PhpParser\Node\Stmt\Class_`. (But as usual, the old name is still available)
-
-* [PHP 5.6] Added support for the power operator `**` (node `Expr\BinaryOp\Pow`) and the compound power assignment
-  operator `**=` (node `Expr\AssignOp\Pow`).
-
-* [PHP 5.6] Added support for variadic functions: `Param` nodes now have `variadic` as a boolean subnode.
-
-* [PHP 5.6] Added support for argument unpacking: `Arg` nodes now have `unpack` as a boolean subnode.
-
-* [PHP 5.6] Added support for aliasing of functions and constants. `Stmt\Use_` nodes now have an integral `type`
-  subnode, which is one of `Stmt\Use_::TYPE_NORMAL` (`use`), `Stmt\Use_::TYPE_FUNCTION` (`use function`) or
-  `Stmt\Use_::TYPE_CONSTANT` (`use const`).
-
-  The `NameResolver` now also supports resolution of such aliases.
-
-* [PHP 5.6] Added support for constant scalar expressions. This means that certain expressions are now allowed as the
-  initializer for constants, properties, parameters, static variables, etc.
-
-* [BC] Improved pretty printing of empty statements lists, which are now printed as `{\n}` instead of `{\n    \n}`.
-  This changes the behavior of the protected `PrettyPrinterAbstract::pStmts()` method, so custom pretty printing code
-  making use it of may need to be adjusted.
-
-* Changed the order of some subnodes to be consistent with their order in the sour code. For example `Stmt\If->cond`
-  will now appear before `Stmt\If->stmts` etc.
-
-* Added `Scalar\MagicConstant->getName()`, which returns the name of the magic constant (e.g. `__CLASS__`).
-
-**The following changes are also included in 0.9.5**:
-
-* [BC] Deprecated `PHPParser_Template` and `PHPParser_TemplateLoader`. This functionality does not belong in the main project
-  and - as far as I know - nobody is using it.
-
-* Add `NodeTraverser::removeVisitor()` method, which removes a visitor from the node traverser. This also modifies the
-  corresponding `NodeTraverserInterface`.
-
-* Fix alias resolution in `NameResolver`: Class names are now correctly handled as case-insensitive.
-
-* The undefined variable error, which is used to the lexer to reset the error state, will no longer interfere with
-  custom error handlers.
-
 ---
 
-**This changelog only includes changes from the 1.0 and 2.0 series. For older changes see the
-[0.9 series changelog][https://github.com/nikic/PHP-Parser/blob/0.9/CHANGELOG.md].**
+**This changelog only includes changes from the 2.0 series. For older changes see the
+[1.x series changelog](https://github.com/nikic/PHP-Parser/blob/1.x/CHANGELOG.md) and the
+[0.9 series changelog](https://github.com/nikic/PHP-Parser/blob/0.9/CHANGELOG.md).**

README.md

@@ -1,12 +1,14 @@
 PHP Parser
 ==========
 
-This is a PHP 5.2 to PHP 5.6 parser written in PHP. Its purpose is to simplify static code analysis and
+[![Build Status](https://travis-ci.org/nikic/PHP-Parser.svg?branch=master)](https://travis-ci.org/nikic/PHP-Parser) [![Coverage Status](https://coveralls.io/repos/github/nikic/PHP-Parser/badge.svg?branch=master)](https://coveralls.io/github/nikic/PHP-Parser?branch=master)
+
+This is a PHP 5.2 to PHP 7.0 parser written in PHP. Its purpose is to simplify static code analysis and
 manipulation.
 
-[**Documentation for version 1.x**][doc_1_x] (stable; for running on PHP >= 5.3).
+[**Documentation for version 2.x**][doc_master] (stable; for running on PHP >= 5.4; for parsing PHP 5.2 to PHP 7.0).
 
-[Documentation for version 2.x-dev][doc_master] (dev; for running on PHP >= 5.4).
+[Documentation for version 1.x][doc_1_x] (unsupported; for running on PHP >= 5.3; for parsing PHP 5.2 to PHP 5.6).
 
 In a Nutshell
 -------------
@@ -70,18 +72,25 @@ programming errors or security issues).
 Additionally, you can convert a syntax tree back to PHP code. This allows you to do code preprocessing
 (like automatedly porting code to older PHP versions).
 
+Installation
+------------
+
+The preferred installation method is [composer](https://getcomposer.org):
+
+    php composer.phar require nikic/php-parser
+
 Documentation
 -------------
 
  1. [Introduction](doc/0_Introduction.markdown)
- 2. [Installation](doc/1_Installation.markdown)
- 3. [Usage of basic components](doc/2_Usage_of_basic_components.markdown)
- 4. [Other node tree representations](doc/3_Other_node_tree_representations.markdown)
- 5. [Code generation](doc/4_Code_generation.markdown)
+ 2. [Usage of basic components](doc/2_Usage_of_basic_components.markdown)
+ 3. [Other node tree representations](doc/3_Other_node_tree_representations.markdown)
+ 4. [Code generation](doc/4_Code_generation.markdown)
 
 Component documentation:
 
- 1. [Lexer](doc/component/Lexer.markdown)
+ 1. [Error](doc/component/Error.markdown)
+ 2. [Lexer](doc/component/Lexer.markdown)
 
  [doc_1_x]: https://github.com/nikic/PHP-Parser/tree/1.x/doc
  [doc_master]: https://github.com/nikic/PHP-Parser/tree/master/doc

UPGRADE-2.0.md

@@ -68,4 +68,7 @@ deprecated. Instead `Name::concat()` and `Name->slice()` should be used.
 * The legacy node format has been removed. If you use custom nodes, they are now expected to
   implement a `getSubNodeNames()` method.
 * The default value for `Scalar` node constructors was removed. This means that something like
-  `new LNumber()` should be replaced by `new LNumber(0)`.
+  `new LNumber()` should be replaced by `new LNumber(0)`.
+* String parts of encapsed strings are now represented using `Scalar\EncapsStringPart` nodes, while
+  previously raw strings were used. This affects the `parts` child of `Scalar\Encaps` and
+  `Expr\ShellExec`.

bin/php-parse

@@ -1,7 +1,12 @@
 #!/usr/bin/env php
 <?php
 
-require __DIR__ . '/../lib/bootstrap.php';
+foreach ([__DIR__ . '/../../../autoload.php', __DIR__ . '/../vendor/autoload.php'] as $file) {
+    if (file_exists($file)) {
+        require $file;
+        break;
+    }
+}
 
 ini_set('xdebug.max_nesting_level', 3000);
 
@@ -22,10 +27,10 @@ if (empty($files)) {
 }
 
 $lexer = new PhpParser\Lexer\Emulative(array('usedAttributes' => array(
-    'startLine', 'endLine', 'startFilePos', 'endFilePos'
+    'startLine', 'endLine', 'startFilePos', 'endFilePos', 'comments'
 )));
 $parser = (new PhpParser\ParserFactory)->create(PhpParser\ParserFactory::PREFER_PHP7, $lexer);
-$dumper = new PhpParser\NodeDumper;
+$dumper = new PhpParser\NodeDumper(['dumpComments' => true]);
 $prettyPrinter = new PhpParser\PrettyPrinter\Standard;
 $serializer = new PhpParser\Serializer\XML;
 
@@ -81,11 +86,13 @@ foreach ($files as $file) {
     }
 }
 
-function showHelp($error) {
-    die($error . "\n\n" .
-        <<<OUTPUT
-Usage: php php-parse.php [operations] file1.php [file2.php ...]
-   or: php php-parse.php [operations] "<?php code"
+function showHelp($error = '') {
+    if ($error) {
+        echo $error . "\n\n";
+    }
+    die(<<<OUTPUT
+Usage: php-parse [operations] file1.php [file2.php ...]
+   or: php-parse [operations] "<?php code"
 Turn PHP source code into an abstract syntax tree.
 
 Operations is a list of the following options (--dump by default):
@@ -96,9 +103,10 @@ Operations is a list of the following options (--dump by default):
         --var-dump          var_dump() nodes (for exact structure)
     -N, --resolve-names     Resolve names using NodeVisitor\NameResolver
     -c, --with-column-info  Show column-numbers for errors (if available)
+    -h, --help              Display this page
 
 Example:
-    php php-parse.php -d -p -N -d file.php
+    php-parse -d -p -N -d file.php
 
     Dumps nodes, pretty prints them, then resolves names and dumps them again.
 
@@ -145,6 +153,10 @@ function parseArgs($args) {
             case '-c';
                 $attributes['with-column-info'] = true;
                 break;
+            case '--help':
+            case '-h';
+                showHelp();
+                break;
             case '--':
                 $parseOptions = false;
                 break;

composer.json

@@ -13,12 +13,18 @@
         "php": ">=5.4",
         "ext-tokenizer": "*"
     },
-    "autoload": {
-        "files": ["lib/bootstrap.php"]
+    "require-dev": {
+        "phpunit/phpunit": "~4.0"
     },
+    "autoload": {
+        "psr-4": {
+            "PhpParser\\": "lib/PhpParser"
+        }
+    },
+    "bin": ["bin/php-parse"],
     "extra": {
         "branch-alias": {
-            "dev-master": "2.0-dev"
+            "dev-master": "2.1-dev"
         }
     }
 }

doc/0_Introduction.markdown

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-This project is a PHP 5.2 to PHP 5.6 parser **written in PHP itself**.
+This project is a PHP 5.2 to PHP 7.0 parser **written in PHP itself**.
 
 What is this for?
 -----------------
@@ -18,7 +18,7 @@ For example an AST abstracts away the fact that in PHP variables can be written
 as `$$bar`, `${'foobar'}` or even `${!${''}=barfoo()}`. You don't have to worry about recognizing
 all the different syntaxes from a stream of tokens.
 
-Another questions is: Why would I want to have a PHP parser *written in PHP*? Well, PHP might not be
+Another question is: Why would I want to have a PHP parser *written in PHP*? Well, PHP might not be
 a language especially suited for fast parsing, but processing the AST is much easier in PHP than it
 would be in other, faster languages like C. Furthermore the people most probably wanting to do
 programmatic PHP code analysis are incidentally PHP developers, not C developers.

doc/1_Installation.markdown

@@ -1,31 +0,0 @@
-Installation
-============
-
-There are multiple ways to include the PHP parser into your project:
-
-Installing via Composer
------------------------
-
-Run the following command inside your project:
-
-    php composer.phar require nikic/php-parser
-
-If you haven't installed [Composer][1] yet, you can do so using:
-
-    curl -s http://getcomposer.org/installer | php
-
-Installing as a Git Submodule
------------------------------
-
-Run the following command to install the parser into the `vendor/PHP-Parser` folder:
-
-    git submodule add git://github.com/nikic/PHP-Parser.git vendor/PHP-Parser
-
-Installing from the Zip- or Tarball
------------------------------------
-
-Download the latest version from [the download page][2], unpack it and move the files somewhere into your project.
-
-
- [1]: https://getcomposer.org/
- [2]: https://github.com/nikic/PHP-Parser/tags

doc/2_Usage_of_basic_components.markdown

@@ -6,8 +6,7 @@ This document explains how to use the parser, the pretty printer and the node tr
 Bootstrapping
 -------------
 
-The library needs to register a class autoloader. When using composer, include the usual
-`vendor/autoload.php` file (alternatively you can use the bundled `lib/bootstrap.php` file):
+To bootstrap the library, include the autoloader generated by composer:
 
 ```php
 require 'path/to/vendor/autoload.php';
@@ -19,7 +18,9 @@ Additionally you may want to set the `xdebug.max_nesting_level` ini option to a
 ini_set('xdebug.max_nesting_level', 3000);
 ```
 
-This ensures that there will be no errors when traversing highly nested node trees.
+This ensures that there will be no errors when traversing highly nested node trees. However, it is
+preferable to disable XDebug completely, as it can easily make this library more than five times
+slower.
 
 Parsing
 -------
@@ -269,7 +270,7 @@ The `enterNode()` method can additionally return the value `NodeTraverser::DONT_
 which instructs the traverser to skip all children of the current node.
 
 The `leaveNode()` method can additionally return the value `NodeTraverser::REMOVE_NODE`, in which
-case the current node will be removed from the parent array. Furthermove it is possible to return
+case the current node will be removed from the parent array. Furthermore it is possible to return
 an array of nodes, which will be merged into the parent array at the offset of the current node.
 I.e. if in `array(A, B, C)` the node `B` should be replaced with `array(X, Y, Z)` the result will
 be `array(A, X, Y, Z, C)`.
@@ -377,7 +378,7 @@ create a name with backslashes either write `$node->toString()` or `(string) $no
 a new name from the string and return it. Returning a new node replaces the old node.
 
 Another thing we need to do is change the class/function/const declarations. Currently they contain
-only the shortname (i.e. the last part of the name), but they need to contain the complete name inclduing
+only the shortname (i.e. the last part of the name), but they need to contain the complete name including
 the namespace prefix:
 
 ```php

doc/4_Code_generation.markdown

@@ -29,6 +29,7 @@ $node = $factory->namespace('Name\Space')
         ->addStmt($factory->method('someMethod')
             ->makePublic()
             ->makeAbstract() // ->makeFinal()
+            ->setReturnType('bool')
             ->addParam($factory->param('someParam')->setTypeHint('SomeClass'))
             ->setDocComment('/**
                               * This method does something.
@@ -74,7 +75,7 @@ abstract class SomeClass extends SomeOtherClass implements A\Few, \Interfaces
      *
      * @param SomeClass And takes a parameter
      */
-    public abstract function someMethod(SomeClass $someParam);
+    public abstract function someMethod(SomeClass $someParam) : bool;
     protected function anotherMethod($someParam = 'test')
     {
         print $someParam;

grammar/php5.y

@@ -9,11 +9,17 @@ start:
     top_statement_list                                      { $$ = $this->handleNamespaces($1); }
 ;
 
-top_statement_list:
-      top_statement_list top_statement                      { pushNormalizing($1, $2); }
+top_statement_list_ex:
+      top_statement_list_ex top_statement                   { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+top_statement_list:
+      top_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
 reserved_non_modifiers:
       T_INCLUDE | T_INCLUDE_ONCE | T_EVAL | T_REQUIRE | T_REQUIRE_ONCE | T_LOGICAL_OR | T_LOGICAL_XOR | T_LOGICAL_AND
     | T_INSTANCEOF | T_NEW | T_CLONE | T_EXIT | T_IF | T_ELSEIF | T_ELSE | T_ENDIF | T_ECHO | T_DO | T_WHILE
@@ -21,6 +27,7 @@ reserved_non_modifiers:
     | T_FINALLY | T_THROW | T_USE | T_INSTEADOF | T_GLOBAL | T_VAR | T_UNSET | T_ISSET | T_EMPTY | T_CONTINUE | T_GOTO
     | T_FUNCTION | T_CONST | T_RETURN | T_PRINT | T_YIELD | T_LIST | T_SWITCH | T_ENDSWITCH | T_CASE | T_DEFAULT
     | T_BREAK | T_ARRAY | T_CALLABLE | T_EXTENDS | T_IMPLEMENTS | T_NAMESPACE | T_TRAIT | T_INTERFACE | T_CLASS
+    | T_CLASS_C | T_TRAIT_C | T_FUNC_C | T_METHOD_C | T_LINE | T_FILE | T_DIR | T_NS_C | T_HALT_COMPILER
 ;
 
 semi_reserved:
@@ -53,7 +60,7 @@ top_statement:
     | T_NAMESPACE '{' top_statement_list '}'                { $$ = Stmt\Namespace_[null,     $3]; }
     | T_USE use_declarations ';'                            { $$ = Stmt\Use_[$2, Stmt\Use_::TYPE_NORMAL]; }
     | T_USE use_type use_declarations ';'                   { $$ = Stmt\Use_[$3, $2]; }
-    | group_use_declaration                                 { $$ = $1; }
+    | group_use_declaration ';'                             { $$ = $1; }
     | T_CONST constant_declaration_list ';'                 { $$ = Stmt\Const_[$2]; }
 ;
 
@@ -64,10 +71,20 @@ use_type:
 
 /* Using namespace_name_parts here to avoid s/r conflict on T_NS_SEPARATOR */
 group_use_declaration:
-      T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' use_declarations '}'
+      T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
           { $$ = Stmt\GroupUse[Name[$3], $6, $2]; }
+    | T_USE use_type T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+          { $$ = Stmt\GroupUse[Name[$4], $7, $2]; }
     | T_USE namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
           { $$ = Stmt\GroupUse[Name[$2], $5, Stmt\Use_::TYPE_UNKNOWN]; }
+    | T_USE T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
+          { $$ = Stmt\GroupUse[Name[$3], $6, Stmt\Use_::TYPE_UNKNOWN]; }
+;
+
+unprefixed_use_declarations:
+      unprefixed_use_declarations ',' unprefixed_use_declaration
+          { push($1, $3); }
+    | unprefixed_use_declaration                            { init($1); }
 ;
 
 use_declarations:
@@ -75,21 +92,24 @@ use_declarations:
     | use_declaration                                       { init($1); }
 ;
 
-use_declaration:
-      namespace_name                                        { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; }
-    | namespace_name T_AS T_STRING                          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; }
-    | T_NS_SEPARATOR namespace_name                         { $$ = Stmt\UseUse[$2, null, Stmt\Use_::TYPE_UNKNOWN]; }
-    | T_NS_SEPARATOR namespace_name T_AS T_STRING           { $$ = Stmt\UseUse[$2, $4, Stmt\Use_::TYPE_UNKNOWN]; }
-;
-
 inline_use_declarations:
       inline_use_declarations ',' inline_use_declaration    { push($1, $3); }
     | inline_use_declaration                                { init($1); }
 ;
 
+unprefixed_use_declaration:
+      namespace_name                                        { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; }
+    | namespace_name T_AS T_STRING                          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; }
+;
+
+use_declaration:
+      unprefixed_use_declaration                            { $$ = $1; }
+    | T_NS_SEPARATOR unprefixed_use_declaration             { $$ = $2; }
+;
+
 inline_use_declaration:
-      use_declaration                                       { $$ = $1; $$->type = Stmt\Use_::TYPE_NORMAL; }
-    | use_type use_declaration                              { $$ = $2; $$->type = $1; }
+      unprefixed_use_declaration                            { $$ = $1; $$->type = Stmt\Use_::TYPE_NORMAL; }
+    | use_type unprefixed_use_declaration                   { $$ = $2; $$->type = $1; }
 ;
 
 constant_declaration_list:
@@ -110,11 +130,17 @@ class_const:
     identifier '=' static_scalar                            { $$ = Node\Const_[$1, $3]; }
 ;
 
-inner_statement_list:
-      inner_statement_list inner_statement                  { pushNormalizing($1, $2); }
+inner_statement_list_ex:
+      inner_statement_list_ex inner_statement               { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+inner_statement_list:
+      inner_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
 inner_statement:
       statement                                             { $$ = $1; }
     | function_declaration_statement                        { $$ = $1; }
@@ -123,8 +149,8 @@ inner_statement:
           { throw new Error('__HALT_COMPILER() can only be used from the outermost scope', attributes()); }
 ;
 
-statement:
-      '{' inner_statement_list '}'                          { $$ = $2; }
+non_empty_statement:
+      '{' inner_statement_list '}'                          { $$ = $2; prependLeadingComments($$); }
     | T_IF parentheses_expr statement elseif_list else_single
           { $$ = Stmt\If_[$2, ['stmts' => toArray($3), 'elseifs' => $4, 'else' => $5]]; }
     | T_IF parentheses_expr ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
@@ -152,15 +178,22 @@ statement:
     | T_FOREACH '(' expr T_AS variable T_DOUBLE_ARROW foreach_variable ')' foreach_statement
           { $$ = Stmt\Foreach_[$3, $7[0], ['keyVar' => $5, 'byRef' => $7[1], 'stmts' => $9]]; }
     | T_DECLARE '(' declare_list ')' declare_statement      { $$ = Stmt\Declare_[$3, $5]; }
-    | ';'                                                   { $$ = array(); /* means: no statement */ }
     | T_TRY '{' inner_statement_list '}' catches optional_finally
           { $$ = Stmt\TryCatch[$3, $5, $6]; }
     | T_THROW expr ';'                                      { $$ = Stmt\Throw_[$2]; }
     | T_GOTO T_STRING ';'                                   { $$ = Stmt\Goto_[$2]; }
     | T_STRING ':'                                          { $$ = Stmt\Label[$1]; }
+    | expr error                                            { $$ = $1; }
     | error                                                 { $$ = array(); /* means: no statement */ }
 ;
 
+statement:
+      non_empty_statement                                   { $$ = $1; }
+    | ';'
+          { makeNop($$, $this->startAttributeStack[#1]);
+            if ($$ === null) $$ = array(); /* means: no statement */ }
+;
+
 catches:
       /* empty */                                           { init(); }
     | catches catch                                         { push($1, $2); }
@@ -242,7 +275,8 @@ foreach_statement:
 ;
 
 declare_statement:
-      statement                                             { $$ = toArray($1); }
+      non_empty_statement                                   { $$ = toArray($1); }
+    | ';'                                                   { $$ = null; }
     | ':' inner_statement_list T_ENDDECLARE ';'             { $$ = $2; }
 ;
 
@@ -555,7 +589,10 @@ expr:
     | T_OBJECT_CAST expr                                    { $$ = Expr\Cast\Object_ [$2]; }
     | T_BOOL_CAST expr                                      { $$ = Expr\Cast\Bool_   [$2]; }
     | T_UNSET_CAST expr                                     { $$ = Expr\Cast\Unset_  [$2]; }
-    | T_EXIT exit_expr                                      { $$ = Expr\Exit_        [$2]; }
+    | T_EXIT exit_expr
+          { $attrs = attributes();
+            $attrs['kind'] = strtolower($1) === 'exit' ? Expr\Exit_::KIND_EXIT : Expr\Exit_::KIND_DIE;
+            $$ = new Expr\Exit_($2, $attrs); }
     | '@' expr                                              { $$ = Expr\ErrorSuppress[$2]; }
     | scalar                                                { $$ = $1; }
     | array_expr                                            { $$ = $1; }
@@ -583,14 +620,19 @@ yield_expr:
 ;
 
 array_expr:
-      T_ARRAY '(' array_pair_list ')'                       { $$ = Expr\Array_[$3]; }
-    | '[' array_pair_list ']'                               { $$ = Expr\Array_[$2]; }
+      T_ARRAY '(' array_pair_list ')'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_LONG;
+            $$ = new Expr\Array_($3, $attrs); }
+    | '[' array_pair_list ']'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_SHORT;
+            $$ = new Expr\Array_($2, $attrs); }
 ;
 
 scalar_dereference:
       array_expr '[' dim_offset ']'                         { $$ = Expr\ArrayDimFetch[$1, $3]; }
     | T_CONSTANT_ENCAPSED_STRING '[' dim_offset ']'
-          { $$ = Expr\ArrayDimFetch[Scalar\String_[Scalar\String_::parse($1, false)], $3]; }
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = Expr\ArrayDimFetch[new Scalar\String_(Scalar\String_::parse($1), $attrs), $3]; }
     | constant '[' dim_offset ']'                           { $$ = Expr\ArrayDimFetch[$1, $3]; }
     | scalar_dereference '[' dim_offset ']'                 { $$ = Expr\ArrayDimFetch[$1, $3]; }
     /* alternative array syntax missing intentionally */
@@ -691,7 +733,7 @@ exit_expr:
 backticks_expr:
       /* empty */                                           { $$ = array(); }
     | T_ENCAPSED_AND_WHITESPACE
-          { $$ = array(Scalar\String_::parseEscapeSequences($1, '`', false)); }
+          { $$ = array(Scalar\EncapsedStringPart[Scalar\String_::parseEscapeSequences($1, '`', false)]); }
     | encaps_list                                           { parseEncapsed($1, '`', false); $$ = $1; }
 ;
 
@@ -701,9 +743,11 @@ ctor_arguments:
 ;
 
 common_scalar:
-      T_LNUMBER                                             { $$ = Scalar\LNumber[Scalar\LNumber::parse($1)]; }
+      T_LNUMBER                                             { $$ = Scalar\LNumber::fromString($1, attributes(), true); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
-    | T_CONSTANT_ENCAPSED_STRING                            { $$ = Scalar\String_[Scalar\String_::parse($1, false)]; }
+    | T_CONSTANT_ENCAPSED_STRING
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = new Scalar\String_(Scalar\String_::parse($1, false), $attrs); }
     | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
     | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
     | T_DIR                                                 { $$ = Scalar\MagicConst\Dir[]; }
@@ -713,9 +757,11 @@ common_scalar:
     | T_FUNC_C                                              { $$ = Scalar\MagicConst\Function_[]; }
     | T_NS_C                                                { $$ = Scalar\MagicConst\Namespace_[]; }
     | T_START_HEREDOC T_ENCAPSED_AND_WHITESPACE T_END_HEREDOC
-          { $$ = Scalar\String_[Scalar\String_::parseDocString($1, $2, false)]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_(Scalar\String_::parseDocString($1, $2, false), $attrs); }
     | T_START_HEREDOC T_END_HEREDOC
-          { $$ = Scalar\String_['']; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_('', $attrs); }
 ;
 
 static_scalar:
@@ -773,9 +819,11 @@ scalar:
       common_scalar                                         { $$ = $1; }
     | constant                                              { $$ = $1; }
     | '"' encaps_list '"'
-          { parseEncapsed($2, '"', false); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
+            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
     | T_START_HEREDOC encaps_list T_END_HEREDOC
-          { parseEncapsedDoc($2, false); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            parseEncapsedDoc($2, true); $$ = new Scalar\Encapsed($2, $attrs); }
 ;
 
 static_array_pair_list:
@@ -903,9 +951,13 @@ array_pair:
 
 encaps_list:
       encaps_list encaps_var                                { push($1, $2); }
-    | encaps_list T_ENCAPSED_AND_WHITESPACE                 { push($1, $2); }
+    | encaps_list encaps_string_part                        { push($1, $2); }
     | encaps_var                                            { init($1); }
-    | T_ENCAPSED_AND_WHITESPACE encaps_var                  { init($1, $2); }
+    | encaps_string_part encaps_var                         { init($1, $2); }
+;
+
+encaps_string_part:
+      T_ENCAPSED_AND_WHITESPACE                             { $$ = Scalar\EncapsedStringPart[$1]; }
 ;
 
 encaps_var:

grammar/php7.y

@@ -9,11 +9,17 @@ start:
     top_statement_list                                      { $$ = $this->handleNamespaces($1); }
 ;
 
-top_statement_list:
-      top_statement_list top_statement                      { pushNormalizing($1, $2); }
+top_statement_list_ex:
+      top_statement_list_ex top_statement                   { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+top_statement_list:
+      top_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
 reserved_non_modifiers:
       T_INCLUDE | T_INCLUDE_ONCE | T_EVAL | T_REQUIRE | T_REQUIRE_ONCE | T_LOGICAL_OR | T_LOGICAL_XOR | T_LOGICAL_AND
     | T_INSTANCEOF | T_NEW | T_CLONE | T_EXIT | T_IF | T_ELSEIF | T_ELSE | T_ENDIF | T_ECHO | T_DO | T_WHILE
@@ -21,6 +27,7 @@ reserved_non_modifiers:
     | T_FINALLY | T_THROW | T_USE | T_INSTEADOF | T_GLOBAL | T_VAR | T_UNSET | T_ISSET | T_EMPTY | T_CONTINUE | T_GOTO
     | T_FUNCTION | T_CONST | T_RETURN | T_PRINT | T_YIELD | T_LIST | T_SWITCH | T_ENDSWITCH | T_CASE | T_DEFAULT
     | T_BREAK | T_ARRAY | T_CALLABLE | T_EXTENDS | T_IMPLEMENTS | T_NAMESPACE | T_TRAIT | T_INTERFACE | T_CLASS
+    | T_CLASS_C | T_TRAIT_C | T_FUNC_C | T_METHOD_C | T_LINE | T_FILE | T_DIR | T_NS_C | T_HALT_COMPILER
 ;
 
 semi_reserved:
@@ -53,7 +60,7 @@ top_statement:
     | T_NAMESPACE '{' top_statement_list '}'                { $$ = Stmt\Namespace_[null,     $3]; }
     | T_USE use_declarations ';'                            { $$ = Stmt\Use_[$2, Stmt\Use_::TYPE_NORMAL]; }
     | T_USE use_type use_declarations ';'                   { $$ = Stmt\Use_[$3, $2]; }
-    | group_use_declaration                                 { $$ = $1; }
+    | group_use_declaration ';'                             { $$ = $1; }
     | T_CONST constant_declaration_list ';'                 { $$ = Stmt\Const_[$2]; }
 ;
 
@@ -64,10 +71,20 @@ use_type:
 
 /* Using namespace_name_parts here to avoid s/r conflict on T_NS_SEPARATOR */
 group_use_declaration:
-      T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' use_declarations '}'
+      T_USE use_type namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
           { $$ = Stmt\GroupUse[Name[$3], $6, $2]; }
+    | T_USE use_type T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' unprefixed_use_declarations '}'
+          { $$ = Stmt\GroupUse[Name[$4], $7, $2]; }
     | T_USE namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
           { $$ = Stmt\GroupUse[Name[$2], $5, Stmt\Use_::TYPE_UNKNOWN]; }
+    | T_USE T_NS_SEPARATOR namespace_name_parts T_NS_SEPARATOR '{' inline_use_declarations '}'
+          { $$ = Stmt\GroupUse[Name[$3], $6, Stmt\Use_::TYPE_UNKNOWN]; }
+;
+
+unprefixed_use_declarations:
+      unprefixed_use_declarations ',' unprefixed_use_declaration
+          { push($1, $3); }
+    | unprefixed_use_declaration                            { init($1); }
 ;
 
 use_declarations:
@@ -75,21 +92,24 @@ use_declarations:
     | use_declaration                                       { init($1); }
 ;
 
-use_declaration:
-      namespace_name                                        { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; }
-    | namespace_name T_AS T_STRING                          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; }
-    | T_NS_SEPARATOR namespace_name                         { $$ = Stmt\UseUse[$2, null, Stmt\Use_::TYPE_UNKNOWN]; }
-    | T_NS_SEPARATOR namespace_name T_AS T_STRING           { $$ = Stmt\UseUse[$2, $4, Stmt\Use_::TYPE_UNKNOWN]; }
-;
-
 inline_use_declarations:
       inline_use_declarations ',' inline_use_declaration    { push($1, $3); }
     | inline_use_declaration                                { init($1); }
 ;
 
+unprefixed_use_declaration:
+      namespace_name                                        { $$ = Stmt\UseUse[$1, null, Stmt\Use_::TYPE_UNKNOWN]; }
+    | namespace_name T_AS T_STRING                          { $$ = Stmt\UseUse[$1, $3, Stmt\Use_::TYPE_UNKNOWN]; }
+;
+
+use_declaration:
+      unprefixed_use_declaration                            { $$ = $1; }
+    | T_NS_SEPARATOR unprefixed_use_declaration             { $$ = $2; }
+;
+
 inline_use_declaration:
-      use_declaration                                       { $$ = $1; $$->type = Stmt\Use_::TYPE_NORMAL; }
-    | use_type use_declaration                              { $$ = $2; $$->type = $1; }
+      unprefixed_use_declaration                            { $$ = $1; $$->type = Stmt\Use_::TYPE_NORMAL; }
+    | use_type unprefixed_use_declaration                   { $$ = $2; $$->type = $1; }
 ;
 
 constant_declaration_list:
@@ -110,11 +130,17 @@ class_const:
     identifier '=' expr                                     { $$ = Node\Const_[$1, $3]; }
 ;
 
-inner_statement_list:
-      inner_statement_list inner_statement                  { pushNormalizing($1, $2); }
+inner_statement_list_ex:
+      inner_statement_list_ex inner_statement               { pushNormalizing($1, $2); }
     | /* empty */                                           { init(); }
 ;
 
+inner_statement_list:
+      inner_statement_list_ex
+          { makeNop($nop, $this->lookaheadStartAttributes);
+            if ($nop !== null) { $1[] = $nop; } $$ = $1; }
+;
+
 inner_statement:
       statement                                             { $$ = $1; }
     | function_declaration_statement                        { $$ = $1; }
@@ -123,8 +149,8 @@ inner_statement:
           { throw new Error('__HALT_COMPILER() can only be used from the outermost scope', attributes()); }
 ;
 
-statement:
-      '{' inner_statement_list '}'                          { $$ = $2; }
+non_empty_statement:
+      '{' inner_statement_list '}'                          { $$ = $2; prependLeadingComments($$); }
     | T_IF '(' expr ')' statement elseif_list else_single
           { $$ = Stmt\If_[$3, ['stmts' => toArray($5), 'elseifs' => $6, 'else' => $7]]; }
     | T_IF '(' expr ')' ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
@@ -148,15 +174,22 @@ statement:
     | T_FOREACH '(' expr T_AS variable T_DOUBLE_ARROW foreach_variable ')' foreach_statement
           { $$ = Stmt\Foreach_[$3, $7[0], ['keyVar' => $5, 'byRef' => $7[1], 'stmts' => $9]]; }
     | T_DECLARE '(' declare_list ')' declare_statement      { $$ = Stmt\Declare_[$3, $5]; }
-    | ';'                                                   { $$ = array(); /* means: no statement */ }
     | T_TRY '{' inner_statement_list '}' catches optional_finally
           { $$ = Stmt\TryCatch[$3, $5, $6]; }
     | T_THROW expr ';'                                      { $$ = Stmt\Throw_[$2]; }
     | T_GOTO T_STRING ';'                                   { $$ = Stmt\Goto_[$2]; }
     | T_STRING ':'                                          { $$ = Stmt\Label[$1]; }
+    | expr error                                            { $$ = $1; }
     | error                                                 { $$ = array(); /* means: no statement */ }
 ;
 
+statement:
+      non_empty_statement                                   { $$ = $1; }
+    | ';'
+          { makeNop($$, $this->startAttributeStack[#1]);
+            if ($$ === null) $$ = array(); /* means: no statement */ }
+;
+
 catches:
       /* empty */                                           { init(); }
     | catches catch                                         { push($1, $2); }
@@ -238,7 +271,8 @@ foreach_statement:
 ;
 
 declare_statement:
-      statement                                             { $$ = toArray($1); }
+      non_empty_statement                                   { $$ = toArray($1); }
+    | ';'                                                   { $$ = null; }
     | ':' inner_statement_list T_ENDDECLARE ';'             { $$ = $2; }
 ;
 
@@ -478,7 +512,6 @@ expr:
     | list_expr '=' expr                                    { $$ = Expr\Assign[$1, $3]; }
     | variable '=' expr                                     { $$ = Expr\Assign[$1, $3]; }
     | variable '=' '&' variable                             { $$ = Expr\AssignRef[$1, $4]; }
-    | variable '=' '&' new_expr                             { $$ = Expr\AssignRef[$1, $4]; }
     | new_expr                                              { $$ = $1; }
     | T_CLONE expr                                          { $$ = Expr\Clone_[$2]; }
     | variable T_PLUS_EQUAL expr                            { $$ = Expr\AssignOp\Plus      [$1, $3]; }
@@ -546,7 +579,10 @@ expr:
     | T_OBJECT_CAST expr                                    { $$ = Expr\Cast\Object_ [$2]; }
     | T_BOOL_CAST expr                                      { $$ = Expr\Cast\Bool_   [$2]; }
     | T_UNSET_CAST expr                                     { $$ = Expr\Cast\Unset_  [$2]; }
-    | T_EXIT exit_expr                                      { $$ = Expr\Exit_        [$2]; }
+    | T_EXIT exit_expr
+          { $attrs = attributes();
+            $attrs['kind'] = strtolower($1) === 'exit' ? Expr\Exit_::KIND_EXIT : Expr\Exit_::KIND_DIE;
+            $$ = new Expr\Exit_($2, $attrs); }
     | '@' expr                                              { $$ = Expr\ErrorSuppress[$2]; }
     | scalar                                                { $$ = $1; }
     | '`' backticks_expr '`'                                { $$ = Expr\ShellExec[$2]; }
@@ -622,7 +658,8 @@ exit_expr:
 
 backticks_expr:
       /* empty */                                           { $$ = array(); }
-    | T_ENCAPSED_AND_WHITESPACE                             { $$ = array(Scalar\String_::parseEscapeSequences($1, '`')); }
+    | T_ENCAPSED_AND_WHITESPACE
+          { $$ = array(Scalar\EncapsedStringPart[Scalar\String_::parseEscapeSequences($1, '`')]); }
     | encaps_list                                           { parseEncapsed($1, '`', true); $$ = $1; }
 ;
 
@@ -638,13 +675,19 @@ constant:
 ;
 
 dereferencable_scalar:
-      T_ARRAY '(' array_pair_list ')'                       { $$ = Expr\Array_[$3]; }
-    | '[' array_pair_list ']'                               { $$ = Expr\Array_[$2]; }
-    | T_CONSTANT_ENCAPSED_STRING                            { $$ = Scalar\String_[Scalar\String_::parse($1)]; }
+      T_ARRAY '(' array_pair_list ')'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_LONG;
+            $$ = new Expr\Array_($3, $attrs); }
+    | '[' array_pair_list ']'
+          { $attrs = attributes(); $attrs['kind'] = Expr\Array_::KIND_SHORT;
+            $$ = new Expr\Array_($2, $attrs); }
+    | T_CONSTANT_ENCAPSED_STRING
+          { $attrs = attributes(); $attrs['kind'] = strKind($1);
+            $$ = new Scalar\String_(Scalar\String_::parse($1), $attrs); }
 ;
 
 scalar:
-      T_LNUMBER                                             { $$ = Scalar\LNumber[Scalar\LNumber::parse($1)]; }
+      T_LNUMBER                                             { $$ = Scalar\LNumber::fromString($1, attributes()); }
     | T_DNUMBER                                             { $$ = Scalar\DNumber[Scalar\DNumber::parse($1)]; }
     | T_LINE                                                { $$ = Scalar\MagicConst\Line[]; }
     | T_FILE                                                { $$ = Scalar\MagicConst\File[]; }
@@ -657,13 +700,17 @@ scalar:
     | dereferencable_scalar                                 { $$ = $1; }
     | constant                                              { $$ = $1; }
     | T_START_HEREDOC T_ENCAPSED_AND_WHITESPACE T_END_HEREDOC
-          { $$ = Scalar\String_[Scalar\String_::parseDocString($1, $2)]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_(Scalar\String_::parseDocString($1, $2), $attrs); }
     | T_START_HEREDOC T_END_HEREDOC
-          { $$ = Scalar\String_['']; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            $$ = new Scalar\String_('', $attrs); }
     | '"' encaps_list '"'
-          { parseEncapsed($2, '"', true); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); $attrs['kind'] = Scalar\String_::KIND_DOUBLE_QUOTED;
+            parseEncapsed($2, '"', true); $$ = new Scalar\Encapsed($2, $attrs); }
     | T_START_HEREDOC encaps_list T_END_HEREDOC
-          { parseEncapsedDoc($2, true); $$ = Scalar\Encapsed[$2]; }
+          { $attrs = attributes(); setDocStringAttrs($attrs, $1);
+            parseEncapsedDoc($2, true); $$ = new Scalar\Encapsed($2, $attrs); }
 ;
 
 optional_comma:
@@ -770,9 +817,13 @@ array_pair:
 
 encaps_list:
       encaps_list encaps_var                                { push($1, $2); }
-    | encaps_list T_ENCAPSED_AND_WHITESPACE                 { push($1, $2); }
+    | encaps_list encaps_string_part                        { push($1, $2); }
     | encaps_var                                            { init($1); }
-    | T_ENCAPSED_AND_WHITESPACE encaps_var                  { init($1, $2); }
+    | encaps_string_part encaps_var                         { init($1, $2); }
+;
+
+encaps_string_part:
+      T_ENCAPSED_AND_WHITESPACE                             { $$ = Scalar\EncapsedStringPart[$1]; }
 ;
 
 encaps_var:

grammar/rebuildParsers.php

@@ -85,7 +85,7 @@ foreach ($grammarFileToName as $grammarFile => $name) {
 
 function resolveNodes($code) {
     return preg_replace_callback(
-        '~(?<name>[A-Z][a-zA-Z_\\\\]++)\s*' . PARAMS . '~',
+        '~\b(?<name>[A-Z][a-zA-Z_\\\\]++)\s*' . PARAMS . '~',
         function($matches) {
             // recurse
             $matches['params'] = resolveNodes($matches['params']);
@@ -156,17 +156,52 @@ function resolveMacros($code) {
             if ('parseEncapsed' == $name) {
                 assertArgs(3, $args, $name);
 
-                return 'foreach (' . $args[0] . ' as &$s) { if (is_string($s)) {'
-                     . ' $s = Node\Scalar\String_::parseEscapeSequences($s, ' . $args[1] . ', ' . $args[2] . '); } }';
+                return 'foreach (' . $args[0] . ' as $s) { if ($s instanceof Node\Scalar\EncapsedStringPart) {'
+                     . ' $s->value = Node\Scalar\String_::parseEscapeSequences($s->value, ' . $args[1] . ', ' . $args[2] . '); } }';
             }
 
             if ('parseEncapsedDoc' == $name) {
                 assertArgs(2, $args, $name);
 
-                return 'foreach (' . $args[0] . ' as &$s) { if (is_string($s)) {'
-                     . ' $s = Node\Scalar\String_::parseEscapeSequences($s, null, ' . $args[1] . '); } }'
-                     . ' $s = preg_replace(\'~(\r\n|\n|\r)$~\', \'\', $s);'
-                     . ' if (\'\' === $s) array_pop(' . $args[0] . ');';
+                return 'foreach (' . $args[0] . ' as $s) { if ($s instanceof Node\Scalar\EncapsedStringPart) {'
+                     . ' $s->value = Node\Scalar\String_::parseEscapeSequences($s->value, null, ' . $args[1] . '); } }'
+                     . ' $s->value = preg_replace(\'~(\r\n|\n|\r)\z~\', \'\', $s->value);'
+                     . ' if (\'\' === $s->value) array_pop(' . $args[0] . ');';
+            }
+
+            if ('makeNop' == $name) {
+                assertArgs(2, $args, $name);
+
+                return '$startAttributes = ' . $args[1] . ';'
+                . ' if (isset($startAttributes[\'comments\']))'
+                . ' { ' . $args[0] . ' = new Stmt\Nop([\'comments\' => $startAttributes[\'comments\']]); }'
+                . ' else { ' . $args[0] . ' = null; }';
+            }
+
+            if ('strKind' == $name) {
+                assertArgs(1, $args, $name);
+
+                return '(' . $args[0] . '[0] === "\'" || (' . $args[0] . '[1] === "\'" && '
+                     . '(' . $args[0] . '[0] === \'b\' || ' . $args[0] . '[0] === \'B\')) '
+                     . '? Scalar\String_::KIND_SINGLE_QUOTED : Scalar\String_::KIND_DOUBLE_QUOTED)';
+            }
+
+            if ('setDocStringAttrs' == $name) {
+                assertArgs(2, $args, $name);
+
+                return $args[0] . '[\'kind\'] = strpos(' . $args[1] . ', "\'") === false '
+                     . '? Scalar\String_::KIND_HEREDOC : Scalar\String_::KIND_NOWDOC; '
+                     . 'preg_match(\'/\A[bB]?<<<[ \t]*[\\\'"]?([a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*)[\\\'"]?(?:\r\n|\n|\r)\z/\', ' . $args[1] . ', $matches); '
+                     . $args[0] . '[\'docLabel\'] = $matches[1];';
+            }
+
+            if ('prependLeadingComments' == $name) {
+                assertArgs(1, $args, $name);
+
+                return '$attrs = $this->startAttributeStack[#1]; $stmts = ' . $args[0] . '; '
+                . 'if (!empty($attrs[\'comments\']) && isset($stmts[0])) {'
+                . '$stmts[0]->setAttribute(\'comments\', '
+                . 'array_merge($attrs[\'comments\'], $stmts[0]->getAttribute(\'comments\', []))); }';
             }
 
             return $matches[0];

lib/PhpParser/Autoloader.php

@@ -31,7 +31,7 @@ class Autoloader
      */
     static public function autoload($class) {
         if (0 === strpos($class, 'PhpParser\\')) {
-            $fileName = dirname(__DIR__) . '/' . strtr($class, '\\', '/') . '.php';
+            $fileName = __DIR__ . strtr(substr($class, 9), '\\', '/') . '.php';
             if (file_exists($fileName)) {
                 require $fileName;
             }

lib/PhpParser/Builder/Class_.php

@@ -44,8 +44,7 @@ class Class_ extends Declaration
     /**
      * Implements one or more interfaces.
      *
-     * @param Name|string $interface Name of interface to implement
-     * @param Name|string $...       More interfaces to implement
+     * @param Name|string ...$interfaces Names of interfaces to implement
      *
      * @return $this The builder instance (for fluid interface)
      */

lib/PhpParser/Builder/FunctionLike.php

@@ -10,6 +10,7 @@ abstract class FunctionLike extends Declaration
 {
     protected $returnByRef = false;
     protected $params = array();
+    protected $returnType = null;
 
     /**
      * Make the function return by reference.
@@ -55,4 +56,23 @@ abstract class FunctionLike extends Declaration
 
         return $this;
     }
-}
+
+    /**
+     * Sets the return type for PHP 7.
+     *
+     * @param string|Node\Name $type One of array, callable, string, int, float, bool,
+     *                               or a class/interface name.
+     *
+     * @return $this The builder instance (for fluid interface)
+     */
+    public function setReturnType($type)
+    {
+        if (in_array($type, array('array', 'callable', 'string', 'int', 'float', 'bool'))) {
+            $this->returnType = $type;
+        } else {
+            $this->returnType = $this->normalizeName($type);
+        }
+
+        return $this;
+    }
+}

lib/PhpParser/Builder/Function_.php

@@ -40,9 +40,10 @@ class Function_ extends FunctionLike
      */
     public function getNode() {
         return new Stmt\Function_($this->name, array(
-            'byRef'  => $this->returnByRef,
-            'params' => $this->params,
-            'stmts'  => $this->stmts,
+            'byRef'      => $this->returnByRef,
+            'params'     => $this->params,
+            'returnType' => $this->returnType,
+            'stmts'      => $this->stmts,
         ), $this->attributes);
     }
-}
+}

lib/PhpParser/Builder/Interface_.php

@@ -25,8 +25,7 @@ class Interface_ extends Declaration
     /**
      * Extends one or more interfaces.
      *
-     * @param Name|string $interface Name of interface to extend
-     * @param Name|string $...       More interfaces to extend
+     * @param Name|string ...$interfaces Names of interfaces to extend
      *
      * @return $this The builder instance (for fluid interface)
      */

lib/PhpParser/Builder/Method.php

@@ -116,10 +116,11 @@ class Method extends FunctionLike
      */
     public function getNode() {
         return new Stmt\ClassMethod($this->name, array(
-            'type'   => $this->type,
-            'byRef'  => $this->returnByRef,
-            'params' => $this->params,
-            'stmts'  => $this->stmts,
+            'type'       => $this->type,
+            'byRef'      => $this->returnByRef,
+            'params'     => $this->params,
+            'returnType' => $this->returnType,
+            'stmts'      => $this->stmts,
         ), $this->attributes);
     }
 }

lib/PhpParser/Builder/Param.php

@@ -43,7 +43,7 @@ class Param extends PhpParser\BuilderAbstract
      * @return $this The builder instance (for fluid interface)
      */
     public function setTypeHint($type) {
-        if ($type === 'array' || $type === 'callable') {
+        if (in_array($type, array('array', 'callable', 'string', 'int', 'float', 'bool'))) {
             $this->type = $type;
         } else {
             $this->type = $this->normalizeName($type);
@@ -73,4 +73,4 @@ class Param extends PhpParser\BuilderAbstract
             $this->name, $this->default, $this->type, $this->byRef
         );
     }
-}
+}

lib/PhpParser/BuilderFactory.php

@@ -20,7 +20,7 @@ class BuilderFactory
 {
     /**
      * Creates a namespace builder.
-     * 
+     *
      * @param null|string|Node\Name $name Name of the namespace
      *
      * @return Builder\Namespace_ The created namespace builder

lib/PhpParser/Comment.php

@@ -6,16 +6,19 @@ class Comment
 {
     protected $text;
     protected $line;
+    protected $filePos;
 
     /**
      * Constructs a comment node.
      *
-     * @param string $text Comment text (including comment delimiters like /*)
-     * @param int    $line Line number the comment started on
+     * @param string $text         Comment text (including comment delimiters like /*)
+     * @param int    $startLine    Line number the comment started on
+     * @param int    $startFilePos File offset the comment started on
      */
-    public function __construct($text, $line = -1) {
+    public function __construct($text, $startLine = -1, $startFilePos = -1) {
         $this->text = $text;
-        $this->line = $line;
+        $this->line = $startLine;
+        $this->filePos = $startFilePos;
     }
 
     /**
@@ -31,6 +34,8 @@ class Comment
      * Sets the comment text.
      *
      * @param string $text The comment text (including comment delimiters like /*)
+     *
+     * @deprecated Construct a new comment instead
      */
     public function setText($text) {
         $this->text = $text;
@@ -49,11 +54,22 @@ class Comment
      * Sets the line number the comment started on.
      *
      * @param int $line Line number
+     *
+     * @deprecated Construct a new comment instead
      */
     public function setLine($line) {
         $this->line = $line;
     }
 
+    /**
+     * Gets the file offset the comment started on.
+     *
+     * @return int File offset
+     */
+    public function getFilePos() {
+        return $this->filePos;
+    }
+
     /**
      * Gets the comment text.
      *
@@ -75,7 +91,8 @@ class Comment
      */
     public function getReformattedText() {
         $text = trim($this->text);
-        if (false === strpos($text, "\n")) {
+        $newlinePos = strpos($text, "\n");
+        if (false === $newlinePos) {
             // Single line comments don't need further processing
             return $text;
         } elseif (preg_match('((*BSR_ANYCRLF)(*ANYCRLF)^.*(?:\R\s+\*.*)+$)', $text)) {
@@ -105,15 +122,30 @@ class Comment
             //
             //     /* Some text.
             //        Some more text.
+            //          Indented text.
             //        Even more text. */
             //
-            // is handled by taking the length of the "/* " segment and leaving only that
-            // many space characters before the lines. Thus in the above example only three
-            // space characters are left at the start of every line.
-            return preg_replace('(^\s*(?= {' . strlen($matches[0]) . '}(?!\s)))m', '', $text);
+            // is handled by removing the difference between the shortest whitespace prefix on all
+            // lines and the length of the "/* " opening sequence.
+            $prefixLen = $this->getShortestWhitespacePrefixLen(substr($text, $newlinePos + 1));
+            $removeLen = $prefixLen - strlen($matches[0]);
+            return preg_replace('(^\s{' . $removeLen . '})m', '', $text);
         }
 
         // No idea how to format this comment, so simply return as is
         return $text;
     }
+
+    private function getShortestWhitespacePrefixLen($str) {
+        $lines = explode("\n", $str);
+        $shortestPrefixLen = INF;
+        foreach ($lines as $line) {
+            preg_match('(^\s*)', $line, $matches);
+            $prefixLen = strlen($matches[0]);
+            if ($prefixLen < $shortestPrefixLen) {
+                $shortestPrefixLen = $prefixLen;
+            }
+        }
+        return $shortestPrefixLen;
+    }
 }

lib/PhpParser/Lexer.php

@@ -2,6 +2,7 @@
 
 namespace PhpParser;
 
+use PhpParser\Node\Scalar\LNumber;
 use PhpParser\Parser\Tokens;
 
 class Lexer
@@ -21,10 +22,10 @@ class Lexer
      * Creates a Lexer.
      *
      * @param array $options Options array. Currently only the 'usedAttributes' option is supported,
-     *                       which is an array of attributes to add to the AST nodes. Possible attributes
-     *                       are: 'comments', 'startLine', 'endLine', 'startTokenPos', 'endTokenPos',
-     *                       'startFilePos', 'endFilePos'. The option defaults to the first three.
-     *                       For more info see getNextToken() docs.
+     *                       which is an array of attributes to add to the AST nodes. Possible
+     *                       attributes are: 'comments', 'startLine', 'endLine', 'startTokenPos',
+     *                       'endTokenPos', 'startFilePos', 'endFilePos'. The option defaults to the
+     *                       first three. For more info see getNextToken() docs.
      */
     public function __construct(array $options = array()) {
         // map from internal tokens to PhpParser tokens
@@ -32,7 +33,9 @@ class Lexer
 
         // map of tokens to drop while lexing (the map is only used for isset lookup,
         // that's why the value is simply set to 1; the value is never actually used.)
-        $this->dropTokens = array_fill_keys(array(T_WHITESPACE, T_OPEN_TAG), 1);
+        $this->dropTokens = array_fill_keys(
+            array(T_WHITESPACE, T_OPEN_TAG, T_COMMENT, T_DOC_COMMENT), 1
+        );
 
         // the usedAttributes member is a map of the used attribute names to a dummy
         // value (here "true")
@@ -67,14 +70,21 @@ class Lexer
     }
 
     protected function resetErrors() {
-        // set error_get_last() to defined state by forcing an undefined variable error
-        set_error_handler(function() { return false; }, 0);
-        @$undefinedVariable;
-        restore_error_handler();
+        if (function_exists('error_clear_last')) {
+            error_clear_last();
+        } else {
+            // set error_get_last() to defined state by forcing an undefined variable error
+            set_error_handler(function() { return false; }, 0);
+            @$undefinedVariable;
+            restore_error_handler();
+        }
     }
 
     protected function handleErrors() {
         $error = error_get_last();
+        if (null === $error) {
+            return;
+        }
 
         if (preg_match(
             '~^Unterminated comment starting line ([0-9]+)$~',
@@ -113,7 +123,7 @@ class Lexer
      *  * 'startTokenPos' => Offset into the token array of the first token in the node.
      *  * 'endTokenPos'   => Offset into the token array of the last token in the node.
      *  * 'startFilePos'  => Offset into the code string of the first character that is part of the node.
-     *  * 'endFilePos'    => Offset into the code string of the last character that is part of the node
+     *  * 'endFilePos'    => Offset into the code string of the last character that is part of the node.
      *
      * @param mixed $value           Variable to store token content in
      * @param mixed $startAttributes Variable to store start attributes in
@@ -133,6 +143,9 @@ class Lexer
                 $token = "\0";
             }
 
+            if (isset($this->usedAttributes['startLine'])) {
+                $startAttributes['startLine'] = $this->line;
+            }
             if (isset($this->usedAttributes['startTokenPos'])) {
                 $startAttributes['startTokenPos'] = $this->pos;
             }
@@ -140,63 +153,48 @@ class Lexer
                 $startAttributes['startFilePos'] = $this->filePos;
             }
 
-            if (is_string($token)) {
-                // bug in token_get_all
-                if ('b"' === $token) {
-                    $value = 'b"';
+            if (\is_string($token)) {
+                $value = $token;
+                if (isset($token[1])) {
+                    // bug in token_get_all
                     $this->filePos += 2;
                     $id = ord('"');
                 } else {
-                    $value = $token;
                     $this->filePos += 1;
                     $id = ord($token);
                 }
+            } elseif (!isset($this->dropTokens[$token[0]])) {
+                $value = $token[1];
+                $id = $this->tokenMap[$token[0]];
 
-                if (isset($this->usedAttributes['startLine'])) {
-                    $startAttributes['startLine'] = $this->line;
-                }
-                if (isset($this->usedAttributes['endLine'])) {
-                    $endAttributes['endLine'] = $this->line;
-                }
-                if (isset($this->usedAttributes['endTokenPos'])) {
-                    $endAttributes['endTokenPos'] = $this->pos;
-                }
-                if (isset($this->usedAttributes['endFilePos'])) {
-                    $endAttributes['endFilePos'] = $this->filePos - 1;
-                }
-
-                return $id;
+                $this->line += substr_count($value, "\n");
+                $this->filePos += \strlen($value);
             } else {
-                $this->line += substr_count($token[1], "\n");
-                $this->filePos += strlen($token[1]);
-
-                if (T_COMMENT === $token[0]) {
+                if (T_COMMENT === $token[0] || T_DOC_COMMENT === $token[0]) {
                     if (isset($this->usedAttributes['comments'])) {
-                        $startAttributes['comments'][] = new Comment($token[1], $token[2]);
+                        $comment = T_DOC_COMMENT === $token[0]
+                            ? new Comment\Doc($token[1], $this->line, $this->filePos)
+                            : new Comment($token[1], $this->line, $this->filePos);
+                        $startAttributes['comments'][] = $comment;
                     }
-                } elseif (T_DOC_COMMENT === $token[0]) {
-                    if (isset($this->usedAttributes['comments'])) {
-                        $startAttributes['comments'][] = new Comment\Doc($token[1], $token[2]);
-                    }
-                } elseif (!isset($this->dropTokens[$token[0]])) {
-                    $value = $token[1];
-
-                    if (isset($this->usedAttributes['startLine'])) {
-                        $startAttributes['startLine'] = $token[2];
-                    }
-                    if (isset($this->usedAttributes['endLine'])) {
-                        $endAttributes['endLine'] = $this->line;
-                    }
-                    if (isset($this->usedAttributes['endTokenPos'])) {
-                        $endAttributes['endTokenPos'] = $this->pos;
-                    }
-                    if (isset($this->usedAttributes['endFilePos'])) {
-                        $endAttributes['endFilePos'] = $this->filePos - 1;
-                    }
-
-                    return $this->tokenMap[$token[0]];
                 }
+
+                $this->line += substr_count($token[1], "\n");
+                $this->filePos += \strlen($token[1]);
+                continue;
             }
+
+            if (isset($this->usedAttributes['endLine'])) {
+                $endAttributes['endLine'] = $this->line;
+            }
+            if (isset($this->usedAttributes['endTokenPos'])) {
+                $endAttributes['endTokenPos'] = $this->pos;
+            }
+            if (isset($this->usedAttributes['endFilePos'])) {
+                $endAttributes['endFilePos'] = $this->filePos - 1;
+            }
+
+            return $id;
         }
 
         throw new \RuntimeException('Reached end of lexer loop');

lib/PhpParser/Node/Expr/Array_.php

@@ -6,6 +6,10 @@ use PhpParser\Node\Expr;
 
 class Array_ extends Expr
 {
+    // For use in "kind" attribute
+    const KIND_LONG = 1;  // array() syntax
+    const KIND_SHORT = 2; // [] syntax
+
     /** @var ArrayItem[] Items */
     public $items;
 

lib/PhpParser/Node/Expr/AssignOp.php

@@ -4,10 +4,6 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var  Variable
- * @property Expr $expr Expression
- */
 abstract class AssignOp extends Expr
 {
     /** @var Expr Variable */

lib/PhpParser/Node/Expr/AssignRef.php

@@ -4,10 +4,6 @@ namespace PhpParser\Node\Expr;
 
 use PhpParser\Node\Expr;
 
-/**
- * @property Expr $var  Variable reference is assigned to
- * @property Expr $expr Variable which is referenced
- */
 class AssignRef extends Expr
 {
     /** @var Expr Variable reference is assigned to */

lib/PhpParser/Node/Expr/Exit_.php

@@ -6,6 +6,10 @@ use PhpParser\Node\Expr;
 
 class Exit_ extends Expr
 {
+    /* For use in "kind" attribute */
+    const KIND_EXIT = 1;
+    const KIND_DIE = 2;
+
     /** @var null|Expr Expression */
     public $expr;
 

lib/PhpParser/Node/Scalar/EncapsedStringPart.php

@@ -0,0 +1,26 @@
+<?php
+
+namespace PhpParser\Node\Scalar;
+
+use PhpParser\Node\Scalar;
+
+class EncapsedStringPart extends Scalar
+{
+    /** @var string String value */
+    public $value;
+
+    /**
+     * Constructs a node representing a string part of an encapsed string.
+     *
+     * @param string $value      String value
+     * @param array  $attributes Additional attributes
+     */
+    public function __construct($value, array $attributes = array()) {
+        parent::__construct($attributes);
+        $this->value = $value;
+    }
+
+    public function getSubNodeNames() {
+        return array('value');
+    }
+}

lib/PhpParser/Node/Scalar/LNumber.php

@@ -2,10 +2,17 @@
 
 namespace PhpParser\Node\Scalar;
 
+use PhpParser\Error;
 use PhpParser\Node\Scalar;
 
 class LNumber extends Scalar
 {
+    /* For use in "kind" attribute */
+    const KIND_BIN = 2;
+    const KIND_OCT = 8;
+    const KIND_DEC = 10;
+    const KIND_HEX = 16;
+
     /** @var int Number value */
     public $value;
 
@@ -25,37 +32,36 @@ class LNumber extends Scalar
     }
 
     /**
-     * @internal
+     * Constructs an LNumber node from a string number literal.
      *
-     * Parses an LNUMBER token (dec, hex, oct and bin notations) like PHP would.
+     * @param string $str               String number literal (decimal, octal, hex or binary)
+     * @param array  $attributes        Additional attributes
+     * @param bool   $allowInvalidOctal Whether to allow invalid octal numbers (PHP 5)
      *
-     * @param string $str A string number
-     *
-     * @return int The parsed number
+     * @return LNumber The constructed LNumber, including kind attribute
      */
-    public static function parse($str) {
-        // handle plain 0 specially
-        if ('0' === $str) {
-            return 0;
+    public static function fromString($str, array $attributes = array(), $allowInvalidOctal = false) {
+        if ('0' !== $str[0] || '0' === $str) {
+            $attributes['kind'] = LNumber::KIND_DEC;
+            return new LNumber((int) $str, $attributes);
         }
 
-        // if first char is 0 (and number isn't 0) it's a special syntax
-        if ('0' === $str[0]) {
-            // hex
-            if ('x' === $str[1] || 'X' === $str[1]) {
-                return hexdec($str);
-            }
-
-            // bin
-            if ('b' === $str[1] || 'B' === $str[1]) {
-                return bindec($str);
-            }
-
-            // oct (intval instead of octdec to get proper cutting behavior with malformed numbers)
-            return intval($str, 8);
+        if ('x' === $str[1] || 'X' === $str[1]) {
+            $attributes['kind'] = LNumber::KIND_HEX;
+            return new LNumber(hexdec($str), $attributes);
         }
 
-        // dec
-        return (int) $str;
+        if ('b' === $str[1] || 'B' === $str[1]) {
+            $attributes['kind'] = LNumber::KIND_BIN;
+            return new LNumber(bindec($str), $attributes);
+        }
+
+        if (!$allowInvalidOctal && strpbrk($str, '89')) {
+            throw new Error('Invalid numeric literal', $attributes);
+        }
+
+        // use intval instead of octdec to get proper cutting behavior with malformed numbers
+        $attributes['kind'] = LNumber::KIND_OCT;
+        return new LNumber(intval($str, 8), $attributes);
     }
 }

lib/PhpParser/Node/Scalar/String_.php

@@ -7,6 +7,12 @@ use PhpParser\Node\Scalar;
 
 class String_ extends Scalar
 {
+    /* For use in "kind" attribute */
+    const KIND_SINGLE_QUOTED = 1;
+    const KIND_DOUBLE_QUOTED = 2;
+    const KIND_HEREDOC = 3;
+    const KIND_NOWDOC = 4;
+
     /** @var string String value */
     public $value;
 
@@ -48,7 +54,7 @@ class String_ extends Scalar
      */
     public static function parse($str, $parseUnicodeEscape = true) {
         $bLength = 0;
-        if ('b' === $str[0]) {
+        if ('b' === $str[0] || 'B' === $str[0]) {
             $bLength = 1;
         }
 
@@ -135,7 +141,7 @@ class String_ extends Scalar
      */
     public static function parseDocString($startToken, $str, $parseUnicodeEscape = true) {
         // strip last newline (thanks tokenizer for sticking it into the string!)
-        $str = preg_replace('~(\r\n|\n|\r)$~', '', $str);
+        $str = preg_replace('~(\r\n|\n|\r)\z~', '', $str);
 
         // nowdoc string
         if (false !== strpos($startToken, '\'')) {

lib/PhpParser/Node/Stmt/Declare_.php

@@ -14,10 +14,10 @@ class Declare_ extends Node\Stmt
      * Constructs a declare node.
      *
      * @param DeclareDeclare[] $declares   List of declares
-     * @param Node[]           $stmts      Statements
+     * @param Node[]|null      $stmts      Statements
      * @param array            $attributes Additional attributes
      */
-    public function __construct(array $declares, array $stmts, array $attributes = array()) {
+    public function __construct(array $declares, array $stmts = null, array $attributes = array()) {
         parent::__construct($attributes);
         $this->declares = $declares;
         $this->stmts = $stmts;

lib/PhpParser/Node/Stmt/Nop.php

@@ -0,0 +1,13 @@
+<?php
+
+namespace PhpParser\Node\Stmt;
+
+use PhpParser\Node;
+
+/** Nop/empty statement (;). */
+class Nop extends Node\Stmt
+{
+    public function getSubNodeNames() {
+        return array();
+    }
+}

lib/PhpParser/NodeDumper.php

@@ -4,6 +4,18 @@ namespace PhpParser;
 
 class NodeDumper
 {
+    private $dumpComments;
+
+    /**
+     * Constructs a NodeDumper.
+     *
+     * @param array $options Boolean option 'dumpComments' controls whether comments should be
+     *                       dumped
+     */
+    public function __construct(array $options = []) {
+        $this->dumpComments = !empty($options['dumpComments']);
+    }
+
     /**
      * Dumps a node or array.
      *
@@ -31,6 +43,10 @@ class NodeDumper
                     $r .= str_replace("\n", "\n    ", $this->dump($value));
                 }
             }
+
+            if ($this->dumpComments && $comments = $node->getAttribute('comments')) {
+                $r .= "\n    comments: " . str_replace("\n", "\n    ", $this->dump($comments));
+            }
         } elseif (is_array($node)) {
             $r = 'array(';
 
@@ -49,6 +65,8 @@ class NodeDumper
                     $r .= str_replace("\n", "\n    ", $this->dump($value));
                 }
             }
+        } elseif ($node instanceof Comment) {
+            return $node->getReformattedText();
         } else {
             throw new \InvalidArgumentException('Can only dump nodes and arrays.');
         }

lib/PhpParser/NodeTraverser.php

@@ -99,6 +99,12 @@ class NodeTraverser implements NodeTraverserInterface
 
                 foreach ($this->visitors as $visitor) {
                     if (null !== $return = $visitor->leaveNode($subNode)) {
+                        if (is_array($return)) {
+                            throw new \LogicException(
+                                'leaveNode() may only return an array ' .
+                                'if the parent structure is an array'
+                            );
+                        }
                         $subNode = $return;
                     }
                 }

lib/PhpParser/ParserAbstract.php

@@ -88,6 +88,8 @@ abstract class ParserAbstract implements Parser
     protected $startAttributeStack;
     /** @var array End attributes of last *shifted* token */
     protected $endAttributes;
+    /** @var array Start attributes of last *read* token */
+    protected $lookaheadStartAttributes;
 
     /** @var bool Whether to throw on first error */
     protected $throwOnError;
@@ -185,6 +187,7 @@ abstract class ParserAbstract implements Parser
                     // This is necessary to assign some meaningful attributes to /* empty */ productions. They'll get
                     // the attributes of the next token, even though they don't contain it themselves.
                     $this->startAttributeStack[$this->stackPos+1] = $startAttributes;
+                    $this->lookaheadStartAttributes = $startAttributes;
 
                     //$this->traceRead($symbol);
                 }
@@ -348,7 +351,9 @@ abstract class ParserAbstract implements Parser
                 && ($idx = $this->actionBase[$state + $this->YYNLSTATES] + $symbol) >= 0
                 && $idx < $this->actionTableSize && $this->actionCheck[$idx] === $symbol
             ) {
-                if ($this->action[$idx] != $this->unexpectedTokenRule) {
+                if ($this->action[$idx] != $this->unexpectedTokenRule
+                    && $this->action[$idx] != $this->defaultAction
+                ) {
                     if (count($expected) == 4) {
                         /* Too many expected tokens */
                         return array();
@@ -446,7 +451,7 @@ abstract class ParserAbstract implements Parser
     private function getNamespacingStyle(array $stmts) {
         $style = null;
         $hasNotAllowedStmts = false;
-        foreach ($stmts as $stmt) {
+        foreach ($stmts as $i => $stmt) {
             if ($stmt instanceof Node\Stmt\Namespace_) {
                 $currentStyle = null === $stmt->stmts ? 'semicolon' : 'brace';
                 if (null === $style) {
@@ -457,9 +462,23 @@ abstract class ParserAbstract implements Parser
                 } elseif ($style !== $currentStyle) {
                     throw new Error('Cannot mix bracketed namespace declarations with unbracketed namespace declarations', $stmt->getLine());
                 }
-            } elseif (!$stmt instanceof Node\Stmt\Declare_ && !$stmt instanceof Node\Stmt\HaltCompiler) {
-                $hasNotAllowedStmts = true;
+                continue;
             }
+
+            /* declare(), __halt_compiler() and nops can be used before a namespace declaration */
+            if ($stmt instanceof Node\Stmt\Declare_
+                || $stmt instanceof Node\Stmt\HaltCompiler
+                || $stmt instanceof Node\Stmt\Nop) {
+                continue;
+            }
+
+            /* There may be a hashbang line at the very start of the file */
+            if ($i == 0 && $stmt instanceof Node\Stmt\InlineHTML && preg_match('/\A#!.*\r?\n\z/', $stmt->value)) {
+                continue;
+            }
+
+            /* Everything else if forbidden before namespace declarations */
+            $hasNotAllowedStmts = true;
         }
         return $style;
     }

lib/PhpParser/PrettyPrinter/Standard.php

@@ -84,15 +84,71 @@ class Standard extends PrettyPrinterAbstract
     // Scalars
 
     public function pScalar_String(Scalar\String_ $node) {
-        return '\'' . $this->pNoIndent(addcslashes($node->value, '\'\\')) . '\'';
+        $kind = $node->getAttribute('kind', Scalar\String_::KIND_SINGLE_QUOTED);
+        switch ($kind) {
+            case Scalar\String_::KIND_NOWDOC:
+                $label = $node->getAttribute('docLabel');
+                if ($label && !$this->containsEndLabel($node->value, $label)) {
+                    if ($node->value === '') {
+                        return $this->pNoIndent("<<<'$label'\n$label") . $this->docStringEndToken;
+                    }
+
+                    return $this->pNoIndent("<<<'$label'\n$node->value\n$label")
+                         . $this->docStringEndToken;
+                }
+                /* break missing intentionally */
+            case Scalar\String_::KIND_SINGLE_QUOTED:
+                return '\'' . $this->pNoIndent(addcslashes($node->value, '\'\\')) . '\'';
+            case Scalar\String_::KIND_HEREDOC:
+                $label = $node->getAttribute('docLabel');
+                if ($label && !$this->containsEndLabel($node->value, $label)) {
+                    if ($node->value === '') {
+                        return $this->pNoIndent("<<<$label\n$label") . $this->docStringEndToken;
+                    }
+
+                    $escaped = $this->escapeString($node->value, null);
+                    return $this->pNoIndent("<<<$label\n" . $escaped ."\n$label")
+                         . $this->docStringEndToken;
+                }
+            /* break missing intentionally */
+            case Scalar\String_::KIND_DOUBLE_QUOTED:
+                return '"' . $this->escapeString($node->value, '"') . '"';
+        }
+        throw new \Exception('Invalid string kind');
     }
 
     public function pScalar_Encapsed(Scalar\Encapsed $node) {
+        if ($node->getAttribute('kind') === Scalar\String_::KIND_HEREDOC) {
+            $label = $node->getAttribute('docLabel');
+            if ($label && !$this->encapsedContainsEndLabel($node->parts, $label)) {
+                if (count($node->parts) === 1
+                    && $node->parts[0] instanceof Scalar\EncapsedStringPart
+                    && $node->parts[0]->value === ''
+                ) {
+                    return $this->pNoIndent("<<<$label\n$label") . $this->docStringEndToken;
+                }
+
+                return $this->pNoIndent(
+                    "<<<$label\n" . $this->pEncapsList($node->parts, null) . "\n$label"
+                ) . $this->docStringEndToken;
+            }
+        }
         return '"' . $this->pEncapsList($node->parts, '"') . '"';
     }
 
     public function pScalar_LNumber(Scalar\LNumber $node) {
-        return (string) $node->value;
+        $str = (string) $node->value;
+        switch ($node->getAttribute('kind', Scalar\LNumber::KIND_DEC)) {
+            case Scalar\LNumber::KIND_BIN:
+                return '0b' . base_convert($str, 10, 2);
+            case Scalar\LNumber::KIND_OCT:
+                return '0' . base_convert($str, 10, 8);
+            case Scalar\LNumber::KIND_DEC:
+                return $str;
+            case Scalar\LNumber::KIND_HEX:
+                return '0x' . base_convert($str, 10, 16);
+        }
+        throw new \Exception('Invalid number kind');
     }
 
     public function pScalar_DNumber(Scalar\DNumber $node) {
@@ -422,7 +478,13 @@ class Standard extends PrettyPrinterAbstract
     }
 
     public function pExpr_Array(Expr\Array_ $node) {
-        return 'array(' . $this->pCommaSeparated($node->items) . ')';
+        $syntax = $node->getAttribute('kind',
+            $this->options['shortArraySyntax'] ? Expr\Array_::KIND_SHORT : Expr\Array_::KIND_LONG);
+        if ($syntax === Expr\Array_::KIND_SHORT) {
+            return '[' . $this->pCommaSeparated($node->items) . ']';
+        } else {
+            return 'array(' . $this->pCommaSeparated($node->items) . ')';
+        }
     }
 
     public function pExpr_ArrayItem(Expr\ArrayItem $node) {
@@ -489,7 +551,9 @@ class Standard extends PrettyPrinterAbstract
     }
 
     public function pExpr_Exit(Expr\Exit_ $node) {
-        return 'die' . (null !== $node->expr ? '(' . $this->p($node->expr) . ')' : '');
+        $kind = $node->getAttribute('kind', Expr\Exit_::KIND_DIE);
+        return ($kind === Expr\Exit_::KIND_EXIT ? 'exit' : 'die')
+             . (null !== $node->expr ? '(' . $this->p($node->expr) . ')' : '');
     }
 
     public function pExpr_Yield(Expr\Yield_ $node) {
@@ -605,12 +669,12 @@ class Standard extends PrettyPrinterAbstract
     }
 
     public function pStmt_Declare(Stmt\Declare_ $node) {
-        return 'declare (' . $this->pCommaSeparated($node->declares) . ') {'
-             . $this->pStmts($node->stmts) . "\n" . '}';
+        return 'declare (' . $this->pCommaSeparated($node->declares) . ')'
+             . (null !== $node->stmts ? ' {' . $this->pStmts($node->stmts) . "\n" . '}' : ';');
     }
 
     public function pStmt_DeclareDeclare(Stmt\DeclareDeclare $node) {
-        return $node->key . ' = ' . $this->p($node->value);
+        return $node->key . '=' . $this->p($node->value);
     }
 
     // Control flow
@@ -734,6 +798,10 @@ class Standard extends PrettyPrinterAbstract
         return '__halt_compiler();' . $node->remaining;
     }
 
+    public function pStmt_Nop(Stmt\Nop $node) {
+        return '';
+    }
+
     // Helpers
 
     protected function pType($node) {
@@ -768,8 +836,8 @@ class Standard extends PrettyPrinterAbstract
     protected function pEncapsList(array $encapsList, $quote) {
         $return = '';
         foreach ($encapsList as $element) {
-            if (is_string($element)) {
-                $return .= addcslashes($element, "\n\r\t\f\v$" . $quote . "\\");
+            if ($element instanceof Scalar\EncapsedStringPart) {
+                $return .= $this->escapeString($element->value, $quote);
             } else {
                 $return .= '{' . $this->p($element) . '}';
             }
@@ -778,6 +846,45 @@ class Standard extends PrettyPrinterAbstract
         return $return;
     }
 
+    protected function escapeString($string, $quote) {
+        if (null === $quote) {
+            // For doc strings, don't escape newlines
+            $escaped = addcslashes($string, "\t\f\v$\\");
+        } else {
+            $escaped = addcslashes($string, "\n\r\t\f\v$" . $quote . "\\");
+        }
+
+        // Escape other control characters
+        return preg_replace_callback('/([\0-\10\16-\37])(?=([0-7]?))/', function ($matches) {
+            $oct = decoct(ord($matches[1]));
+            if ($matches[2] !== '') {
+                // If there is a trailing digit, use the full three character form
+                return '\\' . str_pad($oct, 3, '0', STR_PAD_LEFT);
+            }
+            return '\\' . $oct;
+        }, $escaped);
+    }
+
+    protected function containsEndLabel($string, $label, $atStart = true, $atEnd = true) {
+        $start = $atStart ? '(?:^|[\r\n])' : '[\r\n]';
+        $end = $atEnd ? '(?:$|[;\r\n])' : '[;\r\n]';
+        return false !== strpos($string, $label)
+            && preg_match('/' . $start . $label . $end . '/', $string);
+    }
+
+    protected function encapsedContainsEndLabel(array $parts, $label) {
+        foreach ($parts as $i => $part) {
+            $atStart = $i === 0;
+            $atEnd = $i === count($parts) - 1;
+            if ($part instanceof Scalar\EncapsedStringPart
+                && $this->containsEndLabel($part->value, $label, $atStart, $atEnd)
+            ) {
+                return true;
+            }
+        }
+        return false;
+    }
+
     protected function pDereferenceLhs(Node $node) {
         if ($node instanceof Expr\Variable
             || $node instanceof Name

lib/PhpParser/PrettyPrinterAbstract.php

@@ -75,10 +75,25 @@ abstract class PrettyPrinterAbstract
     );
 
     protected $noIndentToken;
+    protected $docStringEndToken;
     protected $canUseSemicolonNamespaces;
+    protected $options;
 
-    public function __construct() {
+    /**
+     * Creates a pretty printer instance using the given options.
+     *
+     * Supported options:
+     *  * bool $shortArraySyntax = false: Whether to use [] instead of array() as the default array
+     *                                    syntax, if the node does not specify a format.
+     *
+     * @param array $options Dictionary of formatting options
+     */
+    public function __construct(array $options = []) {
         $this->noIndentToken = '_NO_INDENT_' . mt_rand();
+        $this->docStringEndToken = '_DOC_STRING_END_' . mt_rand();
+
+        $defaultOptions = ['shortArraySyntax' => false];
+        $this->options = $options + $defaultOptions;
     }
 
     /**
@@ -91,7 +106,7 @@ abstract class PrettyPrinterAbstract
     public function prettyPrint(array $stmts) {
         $this->preprocessNodes($stmts);
 
-        return ltrim(str_replace("\n" . $this->noIndentToken, "\n", $this->pStmts($stmts, false)));
+        return ltrim($this->handleMagicTokens($this->pStmts($stmts, false)));
     }
 
     /**
@@ -102,7 +117,7 @@ abstract class PrettyPrinterAbstract
      * @return string Pretty printed node
      */
     public function prettyPrintExpr(Expr $node) {
-        return str_replace("\n" . $this->noIndentToken, "\n", $this->p($node));
+        return $this->handleMagicTokens($this->p($node));
     }
 
     /**
@@ -113,13 +128,17 @@ abstract class PrettyPrinterAbstract
      * @return string Pretty printed statements
      */
     public function prettyPrintFile(array $stmts) {
-        $p = rtrim($this->prettyPrint($stmts));
+        if (!$stmts) {
+            return "<?php\n\n";
+        }
 
-        $p = preg_replace('/^\?>\n?/', '', $p, -1, $count);
-        $p = preg_replace('/<\?php$/', '', $p);
+        $p = "<?php\n\n" . $this->prettyPrint($stmts);
 
-        if (!$count) {
-            $p = "<?php\n\n" . $p;
+        if ($stmts[0] instanceof Stmt\InlineHTML) {
+            $p = preg_replace('/^<\?php\s+\?>\n?/', '', $p);
+        }
+        if ($stmts[count($stmts) - 1] instanceof Stmt\InlineHTML) {
+            $p = preg_replace('/<\?php$/', '', rtrim($p));
         }
 
         return $p;
@@ -140,6 +159,17 @@ abstract class PrettyPrinterAbstract
         }
     }
 
+    protected function handleMagicTokens($str) {
+        // Drop no-indent tokens
+        $str = str_replace($this->noIndentToken, '', $str);
+
+        // Replace doc-string-end tokens with nothing or a newline
+        $str = str_replace($this->docStringEndToken . ";\n", ";\n", $str);
+        $str = str_replace($this->docStringEndToken, "\n", $str);
+
+        return $str;
+    }
+
     /**
      * Pretty prints an array of nodes (statements) and indents them optionally.
      *
@@ -151,10 +181,15 @@ abstract class PrettyPrinterAbstract
     protected function pStmts(array $nodes, $indent = true) {
         $result = '';
         foreach ($nodes as $node) {
-            $result .= "\n"
-                    . $this->pComments($node->getAttribute('comments', array()))
-                    . $this->p($node)
-                    . ($node instanceof Expr ? ';' : '');
+            $comments = $node->getAttribute('comments', array());
+            if ($comments) {
+                $result .= "\n" . $this->pComments($comments);
+                if ($node instanceof Stmt\Nop) {
+                    continue;
+                }
+            }
+
+            $result .= "\n" . $this->p($node) . ($node instanceof Expr ? ';' : '');
         }
 
         if ($indent) {
@@ -252,19 +287,26 @@ abstract class PrettyPrinterAbstract
      *
      * @param string $string Not to be indented string
      *
-     * @return mixed String marked with $this->noIndentToken's.
+     * @return string String marked with $this->noIndentToken's.
      */
     protected function pNoIndent($string) {
         return str_replace("\n", "\n" . $this->noIndentToken, $string);
     }
 
+    /**
+     * Prints reformatted text of the passed comments.
+     *
+     * @param Comment[] $comments List of comments
+     *
+     * @return string Reformatted text of comments
+     */
     protected function pComments(array $comments) {
-        $result = '';
+        $formattedComments = [];
 
         foreach ($comments as $comment) {
-            $result .= $comment->getReformattedText() . "\n";
+            $formattedComments[] = $comment->getReformattedText();
         }
 
-        return $result;
+        return implode("\n", $formattedComments);
     }
 }

test/PhpParser/Builder/FunctionTest.php

@@ -76,6 +76,16 @@ class FunctionTest extends \PHPUnit_Framework_TestCase
         )), $node);
     }
 
+    public function testReturnType() {
+        $node = $this->createFunctionBuilder('test')
+            ->setReturnType('bool')
+            ->getNode();
+
+        $this->assertEquals(new Stmt\Function_('test', array(
+            'returnType' => 'bool'
+        ), array()), $node);
+    }
+
     /**
      * @expectedException \LogicException
      * @expectedExceptionMessage Expected parameter node, got "Name"

test/PhpParser/Builder/MethodTest.php

@@ -120,6 +120,15 @@ class MethodTest extends \PHPUnit_Framework_TestCase
         )), $node);
     }
 
+    public function testReturnType() {
+        $node = $this->createMethodBuilder('test')
+            ->setReturnType('bool')
+            ->getNode();
+        $this->assertEquals(new Stmt\ClassMethod('test', array(
+            'returnType' => 'bool'
+        ), array()), $node);
+    }
+
     /**
      * @expectedException \LogicException
      * @expectedExceptionMessage Cannot add statements to an abstract method

test/PhpParser/CodeParsingTest.php

@@ -13,7 +13,7 @@ class CodeParsingTest extends CodeTestAbstract
      */
     public function testParse($name, $code, $expected, $mode) {
         $lexer = new Lexer\Emulative(array('usedAttributes' => array(
-            'startLine', 'endLine', 'startFilePos', 'endFilePos'
+            'startLine', 'endLine', 'startFilePos', 'endFilePos', 'comments'
         )));
         $parser5 = new Parser\Php5($lexer, array(
             'throwOnError' => false,
@@ -47,7 +47,7 @@ class CodeParsingTest extends CodeTestAbstract
         }
 
         if (null !== $stmts) {
-            $dumper = new NodeDumper;
+            $dumper = new NodeDumper(['dumpComments' => true]);
             $output .= $dumper->dump($stmts);
         }
 

test/PhpParser/CodeTestAbstract.php

@@ -5,14 +5,16 @@ namespace PhpParser;
 abstract class CodeTestAbstract extends \PHPUnit_Framework_TestCase
 {
     protected function getTests($directory, $fileExtension) {
+        $directory = realpath($directory);
         $it = new \RecursiveDirectoryIterator($directory);
         $it = new \RecursiveIteratorIterator($it, \RecursiveIteratorIterator::LEAVES_ONLY);
         $it = new \RegexIterator($it, '(\.' . preg_quote($fileExtension) . '$)');
 
         $tests = array();
         foreach ($it as $file) {
-            $fileName = realpath($file->getPathname());
+            $fileName = $file->getPathname();
             $fileContents = file_get_contents($fileName);
+            $fileContents = canonicalize($fileContents);
 
             // evaluate @@{expr}@@ expressions
             $fileContents = preg_replace_callback(
@@ -24,15 +26,18 @@ abstract class CodeTestAbstract extends \PHPUnit_Framework_TestCase
             );
 
             // parse sections
-            $parts = array_map('trim', explode('-----', $fileContents));
+            $parts = preg_split("/\n-----(?:\n|$)/", $fileContents);
 
             // first part is the name
             $name = array_shift($parts) . ' (' . $fileName . ')';
+            $shortName = ltrim(str_replace($directory, '', $fileName), '/\\');
 
             // multiple sections possible with always two forming a pair
-            foreach (array_chunk($parts, 2) as $chunk) {
-                list($expected, $mode) = $this->extractMode(canonicalize($chunk[1]));
-                $tests[] = array($name, $chunk[0], $expected, $mode);
+            $chunks = array_chunk($parts, 2);
+            foreach ($chunks as $i => $chunk) {
+                $dataSetName = $shortName . (count($chunks) > 1 ? '#' . $i : '');
+                list($expected, $mode) = $this->extractMode($chunk[1]);
+                $tests[$dataSetName] = array($name, $chunk[0], $expected, $mode);
             }
         }
 

test/PhpParser/CommentTest.php

@@ -5,11 +5,12 @@ namespace PhpParser;
 class CommentTest extends \PHPUnit_Framework_TestCase
 {
     public function testGetSet() {
-        $comment = new Comment('/* Some comment */', 1);
+        $comment = new Comment('/* Some comment */', 1, 10);
 
         $this->assertSame('/* Some comment */', $comment->getText());
         $this->assertSame('/* Some comment */', (string) $comment);
         $this->assertSame(1, $comment->getLine());
+        $this->assertSame(10, $comment->getFilePos());
 
         $comment->setText('/* Some other comment */');
         $comment->setLine(10);
@@ -58,6 +59,14 @@ class CommentTest extends \PHPUnit_Framework_TestCase
                 '/* Some text.
    More text.
    Even more text. */'
+            ),
+            array(
+                '/* Some text.
+       More text.
+         Indented text. */',
+                '/* Some text.
+   More text.
+     Indented text. */',
             ),
             // invalid comment -> no reformatting
             array(

test/PhpParser/LexerTest.php

@@ -93,7 +93,9 @@ class LexerTest extends \PHPUnit_Framework_TestCase
                         ord('$'), '$',
                         array(
                             'startLine' => 3,
-                            'comments' => array(new Comment\Doc('/** doc' . "\n" . 'comment */', 2))
+                            'comments' => array(
+                                new Comment\Doc('/** doc' . "\n" . 'comment */', 2, 14),
+                            )
                         ),
                         array('endLine' => 3)
                     ),
@@ -109,10 +111,10 @@ class LexerTest extends \PHPUnit_Framework_TestCase
                         array(
                             'startLine' => 2,
                             'comments' => array(
-                                new Comment('/* comment */', 1),
-                                new Comment('// comment' . "\n", 1),
-                                new Comment\Doc('/** docComment 1 */', 2),
-                                new Comment\Doc('/** docComment 2 */', 2),
+                                new Comment('/* comment */', 1, 6),
+                                new Comment('// comment' . "\n", 1, 20),
+                                new Comment\Doc('/** docComment 1 */', 2, 31),
+                                new Comment\Doc('/** docComment 2 */', 2, 50),
                             ),
                         ),
                         array('endLine' => 2)

test/PhpParser/NodeTraverserTest.php

@@ -200,4 +200,19 @@ class NodeTraverserTest extends \PHPUnit_Framework_TestCase
 
         $this->assertSame($stmts, $traverser->traverse($stmts));
     }
+
+    /**
+     * @expectedException \LogicException
+     * @expectedExceptionMessage leaveNode() may only return an array if the parent structure is an array
+     */
+    public function testReplaceByArrayOnlyAllowedIfParentIsArray() {
+        $stmts = array(new Node\Expr\UnaryMinus(new Node\Scalar\LNumber(42)));
+
+        $visitor = $this->getMock('PhpParser\NodeVisitor');
+        $visitor->method('leaveNode')->willReturn(array(new Node\Scalar\DNumber(42.0)));
+
+        $traverser = new NodeTraverser();
+        $traverser->addVisitor($visitor);
+        $traverser->traverse($stmts);
+    }
 }

test/PhpParser/Parser/MultipleTest.php

@@ -61,7 +61,7 @@ class MultipleTest extends ParserTest {
                 $this->getPrefer5(),
                 [
                     new Expr\Variable(
-                        new Expr\ArrayDimFetch(new Expr\Variable('a'), new LNumber(0))
+                        new Expr\ArrayDimFetch(new Expr\Variable('a'), LNumber::fromString('0'))
                     )
                 ]
             ],
@@ -71,7 +71,7 @@ class MultipleTest extends ParserTest {
                 $this->getPrefer7(),
                 [
                     new Expr\ArrayDimFetch(
-                        new Expr\Variable(new Expr\Variable('a')), new LNumber(0)
+                        new Expr\Variable(new Expr\Variable('a')), LNumber::fromString('0')
                     )
                 ]
             ],

test/PhpParser/ParserTest.php

@@ -3,6 +3,9 @@
 namespace PhpParser;
 
 use PhpParser\Comment;
+use PhpParser\Node\Expr;
+use PhpParser\Node\Scalar;
+use PhpParser\Node\Scalar\String_;
 
 abstract class ParserTest extends \PHPUnit_Framework_TestCase
 {
@@ -54,7 +57,7 @@ EOC;
         $this->assertInstanceOf('PhpParser\Node\Stmt\Function_', $fn);
         $this->assertEquals(array(
             'comments' => array(
-                new Comment\Doc('/** Doc comment */', 2),
+                new Comment\Doc('/** Doc comment */', 2, 6),
             ),
             'startLine' => 3,
             'endLine' => 7,
@@ -76,8 +79,8 @@ EOC;
         $this->assertInstanceOf('PhpParser\Node\Stmt\Echo_', $echo);
         $this->assertEquals(array(
             'comments' => array(
-                new Comment("// Line\n", 4),
-                new Comment("// Comments\n", 5),
+                new Comment("// Line\n", 4, 49),
+                new Comment("// Comments\n", 5, 61),
             ),
             'startLine' => 6,
             'endLine' => 6,
@@ -105,6 +108,58 @@ EOC;
         $parser = $this->getParser($lexer);
         $parser->parse('dummy');
     }
+
+    /**
+     * @dataProvider provideTestKindAttributes
+     */
+    public function testKindAttributes($code, $expectedAttributes) {
+        $parser = $this->getParser(new Lexer);
+        $stmts = $parser->parse("<?php $code;");
+        $attributes = $stmts[0]->getAttributes();
+        foreach ($expectedAttributes as $name => $value) {
+            $this->assertSame($value, $attributes[$name]);
+        }
+    }
+
+    public function provideTestKindAttributes() {
+        return array(
+            array('0', ['kind' => Scalar\LNumber::KIND_DEC]),
+            array('9', ['kind' => Scalar\LNumber::KIND_DEC]),
+            array('07', ['kind' => Scalar\LNumber::KIND_OCT]),
+            array('0xf', ['kind' => Scalar\LNumber::KIND_HEX]),
+            array('0XF', ['kind' => Scalar\LNumber::KIND_HEX]),
+            array('0b1', ['kind' => Scalar\LNumber::KIND_BIN]),
+            array('0B1', ['kind' => Scalar\LNumber::KIND_BIN]),
+            array('[]', ['kind' => Expr\Array_::KIND_SHORT]),
+            array('array()', ['kind' => Expr\Array_::KIND_LONG]),
+            array("'foo'", ['kind' => String_::KIND_SINGLE_QUOTED]),
+            array("b'foo'", ['kind' => String_::KIND_SINGLE_QUOTED]),
+            array("B'foo'", ['kind' => String_::KIND_SINGLE_QUOTED]),
+            array('"foo"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('b"foo"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('B"foo"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('"foo$bar"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('b"foo$bar"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array('B"foo$bar"', ['kind' => String_::KIND_DOUBLE_QUOTED]),
+            array("<<<'STR'\nSTR\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']),
+            array("<<<STR\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("<<<\"STR\"\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("b<<<'STR'\nSTR\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']),
+            array("B<<<'STR'\nSTR\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']),
+            array("<<< \t 'STR'\nSTR\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']),
+            // HHVM doesn't support this due to a lexer bug
+            // (https://github.com/facebook/hhvm/issues/6970)
+            // array("<<<'\xff'\n\xff\n", ['kind' => String_::KIND_NOWDOC, 'docLabel' => "\xff"]),
+            array("<<<\"STR\"\n\$a\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("b<<<\"STR\"\n\$a\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("B<<<\"STR\"\n\$a\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("<<< \t \"STR\"\n\$a\nSTR\n", ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR']),
+            array("die", ['kind' => Expr\Exit_::KIND_DIE]),
+            array("die('done')", ['kind' => Expr\Exit_::KIND_DIE]),
+            array("exit", ['kind' => Expr\Exit_::KIND_EXIT]),
+            array("exit(1)", ['kind' => Expr\Exit_::KIND_EXIT]),
+        );
+    }
 }
 
 class InvalidTokenLexer extends Lexer {

test/PhpParser/PrettyPrinterTest.php

@@ -2,7 +2,10 @@
 
 namespace PhpParser;
 
+use PhpParser\Comment;
 use PhpParser\Node\Expr;
+use PhpParser\Node\Scalar\Encapsed;
+use PhpParser\Node\Scalar\EncapsedStringPart;
 use PhpParser\Node\Scalar\String_;
 use PhpParser\Node\Stmt;
 use PhpParser\PrettyPrinter\Standard;
@@ -11,30 +14,36 @@ require_once __DIR__ . '/CodeTestAbstract.php';
 
 class PrettyPrinterTest extends CodeTestAbstract
 {
-    protected function doTestPrettyPrintMethod($method, $name, $code, $expected, $mode) {
+    protected function doTestPrettyPrintMethod($method, $name, $code, $expected, $modeLine) {
         $lexer = new Lexer\Emulative;
         $parser5 = new Parser\Php5($lexer);
         $parser7 = new Parser\Php7($lexer);
-        $prettyPrinter = new Standard;
+
+        list($version, $options) = $this->parseModeLine($modeLine);
+        $prettyPrinter = new Standard($options);
 
         try {
             $output5 = canonicalize($prettyPrinter->$method($parser5->parse($code)));
         } catch (Error $e) {
             $output5 = null;
-            $this->assertEquals('php7', $mode);
+            if ('php7' !== $version) {
+                throw $e;
+            }
         }
 
         try {
             $output7 = canonicalize($prettyPrinter->$method($parser7->parse($code)));
         } catch (Error $e) {
             $output7 = null;
-            $this->assertEquals('php5', $mode);
+            if ('php5' !== $version) {
+                throw $e;
+            }
         }
 
-        if ('php5' === $mode) {
+        if ('php5' === $version) {
             $this->assertSame($expected, $output5, $name);
             $this->assertNotSame($expected, $output7, $name);
-        } else if ('php7' === $mode) {
+        } else if ('php7' === $version) {
             $this->assertSame($expected, $output7, $name);
             $this->assertNotSame($expected, $output5, $name);
         } else {
@@ -80,4 +89,76 @@ class PrettyPrinterTest extends CodeTestAbstract
         ));
         $this->assertEquals("function () {\n    return 'a\nb';\n}", $prettyPrinter->prettyPrintExpr($expr));
     }
+
+    public function testCommentBeforeInlineHTML() {
+        $prettyPrinter = new PrettyPrinter\Standard;
+        $comment = new Comment\Doc("/**\n * This is a comment\n */");
+        $stmts = [new Stmt\InlineHTML('Hello World!', ['comments' => [$comment]])];
+        $expected = "<?php\n\n/**\n * This is a comment\n */\n?>\nHello World!";
+        $this->assertSame($expected, $prettyPrinter->prettyPrintFile($stmts));
+    }
+
+    private function parseModeLine($modeLine) {
+        $parts = explode(' ', $modeLine, 2);
+        $version = isset($parts[0]) ? $parts[0] : 'both';
+        $options = isset($parts[1]) ? json_decode($parts[1], true) : [];
+        return [$version, $options];
+    }
+
+    public function testArraySyntaxDefault() {
+        $prettyPrinter = new Standard(['shortArraySyntax' => true]);
+        $expr = new Expr\Array_([
+            new Expr\ArrayItem(new String_('val'), new String_('key'))
+        ]);
+        $expected = "['key' => 'val']";
+        $this->assertSame($expected, $prettyPrinter->prettyPrintExpr($expr));
+    }
+
+    /**
+     * @dataProvider provideTestKindAttributes
+     */
+    public function testKindAttributes($node, $expected) {
+        $prttyPrinter = new PrettyPrinter\Standard;
+        $result = $prttyPrinter->prettyPrintExpr($node);
+        $this->assertSame($expected, $result);
+    }
+
+    public function provideTestKindAttributes() {
+        $nowdoc = ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR'];
+        $heredoc = ['kind' => String_::KIND_HEREDOC, 'docLabel' => 'STR'];
+        return [
+            // Defaults to single quoted
+            [new String_('foo'), "'foo'"],
+            // Explicit single/double quoted
+            [new String_('foo', ['kind' => String_::KIND_SINGLE_QUOTED]), "'foo'"],
+            [new String_('foo', ['kind' => String_::KIND_DOUBLE_QUOTED]), '"foo"'],
+            // Fallback from doc string if no label
+            [new String_('foo', ['kind' => String_::KIND_NOWDOC]), "'foo'"],
+            [new String_('foo', ['kind' => String_::KIND_HEREDOC]), '"foo"'],
+            // Fallback if string contains label
+            [new String_("A\nB\nC", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'A']), "'A\nB\nC'"],
+            [new String_("A\nB\nC", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'B']), "'A\nB\nC'"],
+            [new String_("A\nB\nC", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'C']), "'A\nB\nC'"],
+            [new String_("STR;", ['kind' => String_::KIND_NOWDOC, 'docLabel' => 'STR']), "'STR;'"],
+            // Doc string if label not contained (or not in ending position)
+            [new String_("foo", $nowdoc), "<<<'STR'\nfoo\nSTR\n"],
+            [new String_("foo", $heredoc), "<<<STR\nfoo\nSTR\n"],
+            [new String_("STRx", $nowdoc), "<<<'STR'\nSTRx\nSTR\n"],
+            [new String_("xSTR", $nowdoc), "<<<'STR'\nxSTR\nSTR\n"],
+            // Empty doc string variations (encapsed variant does not occur naturally)
+            [new String_("", $nowdoc), "<<<'STR'\nSTR\n"],
+            [new String_("", $heredoc), "<<<STR\nSTR\n"],
+            [new Encapsed([new EncapsedStringPart('')], $heredoc), "<<<STR\nSTR\n"],
+            // Encapsed doc string variations
+            [new Encapsed([new EncapsedStringPart('foo')], $heredoc), "<<<STR\nfoo\nSTR\n"],
+            [new Encapsed([new EncapsedStringPart('foo'), new Expr\Variable('y')], $heredoc), "<<<STR\nfoo{\$y}\nSTR\n"],
+            [new Encapsed([new EncapsedStringPart("\nSTR"), new Expr\Variable('y')], $heredoc), "<<<STR\n\nSTR{\$y}\nSTR\n"],
+            [new Encapsed([new EncapsedStringPart("\nSTR"), new Expr\Variable('y')], $heredoc), "<<<STR\n\nSTR{\$y}\nSTR\n"],
+            [new Encapsed([new Expr\Variable('y'), new EncapsedStringPart("STR\n")], $heredoc), "<<<STR\n{\$y}STR\n\nSTR\n"],
+            // Encapsed doc string fallback
+            [new Encapsed([new Expr\Variable('y'), new EncapsedStringPart("\nSTR")], $heredoc), '"{$y}\\nSTR"'],
+            [new Encapsed([new EncapsedStringPart("STR\n"), new Expr\Variable('y')], $heredoc), '"STR\\n{$y}"'],
+            [new Encapsed([new EncapsedStringPart("STR")], $heredoc), '"STR"'],
+        ];
+    }
 }

test/PhpParser/Serializer/XMLTest.php

@@ -23,6 +23,9 @@ CODE;
 <AST xmlns:node="http://nikic.github.com/PHPParser/XML/node" xmlns:subNode="http://nikic.github.com/PHPParser/XML/subNode" xmlns:attribute="http://nikic.github.com/PHPParser/XML/attribute" xmlns:scalar="http://nikic.github.com/PHPParser/XML/scalar">
  <scalar:array>
   <node:Stmt_Function>
+   <attribute:startLine>
+    <scalar:int>4</scalar:int>
+   </attribute:startLine>
    <attribute:comments>
     <scalar:array>
      <comment isDocComment="false" line="2">// comment
@@ -30,9 +33,6 @@ CODE;
      <comment isDocComment="true" line="3">/** doc comment */</comment>
     </scalar:array>
    </attribute:comments>
-   <attribute:startLine>
-    <scalar:int>4</scalar:int>
-   </attribute:startLine>
    <attribute:endLine>
     <scalar:int>6</scalar:int>
    </attribute:endLine>
@@ -71,6 +71,9 @@ CODE;
         <attribute:endLine>
          <scalar:int>4</scalar:int>
         </attribute:endLine>
+        <attribute:kind>
+         <scalar:int>10</scalar:int>
+        </attribute:kind>
         <subNode:value>
          <scalar:int>0</scalar:int>
         </subNode:value>
@@ -113,7 +116,7 @@ CODE;
     </scalar:array>
    </subNode:params>
    <subNode:returnType>
-     <scalar:null/>
+    <scalar:null/>
    </subNode:returnType>
    <subNode:stmts>
     <scalar:array>
@@ -133,6 +136,9 @@ CODE;
          <attribute:endLine>
           <scalar:int>5</scalar:int>
          </attribute:endLine>
+         <attribute:kind>
+          <scalar:int>1</scalar:int>
+         </attribute:kind>
          <subNode:value>
           <scalar:string>Foo</scalar:string>
          </subNode:value>

test/bootstrap.php

@@ -2,15 +2,19 @@
 
 namespace PhpParser;
 
-require __DIR__ . '/../lib/bootstrap.php';
+require __DIR__ . '/../vendor/autoload.php';
 
 function canonicalize($str) {
-    // trim from both sides
-    $str = trim($str);
+    // normalize EOL style
+    $str = str_replace("\r\n", "\n", $str);
 
-    // normalize EOL to \n
-    $str = str_replace(array("\r\n", "\r"), "\n", $str);
+    // trim newlines at end
+    $str = rtrim($str, "\n");
 
-    // trim right side of all lines
-    return implode("\n", array_map('rtrim', explode("\n", $str)));
+    // remove trailing whitespace on all lines
+    $lines = explode("\n", $str);
+    $lines = array_map(function($line) {
+        return rtrim($line, " \t");
+    }, $lines);
+    return implode("\n", $lines);
 }

test/code/parser/blockComments.test

@@ -0,0 +1,23 @@
+Comments on blocks
+-----
+<?php
+
+// foo
+{
+    // bar
+    {
+        // baz
+        $a;
+    }
+}
+-----
+array(
+    0: Expr_Variable(
+        name: a
+        comments: array(
+            0: // foo
+            1: // bar
+            2: // baz
+        )
+    )
+)

test/code/parser/comments.test

@@ -0,0 +1,100 @@
+Comments
+-----
+<?php
+
+/** doc 1 */
+/* foobar 1 */
+// foo 1
+// bar 1
+$var;
+
+if ($cond) {
+    /** doc 2 */
+    /* foobar 2 */
+    // foo 2
+    // bar 2
+}
+
+/** doc 3 */
+/* foobar 3 */
+// foo 3
+// bar 3
+-----
+array(
+    0: Expr_Variable(
+        name: var
+        comments: array(
+            0: /** doc 1 */
+            1: /* foobar 1 */
+            2: // foo 1
+            3: // bar 1
+        )
+    )
+    1: Stmt_If(
+        cond: Expr_Variable(
+            name: cond
+        )
+        stmts: array(
+            0: Stmt_Nop(
+                comments: array(
+                    0: /** doc 2 */
+                    1: /* foobar 2 */
+                    2: // foo 2
+                    3: // bar 2
+                )
+            )
+        )
+        elseifs: array(
+        )
+        else: null
+    )
+    2: Stmt_Nop(
+        comments: array(
+            0: /** doc 3 */
+            1: /* foobar 3 */
+            2: // foo 3
+            3: // bar 3
+        )
+    )
+)
+-----
+<?php
+
+/** doc */
+/* foobar */
+// foo
+// bar
+
+?>
+-----
+array(
+    0: Stmt_Nop(
+        comments: array(
+            0: /** doc */
+            1: /* foobar */
+            2: // foo
+            3: // bar
+        )
+    )
+)
+-----
+<?php
+
+// comment
+if (42) {}
+-----
+array(
+    0: Stmt_If(
+        cond: Scalar_LNumber(
+            value: 42
+        )
+        stmts: array(
+        )
+        elseifs: array(
+        )
+        else: null
+        comments: array(
+            0: // comment
+        )
+    )
+)

test/code/parser/errorHandling/eofError.test

@@ -4,10 +4,29 @@ Error positions
 -----
 Syntax error, unexpected EOF from 1:10 to 1:10
 array(
+    0: Expr_ConstFetch(
+        name: Name(
+            parts: array(
+                0: foo
+            )
+        )
+    )
 )
 -----
 <?php foo /* bar */
 -----
 Syntax error, unexpected EOF from 1:20 to 1:20
 array(
-)
+    0: Expr_ConstFetch(
+        name: Name(
+            parts: array(
+                0: foo
+            )
+        )
+    )
+    1: Stmt_Nop(
+        comments: array(
+            0: /* bar */
+        )
+    )
+)

test/code/parser/errorHandling/recovery.test

@@ -10,6 +10,33 @@ Syntax error, unexpected T_STRING from 4:1 to 4:3
 Syntax error, unexpected T_STRING from 5:1 to 5:3
 Syntax error, unexpected EOF from 5:6 to 5:6
 array(
+    0: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: foo
+            )
+        )
+        args: array(
+        )
+    )
+    1: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: bar
+            )
+        )
+        args: array(
+        )
+    )
+    2: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: baz
+            )
+        )
+        args: array(
+        )
+    )
 )
 -----
 <?php
@@ -23,13 +50,22 @@ array(
     0: Expr_FuncCall(
         name: Name(
             parts: array(
-                0: bar
+                0: foo
             )
         )
         args: array(
         )
     )
     1: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: bar
+            )
+        )
+        args: array(
+        )
+    )
+    2: Expr_FuncCall(
         name: Name(
             parts: array(
                 0: baz
@@ -58,6 +94,15 @@ array(
         )
     )
     1: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: bar
+            )
+        )
+        args: array(
+        )
+    )
+    2: Expr_FuncCall(
         name: Name(
             parts: array(
                 0: baz
@@ -81,6 +126,9 @@ array(
             )
         )
     )
+    1: Scalar_LNumber(
+        value: 1
+    )
 )
 -----
 <?php
@@ -97,6 +145,9 @@ array(
         )
         returnType: null
         stmts: array(
+            0: Scalar_LNumber(
+                value: 1
+            )
         )
     )
 )
@@ -172,6 +223,11 @@ array(
             value: 2
         )
     )
+    3: Stmt_Nop(
+        comments: array(
+            0: // The output here drops the loop - would require Error node to handle this
+        )
+    )
 )
 -----
 <?php

test/code/parser/expr/arrayDef.test

@@ -97,6 +97,9 @@ array(
     5: Expr_Array(
         items: array(
         )
+        comments: array(
+            0: // short array syntax
+        )
     )
     6: Expr_Array(
         items: array(

test/code/parser/expr/assign.test

@@ -23,7 +23,6 @@ $a = $b *= $c **= $d;
 
 // by ref assign
 $a =& $b;
-$a =& new B;
 
 // list() assign
 list($a) = $b;
@@ -40,18 +39,30 @@ array(
     0: Expr_Assign(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // simple assign
+            )
         )
         expr: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // simple assign
+        )
     )
     1: Expr_AssignOp_BitwiseAnd(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // combined assign
+            )
         )
         expr: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // combined assign
+        )
     )
     2: Expr_AssignOp_BitwiseOr(
         var: Expr_Variable(
@@ -144,6 +155,9 @@ array(
     13: Expr_Assign(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // chained assign
+            )
         )
         expr: Expr_AssignOp_Mul(
             var: Expr_Variable(
@@ -158,42 +172,43 @@ array(
                 )
             )
         )
+        comments: array(
+            0: // chained assign
+        )
     )
     14: Expr_AssignRef(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // by ref assign
+            )
         )
         expr: Expr_Variable(
             name: b
         )
-    )
-    15: Expr_AssignRef(
-        var: Expr_Variable(
-            name: a
-        )
-        expr: Expr_New(
-            class: Name(
-                parts: array(
-                    0: B
-                )
-            )
-            args: array(
-            )
+        comments: array(
+            0: // by ref assign
         )
     )
-    16: Expr_Assign(
+    15: Expr_Assign(
         var: Expr_List(
             vars: array(
                 0: Expr_Variable(
                     name: a
                 )
             )
+            comments: array(
+                0: // list() assign
+            )
         )
         expr: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // list() assign
+        )
     )
-    17: Expr_Assign(
+    16: Expr_Assign(
         var: Expr_List(
             vars: array(
                 0: Expr_Variable(
@@ -209,7 +224,7 @@ array(
             name: c
         )
     )
-    18: Expr_Assign(
+    17: Expr_Assign(
         var: Expr_List(
             vars: array(
                 0: Expr_Variable(
@@ -232,22 +247,25 @@ array(
             name: e
         )
     )
-    19: Expr_PreInc(
+    18: Expr_PreInc(
+        var: Expr_Variable(
+            name: a
+        )
+        comments: array(
+            0: // inc/dec
+        )
+    )
+    19: Expr_PostInc(
         var: Expr_Variable(
             name: a
         )
     )
-    20: Expr_PostInc(
+    20: Expr_PreDec(
         var: Expr_Variable(
             name: a
         )
     )
-    21: Expr_PreDec(
-        var: Expr_Variable(
-            name: a
-        )
-    )
-    22: Expr_PostDec(
+    21: Expr_PostDec(
         var: Expr_Variable(
             name: a
         )

test/code/parser/expr/assignNewByRef.test

@@ -0,0 +1,39 @@
+Assigning new by reference (PHP 5 only)
+-----
+<?php
+$a =& new B;
+-----
+!!php5
+array(
+    0: Expr_AssignRef(
+        var: Expr_Variable(
+            name: a
+        )
+        expr: Expr_New(
+            class: Name(
+                parts: array(
+                    0: B
+                )
+            )
+            args: array(
+            )
+        )
+    )
+)
+-----
+<?php
+$a =& new B;
+-----
+!!php7
+Syntax error, unexpected T_NEW from 2:7 to 2:9
+array(
+    0: Expr_New(
+        class: Name(
+            parts: array(
+                0: B
+            )
+        )
+        args: array(
+        )
+    )
+)

test/code/parser/expr/fetchAndCall/funcCall.test

@@ -21,9 +21,15 @@ array(
             parts: array(
                 0: a
             )
+            comments: array(
+                0: // function name variations
+            )
         )
         args: array(
         )
+        comments: array(
+            0: // function name variations
+        )
     )
     1: Expr_FuncCall(
         name: Expr_Variable(
@@ -106,12 +112,21 @@ array(
                 parts: array(
                     0: a
                 )
+                comments: array(
+                    0: // array dereferencing
+                )
             )
             args: array(
             )
+            comments: array(
+                0: // array dereferencing
+            )
         )
         dim: Scalar_String(
             value: b
         )
+        comments: array(
+            0: // array dereferencing
+        )
     )
 )

test/code/parser/expr/fetchAndCall/objectAccess.test

@@ -22,8 +22,14 @@ array(
     0: Expr_PropertyFetch(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // property fetch variations
+            )
         )
         name: b
+        comments: array(
+            0: // property fetch variations
+        )
     )
     1: Expr_ArrayDimFetch(
         var: Expr_PropertyFetch(
@@ -50,10 +56,16 @@ array(
     3: Expr_MethodCall(
         var: Expr_Variable(
             name: a
+            comments: array(
+                0: // method call variations
+            )
         )
         name: b
         args: array(
         )
+        comments: array(
+            0: // method call variations
+        )
     )
     4: Expr_MethodCall(
         var: Expr_Variable(
@@ -94,14 +106,23 @@ array(
         var: Expr_MethodCall(
             var: Expr_Variable(
                 name: a
+                comments: array(
+                    0: // array dereferencing
+                )
             )
             name: b
             args: array(
             )
+            comments: array(
+                0: // array dereferencing
+            )
         )
         dim: Scalar_String(
             value: c
         )
+        comments: array(
+            0: // array dereferencing
+        )
     )
     8: Expr_ArrayDimFetch(
         var: Expr_MethodCall(
@@ -116,4 +137,9 @@ array(
             value: c
         )
     )
-)
+    9: Stmt_Nop(
+        comments: array(
+            0: // invalid PHP: drop Support?
+        )
+    )
+)

test/code/parser/expr/fetchAndCall/staticCall.test

@@ -25,10 +25,16 @@ array(
             parts: array(
                 0: A
             )
+            comments: array(
+                0: // method name variations
+            )
         )
         name: b
         args: array(
         )
+        comments: array(
+            0: // method name variations
+        )
     )
     1: Expr_StaticCall(
         class: Name(
@@ -99,24 +105,39 @@ array(
                 parts: array(
                     0: A
                 )
+                comments: array(
+                    0: // array dereferencing
+                )
             )
             name: b
             args: array(
             )
+            comments: array(
+                0: // array dereferencing
+            )
         )
         dim: Scalar_String(
             value: c
         )
+        comments: array(
+            0: // array dereferencing
+        )
     )
     6: Expr_StaticCall(
         class: Name(
             parts: array(
                 0: static
             )
+            comments: array(
+                0: // class name variations
+            )
         )
         name: b
         args: array(
         )
+        comments: array(
+            0: // class name variations
+        )
     )
     7: Expr_StaticCall(
         class: Expr_Variable(
@@ -149,4 +170,4 @@ array(
         args: array(
         )
     )
-)
+)

test/code/parser/expr/fetchAndCall/staticPropertyFetch.test

@@ -19,8 +19,14 @@ array(
             parts: array(
                 0: A
             )
+            comments: array(
+                0: // property name variations
+            )
         )
         name: b
+        comments: array(
+            0: // property name variations
+        )
     )
     1: Expr_StaticPropertyFetch(
         class: Name(
@@ -48,12 +54,21 @@ array(
                 parts: array(
                     0: A
                 )
+                comments: array(
+                    0: // array access
+                )
             )
             name: b
+            comments: array(
+                0: // array access
+            )
         )
         dim: Scalar_String(
             value: c
         )
+        comments: array(
+            0: // array access
+        )
     )
     4: Expr_ArrayDimFetch(
         var: Expr_StaticPropertyFetch(
@@ -68,4 +83,9 @@ array(
             value: c
         )
     )
+    5: Stmt_Nop(
+        comments: array(
+            0: // class name variations can be found in staticCall.test
+        )
+    )
 )

test/code/parser/expr/logic.test

@@ -24,10 +24,16 @@ array(
     0: Expr_BinaryOp_BooleanAnd(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // boolean ops
+            )
         )
         right: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // boolean ops
+        )
     )
     1: Expr_BinaryOp_BooleanOr(
         left: Expr_Variable(
@@ -52,10 +58,16 @@ array(
     4: Expr_BinaryOp_LogicalAnd(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // logical ops
+            )
         )
         right: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // logical ops
+        )
     )
     5: Expr_BinaryOp_LogicalOr(
         left: Expr_Variable(
@@ -77,10 +89,16 @@ array(
         left: Expr_BinaryOp_BooleanAnd(
             left: Expr_Variable(
                 name: a
+                comments: array(
+                    0: // precedence
+                )
             )
             right: Expr_Variable(
                 name: b
             )
+            comments: array(
+                0: // precedence
+            )
         )
         right: Expr_BinaryOp_BooleanAnd(
             left: Expr_Variable(
@@ -90,6 +108,9 @@ array(
                 name: d
             )
         )
+        comments: array(
+            0: // precedence
+        )
     )
     8: Expr_BinaryOp_BooleanAnd(
         left: Expr_BinaryOp_BooleanAnd(

test/code/parser/expr/math.test

@@ -38,6 +38,9 @@ array(
         expr: Expr_Variable(
             name: a
         )
+        comments: array(
+            0: // unary ops
+        )
     )
     1: Expr_UnaryPlus(
         expr: Expr_Variable(
@@ -52,10 +55,16 @@ array(
     3: Expr_BinaryOp_BitwiseAnd(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // binary ops
+            )
         )
         right: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // binary ops
+        )
     )
     4: Expr_BinaryOp_BitwiseOr(
         left: Expr_Variable(
@@ -149,14 +158,23 @@ array(
         left: Expr_BinaryOp_Mul(
             left: Expr_Variable(
                 name: a
+                comments: array(
+                    0: // associativity
+                )
             )
             right: Expr_Variable(
                 name: b
             )
+            comments: array(
+                0: // associativity
+            )
         )
         right: Expr_Variable(
             name: c
         )
+        comments: array(
+            0: // associativity
+        )
     )
     16: Expr_BinaryOp_Mul(
         left: Expr_Variable(
@@ -174,6 +192,9 @@ array(
     17: Expr_BinaryOp_Plus(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // precedence
+            )
         )
         right: Expr_BinaryOp_Mul(
             left: Expr_Variable(
@@ -183,6 +204,9 @@ array(
                 name: c
             )
         )
+        comments: array(
+            0: // precedence
+        )
     )
     18: Expr_BinaryOp_Mul(
         left: Expr_BinaryOp_Plus(
@@ -200,6 +224,9 @@ array(
     19: Expr_BinaryOp_Pow(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // pow is special
+            )
         )
         right: Expr_BinaryOp_Pow(
             left: Expr_Variable(
@@ -209,6 +236,9 @@ array(
                 name: c
             )
         )
+        comments: array(
+            0: // pow is special
+        )
     )
     20: Expr_BinaryOp_Pow(
         left: Expr_BinaryOp_Pow(

test/code/parser/expr/new.test

@@ -50,6 +50,9 @@ array(
         )
         args: array(
         )
+        comments: array(
+            0: // class name variations
+        )
     )
     3: Expr_New(
         class: Expr_ArrayDimFetch(
@@ -84,6 +87,9 @@ array(
         )
         args: array(
         )
+        comments: array(
+            0: // DNCR object access
+        )
     )
     6: Expr_New(
         class: Expr_PropertyFetch(

test/code/parser/expr/shellExec.test

@@ -14,12 +14,16 @@ array(
     )
     1: Expr_ShellExec(
         parts: array(
-            0: test
+            0: Scalar_EncapsedStringPart(
+                value: test
+            )
         )
     )
     2: Expr_ShellExec(
         parts: array(
-            0: test
+            0: Scalar_EncapsedStringPart(
+                value: test
+            )
             1: Expr_Variable(
                 name: A
             )
@@ -27,12 +31,16 @@ array(
     )
     3: Expr_ShellExec(
         parts: array(
-            0: test `
+            0: Scalar_EncapsedStringPart(
+                value: test `
+            )
         )
     )
     4: Expr_ShellExec(
         parts: array(
-            0: test \"
+            0: Scalar_EncapsedStringPart(
+                value: test \"
+            )
         )
     )
 )

test/code/parser/expr/ternaryAndCoalesce.test

@@ -20,6 +20,9 @@ array(
     0: Expr_Ternary(
         cond: Expr_Variable(
             name: a
+            comments: array(
+                0: // ternary
+            )
         )
         if: Expr_Variable(
             name: b
@@ -27,6 +30,9 @@ array(
         else: Expr_Variable(
             name: c
         )
+        comments: array(
+            0: // ternary
+        )
     )
     1: Expr_Ternary(
         cond: Expr_Variable(
@@ -41,6 +47,9 @@ array(
         cond: Expr_Ternary(
             cond: Expr_Variable(
                 name: a
+                comments: array(
+                    0: // precedence
+                )
             )
             if: Expr_Variable(
                 name: b
@@ -48,6 +57,9 @@ array(
             else: Expr_Variable(
                 name: c
             )
+            comments: array(
+                0: // precedence
+            )
         )
         if: Expr_Variable(
             name: d
@@ -55,6 +67,9 @@ array(
         else: Expr_Variable(
             name: e
         )
+        comments: array(
+            0: // precedence
+        )
     )
     3: Expr_Ternary(
         cond: Expr_Variable(
@@ -78,10 +93,16 @@ array(
     4: Expr_BinaryOp_Coalesce(
         left: Expr_Variable(
             name: a
+            comments: array(
+                0: // null coalesce
+            )
         )
         right: Expr_Variable(
             name: b
         )
+        comments: array(
+            0: // null coalesce
+        )
     )
     5: Expr_BinaryOp_Coalesce(
         left: Expr_Variable(
@@ -125,4 +146,4 @@ array(
             name: c
         )
     )
-)
+)

test/code/parser/scalar/constantString.test

@@ -8,8 +8,10 @@ b'';
 b"";
 'Hi';
 b'Hi';
+B'Hi';
 "Hi";
 b"Hi";
+B"Hi";
 '!\'!\\!\a!';
 "!\"!\\!\$!\n!\r!\t!\f!\v!\e!\a";
 "!\xFF!\377!\400!\0!";
@@ -40,14 +42,19 @@ array(
         value: Hi
     )
     8: Scalar_String(
-        value: !'!\!\a!
+        value: Hi
     )
     9: Scalar_String(
-        value: !"!\!$!
-    !
-!@@{ "\t" }@@!@@{ "\f" }@@!@@{ "\v" }@@!@@{ chr(27) /* "\e" */ }@@!\a
+        value: Hi
     )
     10: Scalar_String(
+        value: !'!\!\a!
+    )
+    11: Scalar_String(
+        value: !"!\!$!
+    !@@{ "\r" }@@!@@{ "\t" }@@!@@{ "\f" }@@!@@{ "\v" }@@!@@{ chr(27) /* "\e" */ }@@!\a
+    )
+    12: Scalar_String(
         value: !@@{ chr(255) }@@!@@{ chr(255) }@@!@@{ chr(0) }@@!@@{ chr(0) }@@!
     )
 )

test/code/parser/scalar/docString.test

@@ -24,17 +24,26 @@ EOS;
 Test $a and $b->c test
 EOS;
 
-// comment to force line break before EOF
+b<<<EOS
+Binary
+EOS;
+
 -----
 array(
     0: Scalar_String(
         value:
+        comments: array(
+            0: // empty strings
+        )
     )
     1: Scalar_String(
         value:
     )
     2: Scalar_String(
         value: Test '" $a \n
+        comments: array(
+            0: // constant encapsed strings
+        )
     )
     3: Scalar_String(
         value: Test '" $a
@@ -42,26 +51,40 @@ array(
     )
     4: Scalar_Encapsed(
         parts: array(
-            0: Test
+            0: Scalar_EncapsedStringPart(
+                value: Test
+            )
             1: Expr_Variable(
                 name: a
             )
         )
+        comments: array(
+            0: // encapsed strings
+        )
     )
     5: Scalar_Encapsed(
         parts: array(
-            0: Test
+            0: Scalar_EncapsedStringPart(
+                value: Test
+            )
             1: Expr_Variable(
                 name: a
             )
-            2:  and
+            2: Scalar_EncapsedStringPart(
+                value:  and
+            )
             3: Expr_PropertyFetch(
                 var: Expr_Variable(
                     name: b
                 )
                 name: c
             )
-            4:  test
+            4: Scalar_EncapsedStringPart(
+                value:  test
+            )
         )
     )
+    6: Scalar_String(
+        value: Binary
+    )
 )

test/code/parser/scalar/docStringNewlines.test

@@ -0,0 +1,61 @@
+Trailing newlines in doc strings
+-----
+<?php
+
+<<<'EOF'@@{ "\n\n" }@@EOF;
+<<<'EOF'@@{ "\n\n\n" }@@EOF;
+<<<'EOF'@@{ "\nFoo\n\n" }@@EOF;
+<<<EOF@@{ "\n\$var\n\n" }@@EOF;
+
+<<<'EOF'@@{ "\r\n\r\n" }@@EOF;
+<<<'EOF'@@{ "\r\n\r\n\r\n" }@@EOF;
+<<<'EOF'@@{ "\r\nFoo\r\n\r\n" }@@EOF;
+<<<EOF@@{ "\r\n\$var\r\n\r\n" }@@EOF;
+
+-----
+array(
+    0: Scalar_String(
+        value:
+    )
+    1: Scalar_String(
+        value:
+
+    )
+    2: Scalar_String(
+        value: Foo
+
+    )
+    3: Scalar_Encapsed(
+        parts: array(
+            0: Expr_Variable(
+                name: var
+            )
+            1: Scalar_EncapsedStringPart(
+                value:
+
+            )
+        )
+    )
+    4: Scalar_String(
+        value:
+    )
+    5: Scalar_String(
+        value:
+
+    )
+    6: Scalar_String(
+        value: Foo
+
+    )
+    7: Scalar_Encapsed(
+        parts: array(
+            0: Expr_Variable(
+                name: var
+            )
+            1: Scalar_EncapsedStringPart(
+                value:
+
+            )
+        )
+    )
+)

test/code/parser/scalar/encapsedString.test

@@ -21,6 +21,7 @@ Encapsed strings
 "$$A[B]";
 "A $B C";
 b"$A";
+B"$A";
 -----
 array(
     0: Scalar_Encapsed(
@@ -137,25 +138,35 @@ array(
     )
     11: Scalar_Encapsed(
         parts: array(
-            0: \{
+            0: Scalar_EncapsedStringPart(
+                value: \{
+            )
             1: Expr_Variable(
                 name: A
             )
-            2: }
+            2: Scalar_EncapsedStringPart(
+                value: }
+            )
         )
     )
     12: Scalar_Encapsed(
         parts: array(
-            0: \{
+            0: Scalar_EncapsedStringPart(
+                value: \{
+            )
             1: Expr_Variable(
                 name: A
             )
-            2:  }
+            2: Scalar_EncapsedStringPart(
+                value:  }
+            )
         )
     )
     13: Scalar_Encapsed(
         parts: array(
-            0: \
+            0: Scalar_EncapsedStringPart(
+                value: \
+            )
             1: Expr_Variable(
                 name: A
             )
@@ -163,11 +174,15 @@ array(
     )
     14: Scalar_Encapsed(
         parts: array(
-            0: \{
+            0: Scalar_EncapsedStringPart(
+                value: \{
+            )
             1: Expr_Variable(
                 name: A
             )
-            2:  }
+            2: Scalar_EncapsedStringPart(
+                value:  }
+            )
         )
     )
     15: Scalar_Encapsed(
@@ -177,12 +192,16 @@ array(
                     name: A
                 )
             )
-            1: [B]
+            1: Scalar_EncapsedStringPart(
+                value: [B]
+            )
         )
     )
     16: Scalar_Encapsed(
         parts: array(
-            0: $
+            0: Scalar_EncapsedStringPart(
+                value: $
+            )
             1: Expr_ArrayDimFetch(
                 var: Expr_Variable(
                     name: A
@@ -195,11 +214,15 @@ array(
     )
     17: Scalar_Encapsed(
         parts: array(
-            0: A
+            0: Scalar_EncapsedStringPart(
+                value: A
+            )
             1: Expr_Variable(
                 name: B
             )
-            2:  C
+            2: Scalar_EncapsedStringPart(
+                value:  C
+            )
         )
     )
     18: Scalar_Encapsed(
@@ -209,4 +232,11 @@ array(
             )
         )
     )
-)
+    19: Scalar_Encapsed(
+        parts: array(
+            0: Expr_Variable(
+                name: A
+            )
+        )
+    )
+)

test/code/parser/scalar/float.test

@@ -53,18 +53,22 @@ array(
         value: INF
     )
     10: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
+        comments: array(
+            0: // various integer -> float overflows
+            1: // (all are actually the same number, just in different representations)
+        )
     )
     11: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
     )
     12: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
     )
     13: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
     )
     14: Scalar_DNumber(
-        value: @@{ 0xFFFFFFFFFFFFFFFF }@@
+        value: 1.844674407371E+19
     )
 )

test/code/parser/scalar/int.test

@@ -10,7 +10,6 @@ Different integer syntaxes
 0xfff;
 0XfFf;
 0777;
-0787;
 0b111000111000;
 -----
 array(
@@ -39,9 +38,6 @@ array(
         value: 511
     )
     8: Scalar_LNumber(
-        value: 7
-    )
-    9: Scalar_LNumber(
         value: 3640
     )
 )

test/code/parser/scalar/invalidOctal.test

@@ -0,0 +1,17 @@
+Invalid octal literals
+-----
+<?php
+0787;
+-----
+!!php7
+Invalid numeric literal from 2:1 to 2:4
+-----
+<?php
+0787;
+-----
+!!php5
+array(
+    0: Scalar_LNumber(
+        value: 7
+    )
+)

test/code/parser/semiReserved.test

@@ -13,6 +13,10 @@ class Test {
     public $private;
 
     const TRAIT = 3, FINAL = 4;
+
+    const __CLASS__ = 1, __TRAIT__ = 2, __FUNCTION__ = 3, __METHOD__ = 4, __LINE__ = 5,
+          __FILE__ = 6, __DIR__ = 7, __NAMESPACE__ = 8;
+    // __halt_compiler does not work
 }
 
 $t = new Test;
@@ -130,6 +134,58 @@ array(
                     )
                 )
             )
+            7: Stmt_ClassConst(
+                consts: array(
+                    0: Const(
+                        name: __CLASS__
+                        value: Scalar_LNumber(
+                            value: 1
+                        )
+                    )
+                    1: Const(
+                        name: __TRAIT__
+                        value: Scalar_LNumber(
+                            value: 2
+                        )
+                    )
+                    2: Const(
+                        name: __FUNCTION__
+                        value: Scalar_LNumber(
+                            value: 3
+                        )
+                    )
+                    3: Const(
+                        name: __METHOD__
+                        value: Scalar_LNumber(
+                            value: 4
+                        )
+                    )
+                    4: Const(
+                        name: __LINE__
+                        value: Scalar_LNumber(
+                            value: 5
+                        )
+                    )
+                    5: Const(
+                        name: __FILE__
+                        value: Scalar_LNumber(
+                            value: 6
+                        )
+                    )
+                    6: Const(
+                        name: __DIR__
+                        value: Scalar_LNumber(
+                            value: 7
+                        )
+                    )
+                    7: Const(
+                        name: __NAMESPACE__
+                        value: Scalar_LNumber(
+                            value: 8
+                        )
+                    )
+                )
+            )
         )
     )
     1: Expr_Assign(
@@ -321,4 +377,4 @@ array(
             )
         )
     )
-)
+)

test/code/parser/stmt/class/modifier.test

@@ -38,6 +38,11 @@ array(
         stmts: array(
         )
     )
+    1: Stmt_Nop(
+        comments: array(
+            0: // Type in the partial parse could conceivably be any of 0, 16 or 32
+        )
+    )
 )
 -----
 <?php class A { abstract $a; }

test/code/parser/stmt/declare.test

@@ -2,6 +2,8 @@ Declare
 -----
 <?php
 
+declare (X='Y');
+
 declare (A='B', C='D') {}
 
 declare (A='B', C='D'):
@@ -11,20 +13,13 @@ array(
     0: Stmt_Declare(
         declares: array(
             0: Stmt_DeclareDeclare(
-                key: A
+                key: X
                 value: Scalar_String(
-                    value: B
-                )
-            )
-            1: Stmt_DeclareDeclare(
-                key: C
-                value: Scalar_String(
-                    value: D
+                    value: Y
                 )
             )
         )
-        stmts: array(
-        )
+        stmts: null
     )
     1: Stmt_Declare(
         declares: array(
@@ -44,4 +39,22 @@ array(
         stmts: array(
         )
     )
+    2: Stmt_Declare(
+        declares: array(
+            0: Stmt_DeclareDeclare(
+                key: A
+                value: Scalar_String(
+                    value: B
+                )
+            )
+            1: Stmt_DeclareDeclare(
+                key: C
+                value: Scalar_String(
+                    value: D
+                )
+            )
+        )
+        stmts: array(
+        )
+    )
 )

test/code/parser/stmt/generator/basic.test

@@ -42,6 +42,9 @@ array(
             0: Expr_Yield(
                 key: null
                 value: null
+                comments: array(
+                    0: // statements
+                )
             )
             1: Expr_Yield(
                 key: null
@@ -60,11 +63,17 @@ array(
             3: Expr_Assign(
                 var: Expr_Variable(
                     name: data
+                    comments: array(
+                        0: // expressions
+                    )
                 )
                 expr: Expr_Yield(
                     key: null
                     value: null
                 )
+                comments: array(
+                    0: // expressions
+                )
             )
             4: Expr_Assign(
                 var: Expr_Variable(
@@ -112,6 +121,9 @@ array(
                     )
                 )
                 else: null
+                comments: array(
+                    0: // yield in language constructs with their own parentheses
+                )
             )
             7: Stmt_If(
                 cond: Expr_Yield(
@@ -179,6 +191,9 @@ array(
                     parts: array(
                         0: func
                     )
+                    comments: array(
+                        0: // yield in function calls
+                    )
                 )
                 args: array(
                     0: Arg(
@@ -192,6 +207,9 @@ array(
                         unpack: false
                     )
                 )
+                comments: array(
+                    0: // yield in function calls
+                )
             )
             13: Expr_MethodCall(
                 var: Expr_Variable(
@@ -259,4 +277,4 @@ array(
             )
         )
     )
-)
+)

test/code/parser/stmt/if.test

@@ -81,6 +81,9 @@ array(
             stmts: array(
             )
         )
+        comments: array(
+            0: // without else
+        )
     )
     3: Stmt_If(
         cond: Expr_Variable(
@@ -92,4 +95,9 @@ array(
         )
         else: null
     )
+    4: Stmt_Nop(
+        comments: array(
+            0: // without else
+        )
+    )
 )

test/code/parser/stmt/loop/for.test

@@ -6,7 +6,7 @@ For loop
 for ($i = 0; $i < $c; ++$i) {}
 
 // multiple expressions
-for (;$a,$b;) {}
+for ($a, $b; $c, $d; $e, $f) {}
 
 // infinite loop
 for (;;) {}
@@ -46,11 +46,12 @@ array(
         )
         stmts: array(
         )
+        comments: array(
+            0: // "classical" loop
+        )
     )
     1: Stmt_For(
         init: array(
-        )
-        cond: array(
             0: Expr_Variable(
                 name: a
             )
@@ -58,10 +59,27 @@ array(
                 name: b
             )
         )
+        cond: array(
+            0: Expr_Variable(
+                name: c
+            )
+            1: Expr_Variable(
+                name: d
+            )
+        )
         loop: array(
+            0: Expr_Variable(
+                name: e
+            )
+            1: Expr_Variable(
+                name: f
+            )
         )
         stmts: array(
         )
+        comments: array(
+            0: // multiple expressions
+        )
     )
     2: Stmt_For(
         init: array(
@@ -72,6 +90,9 @@ array(
         )
         stmts: array(
         )
+        comments: array(
+            0: // infinite loop
+        )
     )
     3: Stmt_For(
         init: array(
@@ -82,5 +103,8 @@ array(
         )
         stmts: array(
         )
+        comments: array(
+            0: // alternative syntax
+        )
     )
 )

test/code/parser/stmt/loop/foreach.test

@@ -29,6 +29,9 @@ array(
         )
         stmts: array(
         )
+        comments: array(
+            0: // foreach on variable
+        )
     )
     1: Stmt_Foreach(
         expr: Expr_Variable(
@@ -123,6 +126,9 @@ array(
         )
         stmts: array(
         )
+        comments: array(
+            0: // foreach on expression
+        )
     )
     7: Stmt_Foreach(
         expr: Expr_Variable(
@@ -135,5 +141,8 @@ array(
         )
         stmts: array(
         )
+        comments: array(
+            0: // alternative syntax
+        )
     )
 )

test/code/parser/stmt/namespace/alias.test

@@ -84,6 +84,9 @@ array(
                 alias: A
             )
         )
+        comments: array(
+            0: // evil alias notation - Do Not Use!
+        )
     )
     4: Stmt_Use(
         type: 1
@@ -113,6 +116,9 @@ array(
                 alias: bar
             )
         )
+        comments: array(
+            0: // function and constant aliases
+        )
     )
     6: Stmt_Use(
         type: 2
@@ -159,4 +165,4 @@ array(
             )
         )
     )
-)
+)

test/code/parser/stmt/namespace/groupUse.test

@@ -3,9 +3,9 @@ Group use declarations
 <?php
 use A\{B};
 use A\{B\C, D};
-use A\B\{C\D, E};
+use \A\B\{C\D, E};
 use function A\{b\c, d};
-use const A\{B\C, D};
+use const \A\{B\C, D};
 use A\B\{C\D, function b\c, const D};
 -----
 array(

test/code/parser/stmt/namespace/groupUseErrors.test

@@ -0,0 +1,66 @@
+Invalid group use syntax
+-----
+<?php
+// Missing semicolon
+use Foo\{Bar}
+use Bar\{Foo};
+-----
+Syntax error, unexpected T_USE, expecting ';' from 4:1 to 4:3
+array(
+    0: Stmt_GroupUse(
+        type: 0
+        prefix: Name(
+            parts: array(
+                0: Bar
+            )
+        )
+        uses: array(
+            0: Stmt_UseUse(
+                type: 1
+                name: Name(
+                    parts: array(
+                        0: Foo
+                    )
+                )
+                alias: Foo
+            )
+        )
+    )
+)
+-----
+<?php
+// Missing NS separator
+use Foo {Bar, Baz};
+-----
+Syntax error, unexpected '{', expecting ',' or ';' from 3:9 to 3:9
+array(
+    0: Expr_ConstFetch(
+        name: Name(
+            parts: array(
+                0: Bar
+            )
+        )
+    )
+    1: Expr_ConstFetch(
+        name: Name(
+            parts: array(
+                0: Baz
+            )
+        )
+    )
+)
+-----
+<?php
+// Extra NS separator
+use Foo\{\Bar};
+-----
+Syntax error, unexpected T_NS_SEPARATOR, expecting T_STRING or T_FUNCTION or T_CONST from 3:10 to 3:10
+array(
+    0: Expr_ConstFetch(
+        name: Name_FullyQualified(
+            parts: array(
+                0: Bar
+            )
+        )
+    )
+)

test/code/parser/stmt/namespace/nsAfterHashbang.test

@@ -0,0 +1,22 @@
+Hashbang followed by namespace declaration
+-----
+#!/usr/bin/env php
+<?php
+
+namespace A;
+-----
+array(
+    0: Stmt_InlineHTML(
+        value: #!/usr/bin/env php
+
+    )
+    1: Stmt_Namespace(
+        name: Name(
+            parts: array(
+                0: A
+            )
+        )
+        stmts: array(
+        )
+    )
+)

test/code/parser/stmt/namespace/outsideStmt.test

@@ -19,8 +19,7 @@ array(
                 )
             )
         )
-        stmts: array(
-        )
+        stmts: null
     )
     1: Stmt_Namespace(
         name: Name(
@@ -34,4 +33,26 @@ array(
     2: Stmt_HaltCompiler(
         remaining: Hi!
     )
+)
+-----
+<?php
+/* Comment */
+;
+namespace Foo;
+-----
+array(
+    0: Stmt_Nop(
+        comments: array(
+            0: /* Comment */
+        )
+    )
+    1: Stmt_Namespace(
+        name: Name(
+            parts: array(
+                0: Foo
+            )
+        )
+        stmts: array(
+        )
+    )
 )

test/code/parser/stmt/switch.test

@@ -49,6 +49,9 @@ array(
         )
         cases: array(
         )
+        comments: array(
+            0: // alternative syntax
+        )
     )
     2: Stmt_Switch(
         cond: Expr_Variable(
@@ -56,6 +59,9 @@ array(
         )
         cases: array(
         )
+        comments: array(
+            0: // leading semicolon
+        )
     )
     3: Stmt_Switch(
         cond: Expr_Variable(

test/code/parser/stmt/tryCatch.test

@@ -102,6 +102,9 @@ array(
             )
         )
         finallyStmts: null
+        comments: array(
+            0: // no finally
+        )
     )
     2: Stmt_TryCatch(
         stmts: array(
@@ -110,5 +113,8 @@ array(
         )
         finallyStmts: array(
         )
+        comments: array(
+            0: // no catch
+        )
     )
 )

test/code/prettyPrinter/comments.test

@@ -27,7 +27,6 @@ function justForIndentation()
        Even more text. */
     $foo;
 }
-
 -----
 function justForIndentation()
 {
@@ -53,4 +52,16 @@ function justForIndentation()
        More text.
        Even more text. */
     $foo;
+}
+-----
+<?php
+
+function test()
+{
+    // empty
+}
+-----
+function test()
+{
+    // empty
 }

test/code/prettyPrinter/expr/docStrings.test

@@ -0,0 +1,86 @@
+Literals
+-----
+<?php
+
+<<<'STR'
+STR;
+<<<STR
+STR;
+
+<<<'STR'
+A
+B
+STR;
+<<<STR
+A
+B
+STR;
+
+<<<'STR'
+a\nb$c
+STR;
+<<<STR
+a\\nb\$c
+STR;
+
+<<<STR
+a$b
+{$c->d}
+STR;
+
+call(
+    <<<STR
+A
+STR
+    , <<<STR
+B
+STR
+);
+
+function test() {
+    <<<STR
+Foo
+STR;
+    <<<STR
+    Bar
+STR;
+}
+-----
+<<<'STR'
+STR;
+<<<STR
+STR;
+<<<'STR'
+A
+B
+STR;
+<<<STR
+A
+B
+STR;
+<<<'STR'
+a\nb$c
+STR;
+<<<STR
+a\\nb\$c
+STR;
+<<<STR
+a{$b}
+{$c->d}
+STR;
+call(<<<STR
+A
+STR
+, <<<STR
+B
+STR
+);
+function test()
+{
+    <<<STR
+Foo
+STR;
+    <<<STR
+    Bar
+STR;
+}

test/code/prettyPrinter/expr/intrinsics.test

@@ -0,0 +1,29 @@
+isset, empty, unset, exit, die, clone, eval
+-----
+<?php
+
+isset($a, $a[$b]);
+empty($a);
+empty('foo');
+unset($a, $a[$b]);
+exit;
+exit();
+exit(1);
+die;
+die();
+die('foo');
+clone $foo;
+eval('str');
+-----
+isset($a, $a[$b]);
+empty($a);
+empty('foo');
+unset($a, $a[$b]);
+exit;
+exit;
+exit(1);
+die;
+die;
+die('foo');
+clone $foo;
+eval('str');

test/code/prettyPrinter/expr/literals.test

@@ -40,21 +40,17 @@ FALSE;
 378282246310005.0;
 10000000000000002.0;
 
-// strings (normalized to single quoted)
+// strings (single quoted)
 'a';
 'a
 b';
 "a";
 "a\nb";
 'a\'b';
+
+// strings (double quoted)
 "a'b";
 "a\b";
-<<<'STR'
-a\nb$a
-{$b}
-STR;
-
-// strings (normalized to double quoted)
 "$a";
 "a$b";
 "$a$b";
@@ -68,9 +64,6 @@ STR;
 "\\{ $A }";
 "{$$A}[B]";
 "$$A[B]";
-<<<STR
-a\nb$a\n{$b}
-STR;
 
 // make sure indentation doesn't mess anything up
 function foo()
@@ -81,6 +74,12 @@ b';
     'a
     b';
 }
+
+// shell exec (similar to double quoted string)
+`foo`;
+`foo$a`;
+`foo{$a}bar`;
+`\`\'\"`;
 -----
 // magic constants
 __LINE__;
@@ -101,9 +100,9 @@ FALSE;
 // integers (normalized to decimal)
 0;
 11;
-9;
-17;
-3;
+011;
+0x11;
+0b11;
 // floats (normalized to ... something)
 0.0;
 0.0;
@@ -116,19 +115,16 @@ INF;
 1.0E+84;
 378282246310005.0;
 10000000000000002.0;
-// strings (normalized to single quoted)
-'a';
-'a
-b';
+// strings (single quoted)
 'a';
 'a
 b';
+"a";
+"a\nb";
 'a\'b';
-'a\'b';
-'a\\b';
-'a\\nb$a
-{$b}';
-// strings (normalized to double quoted)
+// strings (double quoted)
+"a'b";
+"a\\b";
 "{$a}";
 "a{$b}";
 "{$a}{$b}";
@@ -142,14 +138,17 @@ b';
 "\\{ {$A} }";
 "{${$A}}[B]";
 "\${$A['B']}";
-"a\nb{$a}\n{$b}";
 // make sure indentation doesn't mess anything up
 function foo()
 {
-    'a
-b';
+    "a\nb";
     'a
 b';
     'a
     b';
 }
+// shell exec (similar to double quoted string)
+`foo`;
+`foo{$a}`;
+`foo{$a}bar`;
+`\`\\'\\"`;