Release PHP-Parser 0.9.3

The end attributes previously were always assigned from the last read token,
which does not necessarily correspond to the last token in the reduced rule.
In particular this occurs if the parser read a new token and based on that
lookahead decided to reduce a rule. The behavior was only correct if the
newly read token was first shifted and then the rule was reduced.

This is fixed by buffering the endAttributes of the new token in a temporary
variable and only assigning them once the token is shifted.

CHANGELOG.md

@@ -1,10 +1,51 @@
-Version 0.9.3-dev
+Version 0.9.4-dev
 -----------------
 
 Nothing yet.
 
+Version 0.9.3 (22.11.2012)
+--------------------------
+
+* [BC] As `list()` in `foreach` is now supported the structure of list assignments changed:
+
+  1. There is no longer a dedicated `AssignList` node; instead a normal `Assign` node is used with a `List` as  `var`.
+  2. Nested lists are now `List` nodes too, instead of just arrays.
+
+* [BC] As arbitrary expressions are allowed in `empty()` now its subnode was renamed from `var` to `expr`.
+
+* [BC] The protected `pSafe()` method in `PrettyPrinterAbstract` was renamed to `pNoIndent()`.
+
+* [PHP 5.5] Add support for arbitrary expressions in `empty()`.
+
+* [PHP 5.5] Add support for constant array / string dereferencing.
+  Examples: `"foo"[2]`, `[1, 2, 3][2]`
+
+* [PHP 5.5] Add support for `yield` expressions. This adds a new `Yield` expression type, with subnodes `key` and
+  `value`.
+
+* [PHP 5.5] Add support for `finally`. This adds a new `finallyStmts` subnode to the `TryCatch` node. If there is no
+  finally clause it will be `null`.
+
+* [PHP 5.5] Add support for `list()` destructuring of `foreach` values.
+  Example: `foreach ($coords as list($x, $y)) { ... }`
+
+* Improve pretty printing of expressions by printing less unnecessary parentheses. In particular concatenations are now
+  printed as `$a . $b . $c . $d . $e` rather than `$a . ($b . ($c . ($d . $e)))`. This is implemented by taking operator
+  associativity into account. New protected methods added to the pretty printer are `pPrec()`, `pInfixOp()`,
+  `pPrefixOp()` and `pPostfixOp()`. This also fixes an issue with extraneous parentheses in closure bodies.
+
+* Fix formatting of fall-through `case` statements in the Zend pretty printer.
+
+* Fix parsing of `$foo =& new Bar`. It is now properly parsed as `AssignRef` (instead of `Assign`).
+
+* Fix assignment of `$endAttributes`. Sometimes the attributes of the token right after the node were assigned, rather
+  than the attributes of the last token in the node.
+
+* `rebuildParser.php` is now designed to be run from the command line rather than from the browser.
+
 Version 0.9.2 (07.07.2012)
 --------------------------
+
 * Add `Class->getMethods()` function, which returns all methods contained in the `stmts` array of the class node. This
   does not take inherited methods into account.
 

composer.json

@@ -3,7 +3,7 @@
     "description": "A PHP parser written in PHP",
     "keywords": ["php", "parser"],
     "type": "library",
-    "license": "BSD",
+    "license": "BSD-3-Clause",
     "authors": [
         {
             "name": "Nikita Popov"

doc/0_Introduction.markdown

@@ -12,7 +12,7 @@ application dealing with code programmatically. A parser constructs an [Abstract
 
 There are other ways of dealing with source code. One that PHP supports natively is using the
 token stream generated by [`token_get_all`][2]. The token stream is much more low level than
-the AST and thus has different applications: It allows to also analyize the exact formating of
+the AST and thus has different applications: It allows to also analyze the exact formatting of
 a file. On the other hand the token stream is much harder to deal with for more complex analysis.
 For example an AST abstracts away the fact that in PHP variables can be written as `$foo`, but also
 as `$$bar`, `${'foobar'}` or even `${!${''}=barfoo()}`. You don't have to worry about recognizing
@@ -21,7 +21,7 @@ all the different syntaxes from a stream of tokens.
 Another questions is: Why would I want to have a PHP parser *written in PHP*? Well, PHP might not be
 a language especially suited for fast parsing, but processing the AST is much easier in PHP than it
 would be in other, faster languages like C. Furthermore the people most probably wanting to do
-programmatic PHP code analysis are incidentially PHP developers, not C developers.
+programmatic PHP code analysis are incidentally PHP developers, not C developers.
 
 What can it parse?
 ------------------

doc/1_Installation.markdown

@@ -15,7 +15,7 @@ Create a `composer.json` file in your project root and use it to define your dep
 
     {
         "require": {
-            "nikic/php-parser": "0.9.2"
+            "nikic/php-parser": "0.9.3"
         }
     }
 
@@ -33,7 +33,7 @@ Installing as a PEAR package
 Run the following two commands:
 
     pear channel-discover nikic.github.com/pear
-    pear install channel://nikic.github.com/pear/PHPParser-0.9.2
+    pear install channel://nikic.github.com/pear/PHPParser-0.9.3
 
 Installing as a Git Submodule
 -----------------------------

grammar/README.md

@@ -24,7 +24,7 @@ Building the parser
 ===================
 
 In order to rebuild the parser, you need [moriyoshi's fork of kmyacc](https://github.com/moriyoshi/kmyacc-forked).
-After you compiled/installed it, run the `rebuildParser.php` file.
+After you compiled/installed it, run the `rebuildParser.php` script.
 
-By default only the Parser.php is built. If you want to build the Parser/Debug.php and the y.output
+By default only the `Parser.php` is built. If you want to build the `Parser/Debug.php` and the `y.output` run the
-file you need to call the file with the debug option: `rebuildParser.php?debug`.
+script with `--debug`. If you want to retain the preprocessed grammar pass `--keep-tmp-grammar`.

grammar/kmyacc.php.parser

@@ -47,7 +47,7 @@ class #(-p)
         , "???"
     );
 
-    /* @var Map which translates lexer tokens to internal tokens */
+    /* @var array Map which translates lexer tokens to internal tokens */
     protected static $translate = array(
         #listvar yytranslate
     );
@@ -194,8 +194,11 @@ class #(-p)
                 $yyn = self::$yydefault[$state];
             } else {
                 if ($tokenId === self::TOKEN_NONE) {
-                    // fetch the next token id from the lexer and fetch additional info by-ref
+                    // Fetch the next token id from the lexer and fetch additional info by-ref.
-                    $origTokenId = $this->lexer->getNextToken($tokenValue, $startAttributes, $endAttributes);
+                    // The end attributes are fetched into a temporary variable and only set once the token is really
+                    // shifted (not during read). Otherwise you would sometimes get off-by-one errors, when a rule is
+                    // reduced after a token was read but not yet shifted.
+                    $origTokenId = $this->lexer->getNextToken($tokenValue, $startAttributes, $nextEndAttributes);
 
                     // map the lexer token id to the internally used token id's
                     $tokenId = $origTokenId >= 0 && $origTokenId < self::TOKEN_MAP_SIZE
@@ -241,6 +244,7 @@ class #(-p)
                         $stateStack[$this->stackPos]     = $state = $yyn;
                         $this->yyastk[$this->stackPos]   = $tokenValue;
                         $attributeStack[$this->stackPos] = $startAttributes;
+                        $endAttributes = $nextEndAttributes;
                         $tokenId = self::TOKEN_NONE;
 
                         if ($yyn < self::YYNLSTATES)

grammar/rebuildParser.php

@@ -1,8 +1,21 @@
 <?php
 
-const GRAMMAR_FILE = './zend_language_parser.phpy';
+$grammarFile           = __DIR__ . '/zend_language_parser.phpy';
-const TMP_FILE     = './tmp_parser.phpy';
+$skeletonFile          = __DIR__ . '/kmyacc.php.parser';
-const RESULT_FILE  = './tmp_parser.php';
+$tmpGrammarFile        = __DIR__ . '/tmp_parser.phpy';
+$tmpResultFile         = __DIR__ . '/tmp_parser.php';
+$parserResultFile      = __DIR__ . '/../lib/PHPParser/Parser.php';
+$debugParserResultFile = __DIR__ . '/../lib/PHPParser/Parser/Debug.php';
+
+// check for kmyacc.exe binary in this directory, otherwise fall back to global name
+$kmyacc = __DIR__ . '/kmyacc.exe';
+if (!file_exists($kmyacc)) {
+    $kmyacc = 'kmyacc';
+}
+
+$options = array_flip($argv);
+$optionDebug = isset($options['--debug']);
+$optionKeepTmpGrammar = isset($options['--keep-tmp-grammar']);
 
 ///////////////////////////////
 /// Utility regex constants ///
@@ -23,42 +36,34 @@ const ARGS   = '\((?<args>[^()]*+(?:\((?&args)\)[^()]*+)*+)\)';
 /// Main script ///
 ///////////////////
 
-echo '<pre>';
-
 echo 'Building temporary preproprocessed grammar file.', "\n";
 
-$grammarCode = file_get_contents(GRAMMAR_FILE);
+$grammarCode = file_get_contents($grammarFile);
 
 $grammarCode = resolveConstants($grammarCode);
 $grammarCode = resolveNodes($grammarCode);
 $grammarCode = resolveMacros($grammarCode);
 $grammarCode = resolveArrays($grammarCode);
 
-file_put_contents(TMP_FILE, $grammarCode);
+file_put_contents($tmpGrammarFile, $grammarCode);
 
-echo 'Building parser. Output: "',
+echo "Building parser.\n";
-     trim(shell_exec('kmyacc -l -m kmyacc.php.parser -p PHPParser_Parser ' . TMP_FILE . ' 2>&1')),
+$output = trim(shell_exec("$kmyacc -l -m $skeletonFile -p PHPParser_Parser $tmpGrammarFile 2>&1"));
-     '"', "\n";
+echo "Output: \"$output\"\n";
 
-rename(RESULT_FILE, '../lib/PHPParser/Parser.php');
+moveFileWithDirCheck($tmpResultFile, $parserResultFile);
 
-if (isset($_GET['debug'])) {
+if ($optionDebug) {
-    echo 'Building debug parser. Output: "',
+    echo "Building debug parser.\n";
-    trim(shell_exec('kmyacc -t -v -l -m kmyacc.php.parser -p PHPParser_Parser ' . TMP_FILE . ' 2>&1')),
+    $output = trim(shell_exec("$kmyacc -t -v -l -m $skeletonFile -p PHPParser_Parser $tmpGrammarFile 2>&1"));
-    '"', "\n";
+    echo "Output: \"$output\"\n";
 
-    if (!is_dir('../lib/PHPParser/Parser')) {
+    moveFileWithDirCheck($tmpResultFile, $debugParserResultFile);
-        mkdir('../lib/PHPParser/Parser');
-    }
-    rename(RESULT_FILE, '../lib/PHPParser/Parser/Debug.php');
 }
 
-
+if (!$optionKeepTmpGrammar) {
-unlink(TMP_FILE);
+    unlink($tmpGrammarFile);
-
+}
-echo 'The following temporary preproprocessed grammar file was used:', "\n", $grammarCode;
-
-echo '</pre>';
 
 ///////////////////////////////
 /// Preprocessing functions ///
@@ -193,6 +198,14 @@ function resolveArrays($code) {
     );
 }
 
+function moveFileWithDirCheck($fromPath, $toPath) {
+    $dir = dirname($toPath);
+    if (!is_dir($dir)) {
+        mkdir($dir, 0777, true);
+    }
+    rename($fromPath, $toPath);
+}
+
 //////////////////////////////
 /// Regex helper functions ///
 //////////////////////////////

grammar/zend_language_parser.phpy

@@ -7,6 +7,7 @@
 %left T_LOGICAL_XOR
 %left T_LOGICAL_AND
 %right T_PRINT
+%right T_YIELD
 %left '=' T_PLUS_EQUAL T_MINUS_EQUAL T_MUL_EQUAL T_DIV_EQUAL T_CONCAT_EQUAL T_MOD_EQUAL T_AND_EQUAL T_OR_EQUAL T_XOR_EQUAL T_SL_EQUAL T_SR_EQUAL
 %left '?' ':'
 %left T_BOOLEAN_OR
@@ -63,6 +64,7 @@
 %token T_RETURN
 %token T_TRY
 %token T_CATCH
+%token T_FINALLY
 %token T_THROW
 %token T_USE
 %token T_INSTEADOF
@@ -169,42 +171,43 @@ inner_statement:
 
 statement:
       '{' inner_statement_list '}'                          { $$ = $2; }
-    | T_IF '(' expr ')' statement elseif_list else_single   { $$ = Stmt_If[$3, [stmts: toArray($5), elseifs: $6, else: $7]]; }
+    | T_IF parentheses_expr statement elseif_list else_single
-    | T_IF '(' expr ')' ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
+          { $$ = Stmt_If[$2, [stmts: toArray($3), elseifs: $4, else: $5]]; }
-          { $$ = Stmt_If[$3, [stmts: $6, elseifs: $7, else: $8]]; }
+    | T_IF parentheses_expr ':' inner_statement_list new_elseif_list new_else_single T_ENDIF ';'
-    | T_WHILE '(' expr ')' while_statement                  { $$ = Stmt_While[$3, $5]; }
+          { $$ = Stmt_If[$2, [stmts: $4, elseifs: $5, else: $6]]; }
-    | T_DO statement T_WHILE '(' expr ')' ';'               { $$ = Stmt_Do   [$5, toArray($2)]; }
+    | T_WHILE parentheses_expr while_statement              { $$ = Stmt_While[$2, $3]; }
+    | T_DO statement T_WHILE parentheses_expr ';'           { $$ = Stmt_Do   [$4, toArray($2)]; }
     | T_FOR '(' for_expr ';'  for_expr ';' for_expr ')' for_statement
           { $$ = Stmt_For[[init: $3, cond: $5, loop: $7, stmts: $9]]; }
-    | T_SWITCH '(' expr ')' switch_case_list                { $$ = Stmt_Switch[$3, $5]; }
+    | T_SWITCH parentheses_expr switch_case_list            { $$ = Stmt_Switch[$2, $3]; }
     | T_BREAK ';'                                           { $$ = Stmt_Break[null]; }
     | T_BREAK expr ';'                                      { $$ = Stmt_Break[$2]; }
     | T_CONTINUE ';'                                        { $$ = Stmt_Continue[null]; }
     | T_CONTINUE expr ';'                                   { $$ = Stmt_Continue[$2]; }
     | T_RETURN ';'                                          { $$ = Stmt_Return[null]; }
     | T_RETURN expr ';'                                     { $$ = Stmt_Return[$2]; }
+    | yield_expr ';'                                        { $$ = $1; }
     | T_GLOBAL global_var_list ';'                          { $$ = Stmt_Global[$2]; }
     | T_STATIC static_var_list ';'                          { $$ = Stmt_Static[$2]; }
     | T_ECHO expr_list ';'                                  { $$ = Stmt_Echo[$2]; }
     | T_INLINE_HTML                                         { $$ = Stmt_InlineHTML[$1]; }
     | expr ';'                                              { $$ = $1; }
     | T_UNSET '(' variables_list ')' ';'                    { $$ = Stmt_Unset[$3]; }
-    | T_FOREACH '(' expr T_AS variable ')' foreach_statement
+    | T_FOREACH '(' expr T_AS foreach_variable ')' foreach_statement
-          { $$ = Stmt_Foreach[$3, $5, [keyVar: null, byRef: false, stmts: $7]]; }
+          { $$ = Stmt_Foreach[$3, $5[0], [keyVar: null, byRef: $5[1], stmts: $7]]; }
-    | T_FOREACH '(' expr T_AS '&' variable ')' foreach_statement
+    | T_FOREACH '(' expr T_AS variable T_DOUBLE_ARROW foreach_variable ')' foreach_statement
-          { $$ = Stmt_Foreach[$3, $6, [keyVar: null, byRef: true, stmts: $8]]; }
+          { $$ = Stmt_Foreach[$3, $7[0], [keyVar: $5, byRef: $7[1], stmts: $9]]; }
-    | T_FOREACH '(' expr T_AS variable T_DOUBLE_ARROW optional_ref variable ')' foreach_statement
-          { $$ = Stmt_Foreach[$3, $8, [keyVar: $5, byRef: $7, stmts: $10]]; }
     | T_DECLARE '(' declare_list ')' declare_statement      { $$ = Stmt_Declare[$3, $5]; }
     | ';'                                                   { $$ = array(); /* means: no statement */ }
-    | T_TRY '{' inner_statement_list '}' catches            { $$ = Stmt_TryCatch[$3, $5]; }
+    | T_TRY '{' inner_statement_list '}' catches optional_finally
+          { $$ = Stmt_TryCatch[$3, $5, $6]; }
     | T_THROW expr ';'                                      { $$ = Stmt_Throw[$2]; }
     | T_GOTO T_STRING ';'                                   { $$ = Stmt_Goto[$2]; }
     | T_STRING ':'                                          { $$ = Stmt_Label[$1]; }
 ;
 
 catches:
-      catch                                                 { init($1); }
+      /* empty */                                           { init(); }
     | catches catch                                         { push($1, $2); }
 ;
 
@@ -213,6 +216,11 @@ catch:
         { $$ = Stmt_Catch[$3, parseVar($4), $7]; }
 ;
 
+optional_finally:
+      /* empty */                                           { $$ = null; }
+    | T_FINALLY '{' inner_statement_list '}'                { $$ = $3; }
+;
+
 variables_list:
       variable                                              { init($1); }
     | variables_list ',' variable                           { push($1, $3); }
@@ -320,7 +328,7 @@ elseif_list:
 ;
 
 elseif:
-      T_ELSEIF '(' expr ')' statement                       { $$ = Stmt_ElseIf[$3, toArray($5)]; }
+      T_ELSEIF parentheses_expr statement                   { $$ = Stmt_ElseIf[$2, toArray($3)]; }
 ;
 
 new_elseif_list:
@@ -329,7 +337,7 @@ new_elseif_list:
 ;
 
 new_elseif:
-     T_ELSEIF '(' expr ')' ':' inner_statement_list         { $$ = Stmt_ElseIf[$3, $6]; }
+     T_ELSEIF parentheses_expr ':' inner_statement_list     { $$ = Stmt_ElseIf[$2, $4]; }
 ;
 
 else_single:
@@ -342,6 +350,12 @@ new_else_single:
     | T_ELSE ':' inner_statement_list                       { $$ = Stmt_Else[$3]; }
 ;
 
+foreach_variable:
+      variable                                              { $$ = array($1, false); }
+    | '&' variable                                          { $$ = array($2, true); }
+    | list_expr                                             { $$ = array($1, false); }
+;
+
 parameter_list:
       non_empty_parameter_list                              { $$ = $1; }
     | /* empty */                                           { $$ = array(); }
@@ -367,8 +381,9 @@ optional_class_type:
 ;
 
 argument_list:
-      non_empty_argument_list                               { $$ = $1; }
+      '(' ')'                                               { $$ = array(); }
-    | /* empty */                                           { $$ = array(); }
+    | '(' non_empty_argument_list ')'                       { $$ = $2; }
+    | '(' yield_expr ')'                                    { $$ = array(Arg[$2, false]); }
 ;
 
 non_empty_argument_list:
@@ -495,10 +510,10 @@ for_expr:
 
 expr:
       variable                                              { $$ = $1; }
-    | T_LIST '(' assignment_list ')' '=' expr               { $$ = Expr_AssignList[$3, $6]; }
+    | list_expr '=' expr                                    { $$ = Expr_Assign[$1, $3]; }
     | variable '=' expr                                     { $$ = Expr_Assign[$1, $3]; }
     | variable '=' '&' variable                             { $$ = Expr_AssignRef[$1, $4]; }
-    | variable '=' '&' new_expr                             { $$ = Expr_Assign[$1, $4]; } /* reference dropped intentially */
+    | variable '=' '&' new_expr                             { $$ = Expr_AssignRef[$1, $4]; }
     | new_expr                                              { $$ = $1; }
     | T_CLONE expr                                          { $$ = Expr_Clone[$2]; }
     | variable T_PLUS_EQUAL expr                            { $$ = Expr_AssignPlus      [$1, $3]; }
@@ -545,16 +560,16 @@ expr:
     | expr '>' expr                                         { $$ = Expr_Greater       [$1, $3]; }
     | expr T_IS_GREATER_OR_EQUAL expr                       { $$ = Expr_GreaterOrEqual[$1, $3]; }
     | expr T_INSTANCEOF class_name_reference                { $$ = Expr_Instanceof    [$1, $3]; }
-    | '(' expr ')'                                          { $$ = $2; }
+    | parentheses_expr                                      { $$ = $1; }
     /* we need a separate '(' new_expr ')' rule to avoid problems caused by a s/r conflict */
     | '(' new_expr ')'                                      { $$ = $2; }
     | expr '?' expr ':' expr                                { $$ = Expr_Ternary[$1, $3,   $5]; }
     | expr '?' ':' expr                                     { $$ = Expr_Ternary[$1, null, $4]; }
     | T_ISSET '(' variables_list ')'                        { $$ = Expr_Isset[$3]; }
-    | T_EMPTY '(' variable ')'                              { $$ = Expr_Empty[$3]; }
+    | T_EMPTY '(' expr ')'                                  { $$ = Expr_Empty[$3]; }
     | T_INCLUDE expr                                        { $$ = Expr_Include[$2, Expr_Include::TYPE_INCLUDE]; }
     | T_INCLUDE_ONCE expr                                   { $$ = Expr_Include[$2, Expr_Include::TYPE_INCLUDE_ONCE]; }
-    | T_EVAL '(' expr ')'                                   { $$ = Expr_Eval[$3]; }
+    | T_EVAL parentheses_expr                               { $$ = Expr_Eval[$2]; }
     | T_REQUIRE expr                                        { $$ = Expr_Include[$2, Expr_Include::TYPE_REQUIRE]; }
     | T_REQUIRE_ONCE expr                                   { $$ = Expr_Include[$2, Expr_Include::TYPE_REQUIRE_ONCE]; }
     | T_INT_CAST expr                                       { $$ = Expr_Cast_Int     [$2]; }
@@ -567,16 +582,40 @@ expr:
     | T_EXIT exit_expr                                      { $$ = Expr_Exit         [$2]; }
     | '@' expr                                              { $$ = Expr_ErrorSuppress[$2]; }
     | scalar                                                { $$ = $1; }
-    | T_ARRAY '(' array_pair_list ')'                       { $$ = Expr_Array[$3]; }
+    | array_expr                                            { $$ = $1; }
-    | '[' array_pair_list ']'                               { $$ = Expr_Array[$2]; }
+    | scalar_dereference                                    { $$ = $1; }
     | '`' backticks_expr '`'                                { $$ = Expr_ShellExec[$2]; }
     | T_PRINT expr                                          { $$ = Expr_Print[$2]; }
+    | T_YIELD                                               { $$ = Expr_Yield[null, null]; }
     | T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars '{' inner_statement_list '}'
           { $$ = Expr_Closure[[static: false, byRef: $2, params: $4, uses: $6, stmts: $8]]; }
     | T_STATIC T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars '{' inner_statement_list '}'
           { $$ = Expr_Closure[[static: true, byRef: $3, params: $5, uses: $7, stmts: $9]]; }
 ;
 
+parentheses_expr:
+      '(' expr ')'                                          { $$ = $2; }
+    | '(' yield_expr ')'                                    { $$ = $2; }
+;
+
+yield_expr:
+      T_YIELD expr                                          { $$ = Expr_Yield[$2, null]; }
+    | T_YIELD expr T_DOUBLE_ARROW expr                      { $$ = Expr_Yield[$4, $2]; }
+;
+
+array_expr:
+      T_ARRAY '(' array_pair_list ')'                       { $$ = Expr_Array[$3]; }
+    | '[' array_pair_list ']'                               { $$ = Expr_Array[$2]; }
+;
+
+scalar_dereference:
+      array_expr '[' dim_offset ']'                         { $$ = Expr_ArrayDimFetch[$1, $3]; }
+    | T_CONSTANT_ENCAPSED_STRING '[' dim_offset ']'
+          { $$ = Expr_ArrayDimFetch[Scalar_String[Scalar_String::parse($1)], $3]; }
+    | scalar_dereference '[' dim_offset ']'                 { $$ = Expr_ArrayDimFetch[$1, $3]; }
+    /* alternative array syntax missing intentionally */
+;
+
 new_expr:
       T_NEW class_name_reference ctor_arguments             { $$ = Expr_New[$2, $3]; }
 ;
@@ -596,28 +635,28 @@ lexical_var:
 ;
 
 function_call:
-      name '(' argument_list ')'                            { $$ = Expr_FuncCall[$1, $3]; }
+      name argument_list                                    { $$ = Expr_FuncCall[$1, $2]; }
-    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM T_STRING '(' argument_list ')'
+    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM T_STRING argument_list
-          { $$ = Expr_StaticCall[$1, $3, $5]; }
+          { $$ = Expr_StaticCall[$1, $3, $4]; }
-    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM '{' expr '}' '(' argument_list ')'
+    | class_name_or_var T_PAAMAYIM_NEKUDOTAYIM '{' expr '}' argument_list
-          { $$ = Expr_StaticCall[$1, $4, $7]; }
+          { $$ = Expr_StaticCall[$1, $4, $6]; }
-    | static_property '(' argument_list ')' {
+    | static_property argument_list {
             if ($1 instanceof PHPParser_Node_Expr_StaticPropertyFetch) {
-                $$ = Expr_StaticCall[$1->class, Expr_Variable[$1->name], $3];
+                $$ = Expr_StaticCall[$1->class, Expr_Variable[$1->name], $2];
             } elseif ($1 instanceof PHPParser_Node_Expr_ArrayDimFetch) {
                 $tmp = $1;
                 while ($tmp->var instanceof PHPParser_Node_Expr_ArrayDimFetch) {
                     $tmp = $tmp->var;
                 }
 
-                $$ = Expr_StaticCall[$tmp->var->class, $1, $3];
+                $$ = Expr_StaticCall[$tmp->var->class, $1, $2];
                 $tmp->var = Expr_Variable[$tmp->var->name];
             } else {
                 throw new Exception;
             }
           }
-    | variable_without_objects '(' argument_list ')'
+    | variable_without_objects argument_list
-          { $$ = Expr_FuncCall[$1, $3]; }
+          { $$ = Expr_FuncCall[$1, $2]; }
     | function_call '[' dim_offset ']'                      { $$ = Expr_ArrayDimFetch[$1, $3]; }
       /* alternative array syntax missing intentionally */
 ;
@@ -660,7 +699,7 @@ object_access_for_dcnr:
 exit_expr:
       /* empty */                                           { $$ = null; }
     | '(' ')'                                               { $$ = null; }
-    | '(' expr ')'                                          { $$ = $2; }
+    | parentheses_expr                                      { $$ = $1; }
 ;
 
 backticks_expr:
@@ -671,13 +710,13 @@ backticks_expr:
 
 ctor_arguments:
       /* empty */                                           { $$ = array(); }
-    | '(' argument_list ')'                                 { $$ = $2; }
+    | argument_list                                         { $$ = $1; }
 ;
 
 common_scalar:
       T_LNUMBER                                             { $$ = Scalar_LNumber[Scalar_LNumber::parse($1)]; }
     | T_DNUMBER                                             { $$ = Scalar_DNumber[Scalar_DNumber::parse($1)]; }
-    | T_CONSTANT_ENCAPSED_STRING                            { $$ = Scalar_String::create($1, $attributes); }
+    | T_CONSTANT_ENCAPSED_STRING                            { $$ = Scalar_String[Scalar_String::parse($1)]; }
     | T_LINE                                                { $$ = Scalar_LineConst[]; }
     | T_FILE                                                { $$ = Scalar_FileConst[]; }
     | T_DIR                                                 { $$ = Scalar_DirConst[]; }
@@ -747,9 +786,9 @@ new_expr_array_deref:
 object_access:
       variable_or_new_expr T_OBJECT_OPERATOR object_property
           { $$ = Expr_PropertyFetch[$1, $3]; }
-    | variable_or_new_expr T_OBJECT_OPERATOR object_property '(' argument_list ')'
+    | variable_or_new_expr T_OBJECT_OPERATOR object_property argument_list
-          { $$ = Expr_MethodCall[$1, $3, $5]; }
+          { $$ = Expr_MethodCall[$1, $3, $4]; }
-    | object_access '(' argument_list ')'                   { $$ = Expr_FuncCall[$1, $3]; }
+    | object_access argument_list                           { $$ = Expr_FuncCall[$1, $2]; }
     | object_access '[' dim_offset ']'                      { $$ = Expr_ArrayDimFetch[$1, $3]; }
     | object_access '{' expr '}'                            { $$ = Expr_ArrayDimFetch[$1, $3]; }
 ;
@@ -802,14 +841,18 @@ object_property:
     | variable_without_objects                              { $$ = $1; }
 ;
 
-assignment_list:
+list_expr:
-      assignment_list ',' assignment_list_element           { push($1, $3); }
+      T_LIST '(' list_expr_elements ')'                     { $$ = Expr_List[$3]; }
-    | assignment_list_element                               { init($1); }
 ;
 
-assignment_list_element:
+list_expr_elements:
+      list_expr_elements ',' list_expr_element              { push($1, $3); }
+    | list_expr_element                                     { init($1); }
+;
+
+list_expr_element:
       variable                                              { $$ = $1; }
-    | T_LIST '(' assignment_list ')'                        { $$ = $3; }
+    | list_expr                                             { $$ = $1; }
     | /* empty */                                           { $$ = null; }
 ;
 

lib/PHPParser/BuilderAbstract.php

@@ -35,7 +35,7 @@ abstract class PHPParser_BuilderAbstract implements PHPParser_Builder {
 
     /**
      * Normalizes a value: Converts nulls, booleans, integers,
-     * floats and strings into their respective nodes
+     * floats, strings and arrays into their respective nodes
      *
      * @param mixed $value The value to normalize
      *

lib/PHPParser/Comment.php

@@ -46,7 +46,7 @@ class PHPParser_Comment
     /**
      * Sets the line number the comment started on.
      *
-     * @param int Line number
+     * @param int $line Line number
      */
     public function setLine($line) {
         $this->line = $line;

lib/PHPParser/Lexer/Emulative.php

@@ -11,31 +11,33 @@ class PHPParser_Lexer_Emulative extends PHPParser_Lexer
     public function __construct() {
         parent::__construct();
 
+        $newKeywordsPerVersion = array(
+            '5.5.0-dev' => array(
+                'finally'       => PHPParser_Parser::T_FINALLY,
+                'yield'         => PHPParser_Parser::T_YIELD,
+            ),
+            '5.4.0-dev' => array(
+                'callable'      => PHPParser_Parser::T_CALLABLE,
+                'insteadof'     => PHPParser_Parser::T_INSTEADOF,
+                'trait'         => PHPParser_Parser::T_TRAIT,
+                '__trait__'     => PHPParser_Parser::T_TRAIT_C,
+            ),
+            '5.3.0-dev' => array(
+                '__dir__'       => PHPParser_Parser::T_DIR,
+                'goto'          => PHPParser_Parser::T_GOTO,
+                'namespace'     => PHPParser_Parser::T_NAMESPACE,
+                '__namespace__' => PHPParser_Parser::T_NS_C,
+            ),
+        );
+
         $this->newKeywords = array();
+        foreach ($newKeywordsPerVersion as $version => $newKeywords) {
+            if (version_compare(PHP_VERSION, $version, '>=')) {
+                break;
+            }
 
-        if (version_compare(PHP_VERSION, '5.4.0RC1', '>=')) {
+            $this->newKeywords += $newKeywords;
-            return;
         }
-
-        // new PHP 5.4 keywords
-        $this->newKeywords += array(
-            'callable'  => PHPParser_Parser::T_CALLABLE,
-            'insteadof' => PHPParser_Parser::T_INSTEADOF,
-            'trait'     => PHPParser_Parser::T_TRAIT,
-            '__trait__' => PHPParser_Parser::T_TRAIT_C,
-        );
-
-        if (version_compare(PHP_VERSION, '5.3.0', '>=')) {
-            return;
-        }
-
-        // new PHP 5.3 keywords
-        $this->newKeywords += array(
-            '__dir__'       => PHPParser_Parser::T_DIR,
-            'goto'          => PHPParser_Parser::T_GOTO,
-            'namespace'     => PHPParser_Parser::T_NAMESPACE,
-            '__namespace__' => PHPParser_Parser::T_NS_C,
-        );
     }
 
     public function startLexing($code) {

lib/PHPParser/Node/Expr/AssignList.php

@@ -1,25 +0,0 @@
-<?php
-
-/**
- * @property array               $vars List of variables to assign to
- * @property PHPParser_Node_Expr $expr Expression
- */
-class PHPParser_Node_Expr_AssignList extends PHPParser_Node_Expr
-{
-    /**
-     * Constructs a list() assignment node.
-     *
-     * @param array               $vars       List of variables to assign to
-     * @param PHPParser_Node_Expr $expr       Expression
-     * @param array               $attributes Additional attributes
-     */
-    public function __construct(array $vars, PHPParser_Node_Expr $expr, array $attributes = array()) {
-        parent::__construct(
-            array(
-                'vars' => $vars,
-                'expr' => $expr
-            ),
-            $attributes
-        );
-    }
-}

lib/PHPParser/Node/Expr/Closure.php

@@ -2,7 +2,7 @@
 
 /**
  * @property PHPParser_Node[]                 $stmts  Statements
- * @property PHPParser_Node_Stmt_FuncParam[]  $params Parameters
+ * @property PHPParser_Node_Param[]           $params Parameters
  * @property PHPParser_Node_Expr_ClosureUse[] $uses   use()s
  * @property bool                             $byRef  Whether to return by reference
  * @property bool                             $static Whether the closure is static

lib/PHPParser/Node/Expr/Empty.php

@@ -1,20 +1,20 @@
 <?php
 
 /**
- * @property PHPParser_Node_Expr $var Variable
+ * @property PHPParser_Node_Expr $expr Expression
  */
 class PHPParser_Node_Expr_Empty extends PHPParser_Node_Expr
 {
     /**
      * Constructs an empty() node.
      *
-     * @param PHPParser_Node_Expr $var        Variable
+     * @param PHPParser_Node_Expr $expr       Expression
      * @param array               $attributes Additional attributes
      */
-    public function __construct(PHPParser_Node_Expr $var, array $attributes = array()) {
+    public function __construct(PHPParser_Node_Expr $expr, array $attributes = array()) {
         parent::__construct(
             array(
-                'var' => $var
+                'expr' => $expr
             ),
             $attributes
         );

lib/PHPParser/Node/Expr/List.php

@@ -0,0 +1,22 @@
+<?php
+
+/**
+ * @property array $vars List of variables to assign to
+ */
+class PHPParser_Node_Expr_List extends PHPParser_Node_Expr
+{
+    /**
+     * Constructs a list() destructuring node.
+     *
+     * @param array $vars       List of variables to assign to
+     * @param array $attributes Additional attributes
+     */
+    public function __construct(array $vars, array $attributes = array()) {
+        parent::__construct(
+            array(
+                'vars' => $vars,
+            ),
+            $attributes
+        );
+    }
+}

lib/PHPParser/Node/Expr/Yield.php

@@ -0,0 +1,25 @@
+<?php
+
+/**
+ * @property null|PHPParser_Node_Expr $value Value expression
+ * @property null|PHPParser_Node_Expr $key   Key expression
+ */
+class PHPParser_Node_Expr_Yield extends PHPParser_Node_Expr
+{
+    /**
+     * Constructs a yield expression node.
+     *
+     * @param null|PHPParser_Node_Expr $value ´    Value expression
+     * @param null|PHPParser_Node_Expr $key        Key expression
+     * @param array                    $attributes Additional attributes
+     */
+    public function __construct(PHPParser_Node_Expr $value = null, PHPParser_Node_Expr $key = null, array $attributes = array()) {
+        parent::__construct(
+            array(
+                'key'   => $key,
+                'value' => $value,
+            ),
+            $attributes
+        );
+    }
+}

lib/PHPParser/Node/Scalar/DNumber.php

@@ -1,7 +1,7 @@
 <?php
 
 /**
- * @property double $value Number value
+ * @property float $value Number value
  */
 class PHPParser_Node_Scalar_DNumber extends PHPParser_Node_Scalar
 {

lib/PHPParser/Node/Scalar/String.php

@@ -32,30 +32,27 @@ class PHPParser_Node_Scalar_String extends PHPParser_Node_Scalar
     }
 
     /**
-     * Creates a String node from a string token (parses escape sequences).
+     * Parses a string token.
      *
-     * @param string $str        String
+     * @param string $str String token content
-     * @param array  $attributes Additional attributes
      *
-     * @return PHPParser_Node_Scalar_String String Node
+     * @return string The parsed string
      */
-    public static function create($str, array $attributes = array()) {
+    public static function parse($str) {
         $bLength = 0;
         if ('b' === $str[0]) {
             $bLength = 1;
         }
 
         if ('\'' === $str[$bLength]) {
-            $str = str_replace(
+            return str_replace(
                 array('\\\\', '\\\''),
                 array(  '\\',   '\''),
                 substr($str, $bLength + 1, -1)
             );
         } else {
-            $str = self::parseEscapeSequences(substr($str, $bLength + 1, -1), '"');
+            return self::parseEscapeSequences(substr($str, $bLength + 1, -1), '"');
         }
-
-        return new self($str, $attributes);
     }
 
     /**

lib/PHPParser/Node/Stmt/Case.php

@@ -16,8 +16,8 @@ class PHPParser_Node_Stmt_Case extends PHPParser_Node_Stmt
     public function __construct($cond, array $stmts = array(), array $attributes = array()) {
         parent::__construct(
             array(
-                'stmts' => $stmts,
                 'cond'  => $cond,
+                'stmts' => $stmts,
             ),
             $attributes
         );

lib/PHPParser/Node/Stmt/Static.php

@@ -1,15 +1,15 @@
 <?php
 
 /**
- * @property PHPParser_Node_Stmts_StaticVar[] $vars Variable definitions
+ * @property PHPParser_Node_Stmt_StaticVar[] $vars Variable definitions
  */
 class PHPParser_Node_Stmt_Static extends PHPParser_Node_Stmt
 {
     /**
      * Constructs a static variables list node.
      *
-     * @param PHPParser_Node_Stmts_StaticVar[] $vars       Variable definitions
+     * @param PHPParser_Node_Stmt_StaticVar[] $vars       Variable definitions
-     * @param array                            $attributes Additional attributes
+     * @param array                           $attributes Additional attributes
      */
     public function __construct(array $vars, array $attributes = array()) {
         parent::__construct(

lib/PHPParser/Node/Stmt/TraitUse.php

@@ -1,17 +1,17 @@
 <?php
 
 /**
- * @property PHPParser_Node_Name[]               $traits      Traits
+ * @property PHPParser_Node_Name[]                    $traits      Traits
- * @property PHPParser_Node_TraitUseAdaptation[] $adaptations Adaptations
+ * @property PHPParser_Node_Stmt_TraitUseAdaptation[] $adaptations Adaptations
  */
 class PHPParser_Node_Stmt_TraitUse extends PHPParser_Node_Stmt
 {
     /**
      * Constructs a trait use node.
      *
-     * @param PHPParser_Node_Name[]               $traits      Traits
+     * @param PHPParser_Node_Name[]                    $traits      Traits
-     * @param PHPParser_Node_TraitUseAdaptation[] $adaptations Adaptations
+     * @param PHPParser_Node_Stmt_TraitUseAdaptation[] $adaptations Adaptations
-     * @param array                               $attributes  Additional attributes
+     * @param array                                    $attributes  Additional attributes
      */
     public function __construct(array $traits, array $adaptations = array(), array $attributes = array()) {
         parent::__construct(

lib/PHPParser/Node/Stmt/TryCatch.php

@@ -1,23 +1,30 @@
 <?php
 
 /**
- * @property PHPParser_Node[]            $stmts   Statements
+ * @property PHPParser_Node[]            $stmts        Statements
- * @property PHPParser_Node_Stmt_Catch[] $catches Catches
+ * @property PHPParser_Node_Stmt_Catch[] $catches      Catches
+ * @property PHPParser_Node[]            $finallyStmts Finally statements
  */
 class PHPParser_Node_Stmt_TryCatch extends PHPParser_Node_Stmt
 {
     /**
      * Constructs a try catch node.
      *
-     * @param PHPParser_Node[]            $stmts      Statements
+     * @param PHPParser_Node[]            $stmts        Statements
-     * @param PHPParser_Node_Stmt_Catch[] $catches    Catches
+     * @param PHPParser_Node_Stmt_Catch[] $catches      Catches
-     * @param array                       $attributes Additional attributes
+     * @param PHPParser_Node[]            $finallyStmts Finally statements (null means no finally clause)
+     * @param array|null                  $attributes   Additional attributes
      */
-    public function __construct(array $stmts, array $catches, array $attributes = array()) {
+    public function __construct(array $stmts, array $catches, array $finallyStmts = null, array $attributes = array()) {
+        if (empty($catches) && null === $finallyStmts) {
+            throw new PHPParser_Error('Cannot use try without catch or finally');
+        }
+
         parent::__construct(
             array(
-                'stmts'   => $stmts,
+                'stmts'        => $stmts,
-                'catches' => $catches,
+                'catches'      => $catches,
+                'finallyStmts' => $finallyStmts,
             ),
             $attributes
         );

lib/PHPParser/Node/Stmt/While.php

@@ -2,7 +2,7 @@
 
 /**
  * @property PHPParser_Node_Expr $cond  Condition
- * @property PHPParsre_Node[]    $stmts Statements
+ * @property PHPParser_Node[]    $stmts Statements
  */
 class PHPParser_Node_Stmt_While extends PHPParser_Node_Stmt
 {

lib/PHPParser/PrettyPrinter/Zend.php

@@ -70,7 +70,7 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
     // Scalars
 
     public function pScalar_String(PHPParser_Node_Scalar_String $node) {
-        return '\'' . $this->pSafe(addcslashes($node->value, '\'\\')) . '\'';
+        return '\'' . $this->pNoIndent(addcslashes($node->value, '\'\\')) . '\'';
     }
 
     public function pScalar_Encapsed(PHPParser_Node_Scalar_Encapsed $node) {
@@ -91,229 +91,225 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
     // Assignments
 
     public function pExpr_Assign(PHPParser_Node_Expr_Assign $node) {
-        return $this->p($node->var) . ' = ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_Assign', $node->var, ' = ', $node->expr);
     }
 
     public function pExpr_AssignRef(PHPParser_Node_Expr_AssignRef $node) {
-        return $this->p($node->var) . ' =& ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignRef', $node->var, ' =& ', $node->expr);
     }
 
     public function pExpr_AssignPlus(PHPParser_Node_Expr_AssignPlus $node) {
-        return $this->p($node->var) . ' += ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignPlus', $node->var, ' += ', $node->expr);
     }
 
     public function pExpr_AssignMinus(PHPParser_Node_Expr_AssignMinus $node) {
-        return $this->p($node->var) . ' -= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignMinus', $node->var, ' -= ', $node->expr);
     }
 
     public function pExpr_AssignMul(PHPParser_Node_Expr_AssignMul $node) {
-        return $this->p($node->var) . ' *= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignMul', $node->var, ' *= ', $node->expr);
     }
 
     public function pExpr_AssignDiv(PHPParser_Node_Expr_AssignDiv $node) {
-        return $this->p($node->var) . ' /= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignDiv', $node->var, ' /= ', $node->expr);
     }
 
     public function pExpr_AssignConcat(PHPParser_Node_Expr_AssignConcat $node) {
-        return $this->p($node->var) . ' .= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignConcat', $node->var, ' .= ', $node->expr);
     }
 
     public function pExpr_AssignMod(PHPParser_Node_Expr_AssignMod $node) {
-        return $this->p($node->var) . ' %= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignMod', $node->var, ' %= ', $node->expr);
     }
 
     public function pExpr_AssignBitwiseAnd(PHPParser_Node_Expr_AssignBitwiseAnd $node) {
-        return $this->p($node->var) . ' &= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignBitwiseAnd', $node->var, ' &= ', $node->expr);
     }
 
     public function pExpr_AssignBitwiseOr(PHPParser_Node_Expr_AssignBitwiseOr $node) {
-        return $this->p($node->var) . ' |= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignBitwiseOr', $node->var, ' |= ', $node->expr);
     }
 
     public function pExpr_AssignBitwiseXor(PHPParser_Node_Expr_AssignBitwiseXor $node) {
-        return $this->p($node->var) . ' ^= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignBitwiseXor', $node->var, ' ^= ', $node->expr);
     }
 
     public function pExpr_AssignShiftLeft(PHPParser_Node_Expr_AssignShiftLeft $node) {
-        return $this->p($node->var) . ' <<= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignShiftLeft', $node->var, ' <<= ', $node->expr);
     }
 
     public function pExpr_AssignShiftRight(PHPParser_Node_Expr_AssignShiftRight $node) {
-        return $this->p($node->var) . ' >>= ' . $this->p($node->expr);
+        return $this->pInfixOp('Expr_AssignShiftRight', $node->var, ' >>= ', $node->expr);
-    }
-
-    public function pExpr_AssignList(PHPParser_Node_Expr_AssignList $node) {
-        return $this->pAssignList($node->vars) . ' = ' . $this->p($node->expr);
     }
 
     // Binary expressions
 
     public function pExpr_Plus(PHPParser_Node_Expr_Plus $node) {
-        return $this->p($node->left) . ' + ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Plus', $node->left, ' + ', $node->right);
     }
 
     public function pExpr_Minus(PHPParser_Node_Expr_Minus $node) {
-        return $this->p($node->left) . ' - ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Minus', $node->left, ' - ', $node->right);
     }
 
     public function pExpr_Mul(PHPParser_Node_Expr_Mul $node) {
-        return $this->p($node->left) . ' * ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Mul', $node->left, ' * ', $node->right);
     }
 
     public function pExpr_Div(PHPParser_Node_Expr_Div $node) {
-        return $this->p($node->left) . ' / ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Div', $node->left, ' / ', $node->right);
     }
 
     public function pExpr_Concat(PHPParser_Node_Expr_Concat $node) {
-        return $this->p($node->left) . ' . ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Concat', $node->left, ' . ', $node->right);
     }
 
     public function pExpr_Mod(PHPParser_Node_Expr_Mod $node) {
-        return $this->p($node->left) . ' % ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Mod', $node->left, ' % ', $node->right);
     }
 
     public function pExpr_BooleanAnd(PHPParser_Node_Expr_BooleanAnd $node) {
-        return $this->p($node->left) . ' && ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_BooleanAnd', $node->left, ' && ', $node->right);
     }
 
     public function pExpr_BooleanOr(PHPParser_Node_Expr_BooleanOr $node) {
-        return $this->p($node->left) . ' || ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_BooleanOr', $node->left, ' || ', $node->right);
     }
 
     public function pExpr_BitwiseAnd(PHPParser_Node_Expr_BitwiseAnd $node) {
-        return $this->p($node->left) . ' & ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_BitwiseAnd', $node->left, ' & ', $node->right);
     }
 
     public function pExpr_BitwiseOr(PHPParser_Node_Expr_BitwiseOr $node) {
-        return $this->p($node->left) . ' | ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_BitwiseOr', $node->left, ' | ', $node->right);
     }
 
     public function pExpr_BitwiseXor(PHPParser_Node_Expr_BitwiseXor $node) {
-        return $this->p($node->left) . ' ^ ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_BitwiseXor', $node->left, ' ^ ', $node->right);
     }
 
     public function pExpr_ShiftLeft(PHPParser_Node_Expr_ShiftLeft $node) {
-        return $this->p($node->left) . ' << ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_ShiftLeft', $node->left, ' << ', $node->right);
     }
 
     public function pExpr_ShiftRight(PHPParser_Node_Expr_ShiftRight $node) {
-        return $this->p($node->left) . ' >> ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_ShiftRight', $node->left, ' >> ', $node->right);
     }
 
     public function pExpr_LogicalAnd(PHPParser_Node_Expr_LogicalAnd $node) {
-        return $this->p($node->left) . ' and ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_LogicalAnd', $node->left, ' and ', $node->right);
     }
 
     public function pExpr_LogicalOr(PHPParser_Node_Expr_LogicalOr $node) {
-        return $this->p($node->left) . ' or ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_LogicalOr', $node->left, ' or ', $node->right);
     }
 
     public function pExpr_LogicalXor(PHPParser_Node_Expr_LogicalXor $node) {
-        return $this->p($node->left) . ' xor ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_LogicalXor', $node->left, ' xor ', $node->right);
     }
 
     public function pExpr_Equal(PHPParser_Node_Expr_Equal $node) {
-        return $this->p($node->left) . ' == ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Equal', $node->left, ' == ', $node->right);
     }
 
     public function pExpr_NotEqual(PHPParser_Node_Expr_NotEqual $node) {
-        return $this->p($node->left) . ' != ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_NotEqual', $node->left, ' != ', $node->right);
     }
 
     public function pExpr_Identical(PHPParser_Node_Expr_Identical $node) {
-        return $this->p($node->left) . ' === ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Identical', $node->left, ' === ', $node->right);
     }
 
     public function pExpr_NotIdentical(PHPParser_Node_Expr_NotIdentical $node) {
-        return $this->p($node->left) . ' !== ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_NotIdentical', $node->left, ' !== ', $node->right);
     }
 
     public function pExpr_Greater(PHPParser_Node_Expr_Greater $node) {
-        return $this->p($node->left) . ' > ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Greater', $node->left, ' > ', $node->right);
     }
 
     public function pExpr_GreaterOrEqual(PHPParser_Node_Expr_GreaterOrEqual $node) {
-        return $this->p($node->left) . ' >= ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_GreaterOrEqual', $node->left, ' >= ', $node->right);
     }
 
     public function pExpr_Smaller(PHPParser_Node_Expr_Smaller $node) {
-        return $this->p($node->left) . ' < ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_Smaller', $node->left, ' < ', $node->right);
     }
 
     public function pExpr_SmallerOrEqual(PHPParser_Node_Expr_SmallerOrEqual $node) {
-        return $this->p($node->left) . ' <= ' . $this->p($node->right);
+        return $this->pInfixOp('Expr_SmallerOrEqual', $node->left, ' <= ', $node->right);
     }
 
     public function pExpr_Instanceof(PHPParser_Node_Expr_Instanceof $node) {
-        return $this->p($node->expr) . ' instanceof ' . $this->p($node->class);
+        return $this->pInfixOp('Expr_Instanceof', $node->expr, ' instanceof ', $node->class);
     }
 
     // Unary expressions
 
     public function pExpr_BooleanNot(PHPParser_Node_Expr_BooleanNot $node) {
-        return '!' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_BooleanNot', '!', $node->expr);
     }
 
     public function pExpr_BitwiseNot(PHPParser_Node_Expr_BitwiseNot $node) {
-        return '~' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_BitwiseNot', '~', $node->expr);
     }
 
     public function pExpr_UnaryMinus(PHPParser_Node_Expr_UnaryMinus $node) {
-        return '-' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_UnaryMinus', '-', $node->expr);
     }
 
     public function pExpr_UnaryPlus(PHPParser_Node_Expr_UnaryPlus $node) {
-        return '+' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_UnaryPlus', '+', $node->expr);
     }
 
     public function pExpr_PreInc(PHPParser_Node_Expr_PreInc $node) {
-        return '++' . $this->p($node->var);
+        return $this->pPrefixOp('Expr_PreInc', '++', $node->var);
     }
 
     public function pExpr_PreDec(PHPParser_Node_Expr_PreDec $node) {
-        return '--' . $this->p($node->var);
+        return $this->pPrefixOp('Expr_PreDec', '--', $node->var);
     }
 
     public function pExpr_PostInc(PHPParser_Node_Expr_PostInc $node) {
-        return $this->p($node->var) . '++';
+        return $this->pPostfixOp('Expr_PostInc', $node->var, '++');
     }
 
     public function pExpr_PostDec(PHPParser_Node_Expr_PostDec $node) {
-        return $this->p($node->var) . '--';
+        return $this->pPostfixOp('Expr_PostDec', $node->var, '--');
     }
 
     public function pExpr_ErrorSuppress(PHPParser_Node_Expr_ErrorSuppress $node) {
-        return '@' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_ErrorSuppress', '@', $node->expr);
     }
 
     // Casts
 
     public function pExpr_Cast_Int(PHPParser_Node_Expr_Cast_Int $node) {
-        return '(int) ' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_Cast_Int', '(int) ', $node->expr);
     }
 
     public function pExpr_Cast_Double(PHPParser_Node_Expr_Cast_Double $node) {
-        return '(double) ' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_Cast_Double', '(double) ', $node->expr);
     }
 
     public function pExpr_Cast_String(PHPParser_Node_Expr_Cast_String $node) {
-        return '(string) ' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_Cast_String', '(string) ', $node->expr);
     }
 
     public function pExpr_Cast_Array(PHPParser_Node_Expr_Cast_Array $node) {
-        return '(array) ' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_Cast_Array', '(array) ', $node->expr);
     }
 
     public function pExpr_Cast_Object(PHPParser_Node_Expr_Cast_Object $node) {
-        return '(object) ' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_Cast_Object', '(object) ', $node->expr);
     }
 
     public function pExpr_Cast_Bool(PHPParser_Node_Expr_Cast_Bool $node) {
-        return '(bool) ' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_Cast_Bool', '(bool) ', $node->expr);
     }
 
     public function pExpr_Cast_Unset(PHPParser_Node_Expr_Cast_Unset $node) {
-        return '(unset) ' . $this->p($node->expr);
+        return $this->pPrefixOp('Expr_Cast_Unset', '(unset) ', $node->expr);
     }
 
     // Function calls and similar constructs
@@ -339,7 +335,7 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
     }
 
     public function pExpr_Empty(PHPParser_Node_Expr_Empty $node) {
-        return 'empty(' . $this->p($node->var) . ')';
+        return 'empty(' . $this->p($node->expr) . ')';
     }
 
     public function pExpr_Isset(PHPParser_Node_Expr_Isset $node) {
@@ -365,6 +361,19 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
         return $map[$node->type] . ' ' . $this->p($node->expr);
     }
 
+    public function pExpr_List(PHPParser_Node_Expr_List $node) {
+        $pList = array();
+        foreach ($node->vars as $var) {
+            if (null === $var) {
+                $pList[] = '';
+            } else {
+                $pList[] = $this->p($var);
+            }
+        }
+
+        return 'list(' . implode(', ', $pList) . ')';
+    }
+
     // Other
 
     public function pExpr_Variable(PHPParser_Node_Expr_Variable $node) {
@@ -430,15 +439,29 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
     }
 
     public function pExpr_Ternary(PHPParser_Node_Expr_Ternary $node) {
-        return $this->p($node->cond) . ' ?'
+        // a bit of cheating: we treat the ternary as a binary op where the ?...: part is the operator.
-             . (null !== $node->if ? ' ' . $this->p($node->if) . ' ' : '')
+        // this is okay because the part between ? and : never needs parentheses.
-             . ': ' . $this->p($node->else);
+        return $this->pInfixOp('Expr_Ternary',
+            $node->cond, ' ?' . (null !== $node->if ? ' ' . $this->p($node->if) . ' ' : '') . ': ', $node->else
+        );
     }
 
     public function pExpr_Exit(PHPParser_Node_Expr_Exit $node) {
         return 'die' . (null !== $node->expr ? '(' . $this->p($node->expr) . ')' : '');
     }
 
+    public function pExpr_Yield(PHPParser_Node_Expr_Yield $node) {
+        if ($node->value === null) {
+            return 'yield';
+        } else {
+            // this is a bit ugly, but currently there is no way to detect whether the parentheses are necessary
+            return '(yield '
+                 . ($node->key !== null ? $this->p($node->key) . ' => ' : '')
+                 . $this->p($node->value)
+                 . ')';
+        }
+    }
+
     // Declarations
 
     public function pStmt_Namespace(PHPParser_Node_Stmt_Namespace $node) {
@@ -580,12 +603,20 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
 
     public function pStmt_Switch(PHPParser_Node_Stmt_Switch $node) {
         return 'switch (' . $this->p($node->cond) . ') {'
-             . "\n" . $this->pImplode($node->cases) . '}';
+             . "\n" . $this->pStmts($node->cases) . "\n" . '}';
+    }
+
+    public function pStmt_Case(PHPParser_Node_Stmt_Case $node) {
+        return (null !== $node->cond ? 'case ' . $this->p($node->cond) : 'default') . ':'
+             . ($node->stmts ? "\n" . $this->pStmts($node->stmts) : '');
     }
 
     public function pStmt_TryCatch(PHPParser_Node_Stmt_TryCatch $node) {
         return 'try {' . "\n" . $this->pStmts($node->stmts) . "\n" . '}'
-             . $this->pImplode($node->catches);
+             . $this->pImplode($node->catches)
+             . ($node->finallyStmts !== null
+                ? ' finally {' . "\n" . $this->pStmts($node->finallyStmts) . "\n" . '}'
+                : '');
     }
 
     public function pStmt_Catch(PHPParser_Node_Stmt_Catch $node) {
@@ -593,11 +624,6 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
              . "\n" . $this->pStmts($node->stmts) . "\n" . '}';
     }
 
-    public function pStmt_Case(PHPParser_Node_Stmt_Case $node) {
-        return (null !== $node->cond ? 'case ' . $this->p($node->cond) : 'default') . ':'
-             . "\n" . $this->pStmts($node->stmts) . "\n";
-    }
-
     public function pStmt_Break(PHPParser_Node_Stmt_Break $node) {
         return 'break' . ($node->num !== null ? ' ' . $this->p($node->num) : '') . ';';
     }
@@ -646,7 +672,7 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
     }
 
     public function pStmt_InlineHTML(PHPParser_Node_Stmt_InlineHTML $node) {
-        return '?>' . $this->pSafe(
+        return '?>' . $this->pNoIndent(
             ("\n" === $node->value[0] || "\r" === $node->value[0] ? "\n" : '')
             . $node->value
         ) . '<?php ';
@@ -688,21 +714,6 @@ class PHPParser_PrettyPrinter_Zend extends PHPParser_PrettyPrinterAbstract
         return $return;
     }
 
-    public function pAssignList(array $elements) {
-        $pAssignList = array();
-        foreach ($elements as $element) {
-            if (null === $element) {
-                $pAssignList[] = '';
-            } elseif (is_array($element)) {
-                $pAssignList[] = $this->pAssignList($element);
-            } else {
-                $pAssignList[] = $this->p($element);
-            }
-        }
-
-        return 'list(' . implode(', ', $pAssignList) . ')';
-    }
-
     public function pVarOrNewExpr(PHPParser_Node $node) {
         if ($node instanceof PHPParser_Node_Expr_New) {
             return '(' . $this->p($node) . ')';

lib/PHPParser/PrettyPrinterAbstract.php

@@ -3,69 +3,68 @@
 abstract class PHPParser_PrettyPrinterAbstract
 {
     protected $precedenceMap = array(
-        'Expr_BitwiseNot'       =>  1,
+        // [precedence, associativity] where for the latter -1 is %left, 0 is %nonassoc and 1 is %right
-        'Expr_PreInc'           =>  1,
+        'Expr_BitwiseNot'       => array( 1,  1),
-        'Expr_PreDec'           =>  1,
+        'Expr_PreInc'           => array( 1,  1),
-        'Expr_PostInc'          =>  1,
+        'Expr_PreDec'           => array( 1,  1),
-        'Expr_PostDec'          =>  1,
+        'Expr_PostInc'          => array( 1, -1),
-        'Expr_UnaryPlus'        =>  1,
+        'Expr_PostDec'          => array( 1, -1),
-        'Expr_UnaryMinus'       =>  1,
+        'Expr_UnaryPlus'        => array( 1,  1),
-        'Expr_Cast_Int'         =>  1,
+        'Expr_UnaryMinus'       => array( 1,  1),
-        'Expr_Cast_Double'      =>  1,
+        'Expr_Cast_Int'         => array( 1,  1),
-        'Expr_Cast_String'      =>  1,
+        'Expr_Cast_Double'      => array( 1,  1),
-        'Expr_Cast_Array'       =>  1,
+        'Expr_Cast_String'      => array( 1,  1),
-        'Expr_Cast_Object'      =>  1,
+        'Expr_Cast_Array'       => array( 1,  1),
-        'Expr_Cast_Bool'        =>  1,
+        'Expr_Cast_Object'      => array( 1,  1),
-        'Expr_Cast_Unset'       =>  1,
+        'Expr_Cast_Bool'        => array( 1,  1),
-        'Expr_ErrorSuppress'    =>  1,
+        'Expr_Cast_Unset'       => array( 1,  1),
-        'Expr_Instanceof'       =>  2,
+        'Expr_ErrorSuppress'    => array( 1,  1),
-        'Expr_BooleanNot'       =>  3,
+        'Expr_Instanceof'       => array( 2,  0),
-        'Expr_Mul'              =>  4,
+        'Expr_BooleanNot'       => array( 3,  1),
-        'Expr_Div'              =>  4,
+        'Expr_Mul'              => array( 4, -1),
-        'Expr_Mod'              =>  4,
+        'Expr_Div'              => array( 4, -1),
-        'Expr_Plus'             =>  5,
+        'Expr_Mod'              => array( 4, -1),
-        'Expr_Minus'            =>  5,
+        'Expr_Plus'             => array( 5, -1),
-        'Expr_Concat'           =>  5,
+        'Expr_Minus'            => array( 5, -1),
-        'Expr_ShiftLeft'        =>  6,
+        'Expr_Concat'           => array( 5, -1),
-        'Expr_ShiftRight'       =>  6,
+        'Expr_ShiftLeft'        => array( 6, -1),
-        'Expr_Smaller'          =>  7,
+        'Expr_ShiftRight'       => array( 6, -1),
-        'Expr_SmallerOrEqual'   =>  7,
+        'Expr_Smaller'          => array( 7,  0),
-        'Expr_Greater'          =>  7,
+        'Expr_SmallerOrEqual'   => array( 7,  0),
-        'Expr_GreaterOrEqual'   =>  7,
+        'Expr_Greater'          => array( 7,  0),
-        'Expr_Equal'            =>  8,
+        'Expr_GreaterOrEqual'   => array( 7,  0),
-        'Expr_NotEqual'         =>  8,
+        'Expr_Equal'            => array( 8,  0),
-        'Expr_Identical'        =>  8,
+        'Expr_NotEqual'         => array( 8,  0),
-        'Expr_NotIdentical'     =>  8,
+        'Expr_Identical'        => array( 8,  0),
-        'Expr_BitwiseAnd'       =>  9,
+        'Expr_NotIdentical'     => array( 8,  0),
-        'Expr_BitwiseXor'       => 10,
+        'Expr_BitwiseAnd'       => array( 9, -1),
-        'Expr_BitwiseOr'        => 11,
+        'Expr_BitwiseXor'       => array(10, -1),
-        'Expr_BooleanAnd'       => 12,
+        'Expr_BitwiseOr'        => array(11, -1),
-        'Expr_BooleanOr'        => 13,
+        'Expr_BooleanAnd'       => array(12, -1),
-        'Expr_Ternary'          => 14,
+        'Expr_BooleanOr'        => array(13, -1),
-        'Expr_Assign'           => 15,
+        'Expr_Ternary'          => array(14, -1),
-        'Expr_AssignPlus'       => 15,
+        // parser uses %left for assignments, but they really behave as %right
-        'Expr_AssignMinus'      => 15,
+        'Expr_Assign'           => array(15,  1),
-        'Expr_AssignMul'        => 15,
+        'Expr_AssignRef'        => array(15,  1),
-        'Expr_AssignDiv'        => 15,
+        'Expr_AssignPlus'       => array(15,  1),
-        'Expr_AssignConcat'     => 15,
+        'Expr_AssignMinus'      => array(15,  1),
-        'Expr_AssignMod'        => 15,
+        'Expr_AssignMul'        => array(15,  1),
-        'Expr_AssignBitwiseAnd' => 15,
+        'Expr_AssignDiv'        => array(15,  1),
-        'Expr_AssignBitwiseOr'  => 15,
+        'Expr_AssignConcat'     => array(15,  1),
-        'Expr_AssignBitwiseXor' => 15,
+        'Expr_AssignMod'        => array(15,  1),
-        'Expr_AssignShiftLeft'  => 15,
+        'Expr_AssignBitwiseAnd' => array(15,  1),
-        'Expr_AssignShiftRight' => 15,
+        'Expr_AssignBitwiseOr'  => array(15,  1),
-        'Expr_AssignList'       => 15,
+        'Expr_AssignBitwiseXor' => array(15,  1),
-        'Expr_LogicalAnd'       => 16,
+        'Expr_AssignShiftLeft'  => array(15,  1),
-        'Expr_LogicalXor'       => 17,
+        'Expr_AssignShiftRight' => array(15,  1),
-        'Expr_LogicalOr'        => 18,
+        'Expr_LogicalAnd'       => array(16, -1),
+        'Expr_LogicalXor'       => array(17, -1),
+        'Expr_LogicalOr'        => array(18, -1),
     );
 
-    protected $precedenceStack;
-    protected $precedenceStackPos;
     protected $noIndentToken;
 
     public function __construct() {
-        $this->precedenceStack = array($this->precedenceStackPos = 0 => 19);
         $this->noIndentToken   = uniqid('_NO_INDENT_');
     }
 
@@ -126,25 +125,51 @@ abstract class PHPParser_PrettyPrinterAbstract
      * @return string Pretty printed node
      */
     protected function p(PHPParser_Node $node) {
+        return $this->{'p' . $node->getType()}($node);
+    }
+
+    protected function pInfixOp($type, PHPParser_Node $leftNode, $operatorString, PHPParser_Node $rightNode) {
+        list($precedence, $associativity) = $this->precedenceMap[$type];
+
+        return $this->pPrec($leftNode, $precedence, $associativity, -1)
+             . $operatorString
+             . $this->pPrec($rightNode, $precedence, $associativity, 1);
+    }
+
+    protected function pPrefixOp($type, $operatorString, PHPParser_Node $node) {
+        list($precedence, $associativity) = $this->precedenceMap[$type];
+        return $operatorString . $this->pPrec($node, $precedence, $associativity, 1);
+    }
+
+    protected function pPostfixOp($type, PHPParser_Node $node, $operatorString) {
+        list($precedence, $associativity) = $this->precedenceMap[$type];
+        return $this->pPrec($node, $precedence, $associativity, -1) . $operatorString;
+    }
+
+    /**
+     * Prints an expression node with the least amount of parentheses necessary to preserve the meaning.
+     *
+     * @param PHPParser_Node $node                Node to pretty print
+     * @param int            $parentPrecedence    Precedence of the parent operator
+     * @param int            $parentAssociativity Associativity of parent operator
+     *                                            (-1 is left, 0 is nonassoc, 1 is right)
+     * @param int            $childPosition       Position of the node relative to the operator
+     *                                            (-1 is left, 1 is right)
+     *
+     * @return string The pretty printed node
+     */
+    protected function pPrec(PHPParser_Node $node, $parentPrecedence, $parentAssociativity, $childPosition) {
         $type = $node->getType();
-
         if (isset($this->precedenceMap[$type])) {
-            $precedence = $this->precedenceMap[$type];
+            $childPrecedence = $this->precedenceMap[$type][0];
-
+            if ($childPrecedence > $parentPrecedence
-            if ($precedence >= $this->precedenceStack[$this->precedenceStackPos]) {
+                || ($parentPrecedence == $childPrecedence && $parentAssociativity != $childPosition)
-                $this->precedenceStack[++$this->precedenceStackPos] = $precedence;
+            ) {
-                $return = '(' . $this->{'p' . $type}($node) . ')';
+                return '(' . $this->{'p' . $type}($node) . ')';
-                --$this->precedenceStackPos;
-            } else {
-                $this->precedenceStack[++$this->precedenceStackPos] = $precedence;
-                $return = $this->{'p' . $type}($node);
-                --$this->precedenceStackPos;
             }
-
-            return $return;
-        } else {
-            return $this->{'p' . $type}($node);
         }
+
+        return $this->{'p' . $type}($node);
     }
 
     /**
@@ -182,7 +207,7 @@ abstract class PHPParser_PrettyPrinterAbstract
      *
      * @return mixed String marked with $this->noIndentToken's.
      */
-    protected function pSafe($string) {
+    protected function pNoIndent($string) {
         return str_replace("\n", "\n" . $this->noIndentToken, $string);
     }
 

test/PHPParser/Tests/Lexer/EmulativeTest.php

@@ -32,10 +32,17 @@ class PHPParser_Tests_Lexer_EmulativeTest extends PHPUnit_Framework_TestCase
 
     public function provideTestReplaceKeywords() {
         return array(
+            // PHP 5.5
+            array('finally',       PHPParser_Parser::T_FINALLY),
+            array('yield',         PHPParser_Parser::T_YIELD),
+
+            // PHP 5.4
             array('callable',      PHPParser_Parser::T_CALLABLE),
             array('insteadof',     PHPParser_Parser::T_INSTEADOF),
             array('trait',         PHPParser_Parser::T_TRAIT),
             array('__TRAIT__',     PHPParser_Parser::T_TRAIT_C),
+
+            // PHP 5.3
             array('__DIR__',       PHPParser_Parser::T_DIR),
             array('goto',          PHPParser_Parser::T_GOTO),
             array('namespace',     PHPParser_Parser::T_NAMESPACE),

test/PHPParser/Tests/Node/Scalar/StringTest.php

@@ -13,12 +13,12 @@ class PHPParser_Tests_Node_Scalar_StringTest extends PHPUnit_Framework_TestCase
     }
 
     /**
-     * @dataProvider provideTestCreate
+     * @dataProvider provideTestParse
      */
     public function testCreate($expected, $string) {
         $this->assertEquals(
             $expected,
-            PHPParser_Node_Scalar_String::create($string)->value
+            PHPParser_Node_Scalar_String::parse($string)
         );
     }
 
@@ -37,7 +37,7 @@ class PHPParser_Tests_Node_Scalar_StringTest extends PHPUnit_Framework_TestCase
         );
     }
 
-    public function provideTestCreate() {
+    public function provideTestParse() {
         $tests = array(
             array('A', '\'A\''),
             array('A', 'b\'A\''),

test/code/parser/expr/assign.test

@@ -137,7 +137,7 @@ array(
             name: b
         )
     )
-    13: Expr_Assign(
+    13: Expr_AssignRef(
         var: Expr_Variable(
             name: a
         )
@@ -151,43 +151,51 @@ array(
             )
         )
     )
-    14: Expr_AssignList(
+    14: Expr_Assign(
-        vars: array(
+        var: Expr_List(
-            0: Expr_Variable(
+            vars: array(
-                name: a
+                0: Expr_Variable(
+                    name: a
+                )
             )
         )
         expr: Expr_Variable(
             name: b
         )
     )
-    15: Expr_AssignList(
+    15: Expr_Assign(
-        vars: array(
+        var: Expr_List(
-            0: Expr_Variable(
+            vars: array(
-                name: a
+                0: Expr_Variable(
-            )
+                    name: a
-            1: null
+                )
-            2: Expr_Variable(
+                1: null
-                name: b
+                2: Expr_Variable(
+                    name: b
+                )
             )
         )
         expr: Expr_Variable(
             name: c
         )
     )
-    16: Expr_AssignList(
+    16: Expr_Assign(
-        vars: array(
+        var: Expr_List(
-            0: Expr_Variable(
+            vars: array(
-                name: a
+                0: Expr_Variable(
-            )
+                    name: a
-            1: array(
+                )
-                0: null
+                1: Expr_List(
-                1: Expr_Variable(
+                    vars: array(
-                    name: c
+                        0: null
+                        1: Expr_Variable(
+                            name: c
+                        )
+                    )
+                )
+                2: Expr_Variable(
+                    name: d
                 )
-            )
-            2: Expr_Variable(
-                name: d
             )
         )
         expr: Expr_Variable(

test/code/parser/expr/fetchAndCall/constantDeref.test

@@ -0,0 +1,181 @@
+Array/string dereferencing
+-----
+<?php
+
+"abc"[2];
+"abc"[2][0][0];
+
+[1, 2, 3][2];
+[1, 2, 3][2][0][0];
+
+array(1, 2, 3)[2];
+array(1, 2, 3)[2][0][0];
+-----
+array(
+    0: Expr_ArrayDimFetch(
+        var: Scalar_String(
+            value: abc
+        )
+        dim: Scalar_LNumber(
+            value: 2
+        )
+    )
+    1: Expr_ArrayDimFetch(
+        var: Expr_ArrayDimFetch(
+            var: Expr_ArrayDimFetch(
+                var: Scalar_String(
+                    value: abc
+                )
+                dim: Scalar_LNumber(
+                    value: 2
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+        dim: Scalar_LNumber(
+            value: 0
+        )
+    )
+    2: Expr_ArrayDimFetch(
+        var: Expr_Array(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 1
+                    )
+                    byRef: false
+                )
+                1: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 2
+                    )
+                    byRef: false
+                )
+                2: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 3
+                    )
+                    byRef: false
+                )
+            )
+        )
+        dim: Scalar_LNumber(
+            value: 2
+        )
+    )
+    3: Expr_ArrayDimFetch(
+        var: Expr_ArrayDimFetch(
+            var: Expr_ArrayDimFetch(
+                var: Expr_Array(
+                    items: array(
+                        0: Expr_ArrayItem(
+                            key: null
+                            value: Scalar_LNumber(
+                                value: 1
+                            )
+                            byRef: false
+                        )
+                        1: Expr_ArrayItem(
+                            key: null
+                            value: Scalar_LNumber(
+                                value: 2
+                            )
+                            byRef: false
+                        )
+                        2: Expr_ArrayItem(
+                            key: null
+                            value: Scalar_LNumber(
+                                value: 3
+                            )
+                            byRef: false
+                        )
+                    )
+                )
+                dim: Scalar_LNumber(
+                    value: 2
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+        dim: Scalar_LNumber(
+            value: 0
+        )
+    )
+    4: Expr_ArrayDimFetch(
+        var: Expr_Array(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 1
+                    )
+                    byRef: false
+                )
+                1: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 2
+                    )
+                    byRef: false
+                )
+                2: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 3
+                    )
+                    byRef: false
+                )
+            )
+        )
+        dim: Scalar_LNumber(
+            value: 2
+        )
+    )
+    5: Expr_ArrayDimFetch(
+        var: Expr_ArrayDimFetch(
+            var: Expr_ArrayDimFetch(
+                var: Expr_Array(
+                    items: array(
+                        0: Expr_ArrayItem(
+                            key: null
+                            value: Scalar_LNumber(
+                                value: 1
+                            )
+                            byRef: false
+                        )
+                        1: Expr_ArrayItem(
+                            key: null
+                            value: Scalar_LNumber(
+                                value: 2
+                            )
+                            byRef: false
+                        )
+                        2: Expr_ArrayItem(
+                            key: null
+                            value: Scalar_LNumber(
+                                value: 3
+                            )
+                            byRef: false
+                        )
+                    )
+                )
+                dim: Scalar_LNumber(
+                    value: 2
+                )
+            )
+            dim: Scalar_LNumber(
+                value: 0
+            )
+        )
+        dim: Scalar_LNumber(
+            value: 0
+        )
+    )
+)

test/code/parser/expr/issetAndEmpty.test

@@ -5,6 +5,8 @@ isset($a);
 isset($a, $b, $c);
 
 empty($a);
+empty(foo());
+empty(array(1, 2, 3));
 -----
 array(
     0: Expr_Isset(
@@ -28,8 +30,46 @@ array(
         )
     )
     2: Expr_Empty(
-        var: Expr_Variable(
+        expr: Expr_Variable(
             name: a
         )
     )
+    3: Expr_Empty(
+        expr: Expr_FuncCall(
+            name: Name(
+                parts: array(
+                    0: foo
+                )
+            )
+            args: array(
+            )
+        )
+    )
+    4: Expr_Empty(
+        expr: Expr_Array(
+            items: array(
+                0: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 1
+                    )
+                    byRef: false
+                )
+                1: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 2
+                    )
+                    byRef: false
+                )
+                2: Expr_ArrayItem(
+                    key: null
+                    value: Scalar_LNumber(
+                        value: 3
+                    )
+                    byRef: false
+                )
+            )
+        )
+    )
 )

test/code/parser/expr/new.test

@@ -15,7 +15,7 @@ new $a->b->c();
 new $a->b['c']();
 new $a->b{'c'}();
 
-// test regression introduces by new dereferencing syntext
+// test regression introduces by new dereferencing syntax
 (new A);
 -----
 array(

test/code/parser/stmt/function/generator.test

@@ -0,0 +1,227 @@
+Generators (yield expression
+-----
+<?php
+
+function gen() {
+    // statements
+    yield;
+    yield $value;
+    yield $key => $value;
+
+    // expressions
+    $data = yield;
+    $data = (yield $value);
+    $data = (yield $key => $value);
+
+    // yield in language constructs with their own parentheses
+    if (yield $foo); elseif (yield $foo);
+    if (yield $foo): elseif (yield $foo): endif;
+    while (yield $foo);
+    do {} while (yield $foo);
+    switch (yield $foo) {}
+    die(yield $foo);
+
+    // yield in function calls
+    func(yield $foo);
+    $foo->func(yield $foo);
+    new Foo(yield $foo);
+}
+-----
+array(
+    0: Stmt_Function(
+        byRef: false
+        params: array(
+        )
+        stmts: array(
+            0: Expr_Yield(
+                key: null
+                value: null
+            )
+            1: Expr_Yield(
+                key: null
+                value: Expr_Variable(
+                    name: value
+                )
+            )
+            2: Expr_Yield(
+                key: Expr_Variable(
+                    name: key
+                )
+                value: Expr_Variable(
+                    name: value
+                )
+            )
+            3: Expr_Assign(
+                var: Expr_Variable(
+                    name: data
+                )
+                expr: Expr_Yield(
+                    key: null
+                    value: null
+                )
+            )
+            4: Expr_Assign(
+                var: Expr_Variable(
+                    name: data
+                )
+                expr: Expr_Yield(
+                    key: null
+                    value: Expr_Variable(
+                        name: value
+                    )
+                )
+            )
+            5: Expr_Assign(
+                var: Expr_Variable(
+                    name: data
+                )
+                expr: Expr_Yield(
+                    key: Expr_Variable(
+                        name: key
+                    )
+                    value: Expr_Variable(
+                        name: value
+                    )
+                )
+            )
+            6: Stmt_If(
+                stmts: array(
+                )
+                elseifs: array(
+                    0: Stmt_ElseIf(
+                        cond: Expr_Yield(
+                            key: null
+                            value: Expr_Variable(
+                                name: foo
+                            )
+                        )
+                        stmts: array(
+                        )
+                    )
+                )
+                else: null
+                cond: Expr_Yield(
+                    key: null
+                    value: Expr_Variable(
+                        name: foo
+                    )
+                )
+            )
+            7: Stmt_If(
+                stmts: array(
+                )
+                elseifs: array(
+                    0: Stmt_ElseIf(
+                        cond: Expr_Yield(
+                            key: null
+                            value: Expr_Variable(
+                                name: foo
+                            )
+                        )
+                        stmts: array(
+                        )
+                    )
+                )
+                else: null
+                cond: Expr_Yield(
+                    key: null
+                    value: Expr_Variable(
+                        name: foo
+                    )
+                )
+            )
+            8: Stmt_While(
+                cond: Expr_Yield(
+                    key: null
+                    value: Expr_Variable(
+                        name: foo
+                    )
+                )
+                stmts: array(
+                )
+            )
+            9: Stmt_Do(
+                cond: Expr_Yield(
+                    key: null
+                    value: Expr_Variable(
+                        name: foo
+                    )
+                )
+                stmts: array(
+                )
+            )
+            10: Stmt_Switch(
+                cond: Expr_Yield(
+                    key: null
+                    value: Expr_Variable(
+                        name: foo
+                    )
+                )
+                cases: array(
+                )
+            )
+            11: Expr_Exit(
+                expr: Expr_Yield(
+                    key: null
+                    value: Expr_Variable(
+                        name: foo
+                    )
+                )
+            )
+            12: Expr_FuncCall(
+                name: Name(
+                    parts: array(
+                        0: func
+                    )
+                )
+                args: array(
+                    0: Arg(
+                        value: Expr_Yield(
+                            key: null
+                            value: Expr_Variable(
+                                name: foo
+                            )
+                        )
+                        byRef: false
+                    )
+                )
+            )
+            13: Expr_MethodCall(
+                var: Expr_Variable(
+                    name: foo
+                )
+                name: func
+                args: array(
+                    0: Arg(
+                        value: Expr_Yield(
+                            key: null
+                            value: Expr_Variable(
+                                name: foo
+                            )
+                        )
+                        byRef: false
+                    )
+                )
+            )
+            14: Expr_New(
+                class: Name(
+                    parts: array(
+                        0: Foo
+                    )
+                )
+                args: array(
+                    0: Arg(
+                        value: Expr_Yield(
+                            key: null
+                            value: Expr_Variable(
+                                name: foo
+                            )
+                        )
+                        byRef: false
+                    )
+                )
+            )
+        )
+        name: gen
+    )
+)

test/code/parser/stmt/loop/foreach.test

@@ -7,6 +7,8 @@ foreach ($a as $b)  {}
 foreach ($a as &$b) {}
 foreach ($a as $b => $c) {}
 foreach ($a as $b => &$c) {}
+foreach ($a as list($a, $b)) {}
+foreach ($a as $a => list($b, , $c)) {}
 
 // foreach on expression
 foreach (array() as $b) {}
@@ -69,6 +71,47 @@ array(
         )
     )
     4: Stmt_Foreach(
+        keyVar: null
+        byRef: false
+        stmts: array(
+        )
+        expr: Expr_Variable(
+            name: a
+        )
+        valueVar: Expr_List(
+            vars: array(
+                0: Expr_Variable(
+                    name: a
+                )
+                1: Expr_Variable(
+                    name: b
+                )
+            )
+        )
+    )
+    5: Stmt_Foreach(
+        keyVar: Expr_Variable(
+            name: a
+        )
+        byRef: false
+        stmts: array(
+        )
+        expr: Expr_Variable(
+            name: a
+        )
+        valueVar: Expr_List(
+            vars: array(
+                0: Expr_Variable(
+                    name: b
+                )
+                1: null
+                2: Expr_Variable(
+                    name: c
+                )
+            )
+        )
+    )
+    6: Stmt_Foreach(
         keyVar: null
         byRef: false
         stmts: array(
@@ -81,7 +124,7 @@ array(
             name: b
         )
     )
-    5: Stmt_Foreach(
+    7: Stmt_Foreach(
         keyVar: null
         byRef: false
         stmts: array(

test/code/parser/stmt/switch.test

@@ -23,23 +23,23 @@ array(
         )
         cases: array(
             0: Stmt_Case(
-                stmts: array(
-                )
                 cond: Scalar_LNumber(
                     value: 0
                 )
-            )
-            1: Stmt_Case(
                 stmts: array(
                 )
+            )
+            1: Stmt_Case(
                 cond: Scalar_LNumber(
                     value: 1
                 )
-            )
-            2: Stmt_Case(
                 stmts: array(
                 )
+            )
+            2: Stmt_Case(
                 cond: null
+                stmts: array(
+                )
             )
         )
     )

test/code/parser/stmt/tryCatch.test

@@ -1,16 +1,92 @@
 Try/catch
 -----
 <?php
+
 try {
-
+    doTry();
 } catch (A $b) {
-
+    doCatchA();
 } catch (B $c) {
-
+    doCatchB();
+} finally {
+    doFinally();
 }
+
+// no finally
+try { }
+catch (A $b) { }
+
+// no catch
+try { }
+finally { }
+
 -----
 array(
     0: Stmt_TryCatch(
+        stmts: array(
+            0: Expr_FuncCall(
+                name: Name(
+                    parts: array(
+                        0: doTry
+                    )
+                )
+                args: array(
+                )
+            )
+        )
+        catches: array(
+            0: Stmt_Catch(
+                type: Name(
+                    parts: array(
+                        0: A
+                    )
+                )
+                var: b
+                stmts: array(
+                    0: Expr_FuncCall(
+                        name: Name(
+                            parts: array(
+                                0: doCatchA
+                            )
+                        )
+                        args: array(
+                        )
+                    )
+                )
+            )
+            1: Stmt_Catch(
+                type: Name(
+                    parts: array(
+                        0: B
+                    )
+                )
+                var: c
+                stmts: array(
+                    0: Expr_FuncCall(
+                        name: Name(
+                            parts: array(
+                                0: doCatchB
+                            )
+                        )
+                        args: array(
+                        )
+                    )
+                )
+            )
+        )
+        finallyStmts: array(
+            0: Expr_FuncCall(
+                name: Name(
+                    parts: array(
+                        0: doFinally
+                    )
+                )
+                args: array(
+                )
+            )
+        )
+    )
+    1: Stmt_TryCatch(
         stmts: array(
         )
         catches: array(
@@ -24,16 +100,15 @@ array(
                 stmts: array(
                 )
             )
-            1: Stmt_Catch(
+        )
-                type: Name(
+        finallyStmts: null
-                    parts: array(
+    )
-                        0: B
+    2: Stmt_TryCatch(
-                    )
+        stmts: array(
-                )
+        )
-                var: c
+        catches: array(
-                stmts: array(
+        )
-                )
+        finallyStmts: array(
-            )
         )
     )
 )

test/code/parser/stmt/tryCatch.test-fail

@@ -0,0 +1,7 @@
+Cannot use try without catch or finally
+-----
+<?php
+
+try { }
+-----
+Cannot use try without catch or finally on line 3

test/code/prettyPrinter/closure.test

@@ -0,0 +1,18 @@
+Closures
+-----
+<?php
+
+$closureWithArgs = function ($arg1, $arg2) {
+    $comment = 'closure body';
+};
+
+$closureWithArgsAndVars = function ($arg1, $arg2) use($var1, $var2) {
+    $comment = 'closure body';
+};
+-----
+$closureWithArgs = function ($arg1, $arg2) {
+    $comment = 'closure body';
+};
+$closureWithArgsAndVars = function ($arg1, $arg2) use($var1, $var2) {
+    $comment = 'closure body';
+};

test/code/prettyPrinter/parentheses.test

@@ -0,0 +1,45 @@
+Pretty printer generates least-parentheses output
+-----
+<?php
+
+echo 'abc' . 'cde' . 'fgh';
+echo 'abc' . ('cde' . 'fgh');
+
+echo 'abc' . 1 + 2 . 'fgh';
+echo 'abc' . (1 + 2) . 'fgh';
+
+echo 1 * 2 + 3 / 4 % 5 . 6;
+echo 1 * (2 + 3) / (4 % (5 . 6));
+
+$a = $b = $c = $d = $f && true;
+($a = $b = $c = $d = $f) && true;
+$a = $b = $c = $d = $f and true;
+$a = $b = $c = $d = ($f and true);
+
+$a ? $b : $c ? $d : $e ? $f : $g;
+$a ? $b : ($c ? $d : ($e ? $f : $g));
+$a ? $b ? $c : $d : $f;
+
+(1 > 0) > (1 < 0);
+
+// this will currently unnecessarily print !($a = $b). This is correct as far as operator precedence is concerned, but
+// the parentheses are not really necessary, because = takes only variables as the left hand side operator.
+!$a = $b;
+-----
+echo 'abc' . 'cde' . 'fgh';
+echo 'abc' . ('cde' . 'fgh');
+echo 'abc' . 1 + 2 . 'fgh';
+echo 'abc' . (1 + 2) . 'fgh';
+echo 1 * 2 + 3 / 4 % 5 . 6;
+echo 1 * (2 + 3) / (4 % (5 . 6));
+$a = $b = $c = $d = $f && true;
+($a = $b = $c = $d = $f) && true;
+$a = $b = $c = $d = $f and true;
+$a = $b = $c = $d = ($f and true);
+$a ? $b : $c ? $d : $e ? $f : $g;
+$a ? $b : ($c ? $d : ($e ? $f : $g));
+$a ? $b ? $c : $d : $f;
+(1 > 0) > (1 < 0);
+// this will currently unnecessarily print !($a = $b). This is correct as far as operator precedence is concerned, but
+// the parentheses are not really necessary, because = takes only variables as the left hand side operator.
+!($a = $b);

test/code/prettyPrinter/switch.test

@@ -0,0 +1,35 @@
+switch/case/default
+-----
+<?php
+
+switch ($expr) {
+    case 0:
+        echo 'First case, with a break';
+        break;
+    case 1:
+        echo 'Second case, which falls through';
+    case 2:
+    case 3:
+    case 4:
+        echo 'Third case, return instead of break';
+        return;
+    default:
+        echo 'Default case';
+        break;
+}
+-----
+switch ($expr) {
+    case 0:
+        echo 'First case, with a break';
+        break;
+    case 1:
+        echo 'Second case, which falls through';
+    case 2:
+    case 3:
+    case 4:
+        echo 'Third case, return instead of break';
+        return;
+    default:
+        echo 'Default case';
+        break;
+}