Release PHP-Parser 4.0.2

This needs to go through something like Encapsed or ShellExec to
determine quotation type. Explicitly throw an exception to avoid
getting an undefined method error.

CHANGELOG.md

@@ -1,8 +1,18 @@
-Version 4.0.2-dev
+Version 4.0.3-dev
 -----------------
 
 Nothing yet.
 
+Version 4.0.2 (2018-06-03)
+--------------------------
+
+### Added
+
+* Improved error recovery inside classes.
+* Support error recovery for `foreach` without `as`.
+* Support error recovery for parameters without variable (`function (Type ) {}`).
+* Support error recovery for functions without body (`function ($foo)`).
+
 Version 4.0.1 (2018-03-25)
 --------------------------
 

README.md

@@ -200,7 +200,7 @@ Component documentation:
    * Converting AST back to PHP code
    * Customizing formatting
    * Formatting-preserving code transformations
- * [AST builders](component/AST_builders.markdown)
+ * [AST builders](doc/component/AST_builders.markdown)
    * Fluent builders for AST nodes
  * [Lexer](doc/component/Lexer.markdown)
    * Lexer options
@@ -209,16 +209,16 @@ Component documentation:
  * [Error handling](doc/component/Error_handling.markdown)
    * Column information for errors
    * Error recovery (parsing of syntactically incorrect code)
- * [Constant expression evaluation](component/Constant_expression_evaluation.markdown)
+ * [Constant expression evaluation](doc/component/Constant_expression_evaluation.markdown)
    * Evaluating constant/property/etc initializers
    * Handling errors and unsupported expressions
- * [JSON representation](component/JSON_representation.markdown)
+ * [JSON representation](doc/component/JSON_representation.markdown)
    * JSON encoding and decoding of ASTs
  * [Performance](doc/component/Performance.markdown)
    * Disabling XDebug
    * Reusing objects
    * Garbage collection impact
- * [Frequently asked questions](component/FAQ.markdown)
+ * [Frequently asked questions](doc/component/FAQ.markdown)
    * Parent and sibling references
 
  [doc_3_x]: https://github.com/nikic/PHP-Parser/tree/3.x/doc

UPGRADE-4.0.md

@@ -17,7 +17,7 @@ HHVM is no longer actively supported.
   comparisons or strict-mode may require adjustment. The following is an exhaustive list of all
   affected subnodes:
 
-   * `Const::$name`
+   * `Const_::$name`
    * `NullableType::$type` (for simple types)
    * `Param::$type` (for simple types)
    * `Expr\ClassConstFetch::$name`
@@ -29,8 +29,8 @@ HHVM is no longer actively supported.
    * `Stmt\Class_::$name`
    * `Stmt\ClassMethod::$name`
    * `Stmt\ClassMethod::$returnType` (for simple types)
-   * `Stmt\Function::$name`
+   * `Stmt\Function_::$name`
-   * `Stmt\Function::$returnType` (for simple types)
+   * `Stmt\Function_::$returnType` (for simple types)
    * `Stmt\Goto_::$name`
    * `Stmt\Interface_::$name`
    * `Stmt\Label::$name`
@@ -48,7 +48,7 @@ HHVM is no longer actively supported.
 * The `name` subnode of `StaticVar` has been renamed to `var` and now contains a `Variable` rather
   than a plain string.
 * The `var` subnode of `ClosureUse` now contains a `Variable` rather than a plain string.
-* The `var` subnode of `Catch` now contains a `Variable` rather than a plain string.
+* The `var` subnode of `Catch_` now contains a `Variable` rather than a plain string.
 * The `alias` subnode of `UseUse` is now `null` if no explicit alias is given. As such,
   `use Foo\Bar` and `use Foo\Bar as Bar` are now represented differently. The `getAlias()` method
   can be used to get the effective alias, even if it is not explicitly given.
@@ -62,7 +62,7 @@ HHVM is no longer actively supported.
 
 ### Removed functionality
 
-* Removed `type` subnode on `Class`, `ClassMethod` and `Property` nodes. Use `flags` instead.
+* Removed `type` subnode on `Class_`, `ClassMethod` and `Property` nodes. Use `flags` instead.
 * The `ClassConst::isStatic()` method has been removed. Constants cannot have a static modifier.
 * The `NodeTraverser` no longer accepts `false` as a return value from a `leaveNode()` method.
   `NodeTraverser::REMOVE_NODE` should be returned instead.

doc/2_Usage_of_basic_components.markdown

@@ -409,7 +409,7 @@ $files = new \RegexIterator($files, '/\.php$/');
 foreach ($files as $file) {
     try {
         // read the file that should be converted
-        $code = file_get_contents($file);
+        $code = file_get_contents($file->getPathName());
 
         // parse
         $stmts = $parser->parse($code);

grammar/php7.y

@@ -240,6 +240,8 @@ non_empty_statement:
           { $$ = Stmt\Foreach_[$3, $5[0], ['keyVar' => null, 'byRef' => $5[1], 'stmts' => $7]]; }
     | T_FOREACH '(' expr T_AS variable T_DOUBLE_ARROW foreach_variable ')' foreach_statement
           { $$ = Stmt\Foreach_[$3, $7[0], ['keyVar' => $5, 'byRef' => $7[1], 'stmts' => $9]]; }
+    | T_FOREACH '(' expr error ')' foreach_statement
+          { $$ = Stmt\Foreach_[$3, new Expr\Error(stackAttributes(#4)), ['stmts' => $6]]; }
     | T_DECLARE '(' declare_list ')' declare_statement      { $$ = Stmt\Declare_[$3, $5]; }
     | T_TRY '{' inner_statement_list '}' catches optional_finally
           { $$ = Stmt\TryCatch[$3, $5, $6]; $this->checkTryCatch($$); }
@@ -295,9 +297,14 @@ optional_ellipsis:
     | T_ELLIPSIS                                            { $$ = true; }
 ;
 
+block_or_error:
+      '{' inner_statement_list '}'                          { $$ = $2; }
+    | error                                                 { $$ = []; }
+;
+
 function_declaration_statement:
-    T_FUNCTION optional_ref identifier '(' parameter_list ')' optional_return_type '{' inner_statement_list '}'
+    T_FUNCTION optional_ref identifier '(' parameter_list ')' optional_return_type block_or_error
-        { $$ = Stmt\Function_[$3, ['byRef' => $2, 'params' => $5, 'returnType' => $7, 'stmts' => $9]]; }
+        { $$ = Stmt\Function_[$3, ['byRef' => $2, 'params' => $5, 'returnType' => $7, 'stmts' => $8]]; }
 ;
 
 class_declaration_statement:
@@ -447,6 +454,8 @@ parameter:
           { $$ = Node\Param[$4, null, $1, $2, $3]; $this->checkParam($$); }
     | optional_param_type optional_ref optional_ellipsis plain_variable '=' expr
           { $$ = Node\Param[$4, $6, $1, $2, $3]; $this->checkParam($$); }
+    | optional_param_type optional_ref optional_ellipsis error
+          { $$ = Node\Param[Expr\Error[], null, $1, $2, $3]; }
 ;
 
 type_expr:
@@ -514,7 +523,7 @@ static_var:
 ;
 
 class_statement_list:
-      class_statement_list class_statement                  { push($1, $2); }
+      class_statement_list class_statement                  { if ($2 !== null) { push($1, $2); } }
     | /* empty */                                           { init(); }
 ;
 
@@ -527,6 +536,7 @@ class_statement:
           { $$ = Stmt\ClassMethod[$4, ['type' => $1, 'byRef' => $3, 'params' => $6, 'returnType' => $8, 'stmts' => $9]];
             $this->checkClassMethod($$, #1); }
     | T_USE class_name_list trait_adaptations               { $$ = Stmt\TraitUse[$2, $3]; }
+    | error                                                 { $$ = null; /* will be skipped */ }
 ;
 
 trait_adaptations:
@@ -562,7 +572,7 @@ trait_method_reference:
 
 method_body:
       ';' /* abstract method */                             { $$ = null; }
-    | '{' inner_statement_list '}'                          { $$ = $2; }
+    | block_or_error                                        { $$ = $1; }
 ;
 
 variable_modifiers:
@@ -708,11 +718,11 @@ expr:
     | T_YIELD expr T_DOUBLE_ARROW expr                      { $$ = Expr\Yield_[$4, $2]; }
     | T_YIELD_FROM expr                                     { $$ = Expr\YieldFrom[$2]; }
     | T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars optional_return_type
-      '{' inner_statement_list '}'
+      block_or_error
-          { $$ = Expr\Closure[['static' => false, 'byRef' => $2, 'params' => $4, 'uses' => $6, 'returnType' => $7, 'stmts' => $9]]; }
+          { $$ = Expr\Closure[['static' => false, 'byRef' => $2, 'params' => $4, 'uses' => $6, 'returnType' => $7, 'stmts' => $8]]; }
     | T_STATIC T_FUNCTION optional_ref '(' parameter_list ')' lexical_vars optional_return_type
-      '{' inner_statement_list '}'
+      block_or_error
-          { $$ = Expr\Closure[['static' => true, 'byRef' => $3, 'params' => $5, 'uses' => $7, 'returnType' => $8, 'stmts' => $10]]; }
+          { $$ = Expr\Closure[['static' => true, 'byRef' => $3, 'params' => $5, 'uses' => $7, 'returnType' => $8, 'stmts' => $9]]; }
 ;
 
 anonymous_class:

lib/PhpParser/BuilderFactory.php

@@ -104,7 +104,7 @@ class BuilderFactory
     /**
      * Creates a namespace/class use builder.
      *
-     * @param string|Node\Name Name to alias
+     * @param string|Node\Name $name Name to alias
      *
      * @return Builder\Use_ The create use builder
      */

lib/PhpParser/Error.php

@@ -15,7 +15,7 @@ class Error extends \RuntimeException
      *                              (or start line of error -- deprecated)
      */
     public function __construct(string $message, $attributes = []) {
-        $this->rawMessage = (string) $message;
+        $this->rawMessage = $message;
         if (is_array($attributes)) {
             $this->attributes = $attributes;
         } else {
@@ -76,7 +76,7 @@ class Error extends \RuntimeException
      * @param string $message Error message
      */
     public function setRawMessage(string $message) {
-        $this->rawMessage = (string) $message;
+        $this->rawMessage = $message;
         $this->updateMessage();
     }
 
@@ -86,7 +86,7 @@ class Error extends \RuntimeException
      * @param int $line Error start line
      */
     public function setStartLine(int $line) {
-        $this->attributes['startLine'] = (int) $line;
+        $this->attributes['startLine'] = $line;
         $this->updateMessage();
     }
 

lib/PhpParser/Node/Expr/BinaryOp.php

@@ -12,7 +12,7 @@ abstract class BinaryOp extends Expr
     public $right;
 
     /**
-     * Constructs a bitwise and node.
+     * Constructs a binary operator node.
      *
      * @param Expr  $left       The left hand side expression
      * @param Expr  $right      The right hand side expression

lib/PhpParser/Node/Param.php

@@ -12,7 +12,7 @@ class Param extends NodeAbstract
     public $byRef;
     /** @var bool Whether this is a variadic argument */
     public $variadic;
-    /** @var Expr\Variable Parameter variable */
+    /** @var Expr\Variable|Expr\Error Parameter variable */
     public $var;
     /** @var null|Expr Default value */
     public $default;
@@ -20,7 +20,7 @@ class Param extends NodeAbstract
     /**
      * Constructs a parameter node.
      *
-     * @param Expr\Variable                 $var        Parameter variable
+     * @param Expr\Variable|Expr\Error      $var        Parameter variable
      * @param null|Expr                     $default    Default value
      * @param null|string|Name|NullableType $type       Typehint
      * @param bool                          $byRef      Whether is passed by reference
@@ -28,7 +28,7 @@ class Param extends NodeAbstract
      * @param array                         $attributes Additional attributes
      */
     public function __construct(
-        Expr\Variable $var, Expr $default = null, $type = null,
+        $var, Expr $default = null, $type = null,
         bool $byRef = false, bool $variadic = false, array $attributes = []
     ) {
         parent::__construct($attributes);

lib/PhpParser/ParserAbstract.php

@@ -109,8 +109,6 @@ abstract class ParserAbstract implements Parser
 
     /** @var ErrorHandler Error handler */
     protected $errorHandler;
-    /** @var Error[] Errors collected during last parse */
-    protected $errors;
     /** @var int Error state, used to avoid error floods */
     protected $errorState;
 
@@ -129,7 +127,6 @@ abstract class ParserAbstract implements Parser
      */
     public function __construct(Lexer $lexer, array $options = []) {
         $this->lexer = $lexer;
-        $this->errors = [];
 
         if (isset($options['throwOnError'])) {
             throw new \LogicException(

lib/PhpParser/PrettyPrinter/Standard.php

@@ -198,6 +198,10 @@ class Standard extends PrettyPrinterAbstract
         return preg_match('/^-?[0-9]+$/', $stringValue) ? $stringValue . '.0' : $stringValue;
     }
 
+    protected function pScalar_EncapsedStringPart(Scalar\EncapsedStringPart $node) {
+        throw new \LogicException('Cannot directly print EncapsedStringPart');
+    }
+
     // Assignments
 
     protected function pExpr_Assign(Expr\Assign $node) {

test/PhpParser/NodeVisitor/FirstFindingVisitorTest.php

@@ -34,6 +34,6 @@ class FirstFindingVisitorTest extends TestCase
         $stmts = [new Node\Stmt\Expression($assign)];
 
         $traverser->traverse($stmts);
-        $this->assertSame(null, $visitor->getFoundNode());
+        $this->assertNull($visitor->getFoundNode());
     }
 }

test/PhpParser/PrettyPrinterTest.php

@@ -208,6 +208,16 @@ class PrettyPrinterTest extends CodeTestAbstract
         $prettyPrinter->prettyPrint($stmts);
     }
 
+    /**
+     * @expectedException \LogicException
+     * @expectedExceptionMessage Cannot directly print EncapsedStringPart
+     */
+    public function testPrettyPrintEncapsedStringPart() {
+        $expr = new Node\Scalar\EncapsedStringPart('foo');
+        $prettyPrinter = new PrettyPrinter\Standard;
+        $prettyPrinter->prettyPrintExpr($expr);
+    }
+
     /**
      * @dataProvider provideTestFormatPreservingPrint
      * @covers \PhpParser\PrettyPrinter\Standard<extended>

test/code/parser/errorHandling/recovery.test

@@ -953,3 +953,317 @@ array(
         )
     )
 )
+-----
+<?php
+
+class Foo {
+    publi $foo;
+    public $bar;
+}
+-----
+!!php7
+Syntax error, unexpected T_STRING from 4:5 to 4:9
+array(
+    0: Stmt_Class(
+        flags: 0
+        name: Identifier(
+            name: Foo
+        )
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_Property(
+                flags: MODIFIER_PUBLIC (1)
+                props: array(
+                    0: Stmt_PropertyProperty(
+                        name: VarLikeIdentifier(
+                            name: bar
+                        )
+                        default: null
+                    )
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+
+foreach ($foo) { $bar; }
+foreach ($foo as ) { $bar; }
+-----
+!!php7
+Syntax error, unexpected ')' from 3:14 to 3:14
+Syntax error, unexpected ')' from 4:18 to 4:18
+array(
+    0: Stmt_Foreach(
+        expr: Expr_Variable(
+            name: foo
+        )
+        keyVar: null
+        byRef: false
+        valueVar: Expr_Error(
+        )
+        stmts: array(
+            0: Stmt_Expression(
+                expr: Expr_Variable(
+                    name: bar
+                )
+            )
+        )
+    )
+    1: Stmt_Foreach(
+        expr: Expr_Variable(
+            name: foo
+        )
+        keyVar: null
+        byRef: false
+        valueVar: Expr_Error(
+        )
+        stmts: array(
+            0: Stmt_Expression(
+                expr: Expr_Variable(
+                    name: bar
+                )
+            )
+        )
+    )
+)
+-----
+<?php
+
+function foo(Type) {
+    $foo;
+}
+
+function foo(Type1 $foo, Type2) {
+    $bar;
+}
+
+function foo(...) {
+    $baz;
+}
+
+function foo(&) {
+    $qux;
+}
+
+function foo(Bar)
+
+class Bar {
+    function foo(Baz)
+}
+
+function(Foo);
+-----
+!!php7
+Syntax error, unexpected ')', expecting T_VARIABLE from 3:18 to 3:18
+Syntax error, unexpected ')', expecting T_VARIABLE from 7:31 to 7:31
+Syntax error, unexpected ')', expecting T_VARIABLE from 11:17 to 11:17
+Syntax error, unexpected ')', expecting T_VARIABLE from 15:15 to 15:15
+Syntax error, unexpected ')', expecting T_VARIABLE from 19:17 to 19:17
+Syntax error, unexpected ')', expecting T_VARIABLE from 22:21 to 22:21
+Syntax error, unexpected ')', expecting T_VARIABLE from 25:13 to 25:13
+array(
+    0: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: foo
+        )
+        params: array(
+            0: Param(
+                type: Name(
+                    parts: array(
+                        0: Type
+                    )
+                )
+                byRef: false
+                variadic: false
+                var: Expr_Error(
+                )
+                default: null
+            )
+        )
+        returnType: null
+        stmts: array(
+            0: Stmt_Expression(
+                expr: Expr_Variable(
+                    name: foo
+                )
+            )
+        )
+    )
+    1: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: foo
+        )
+        params: array(
+            0: Param(
+                type: Name(
+                    parts: array(
+                        0: Type1
+                    )
+                )
+                byRef: false
+                variadic: false
+                var: Expr_Variable(
+                    name: foo
+                )
+                default: null
+            )
+            1: Param(
+                type: Name(
+                    parts: array(
+                        0: Type2
+                    )
+                )
+                byRef: false
+                variadic: false
+                var: Expr_Error(
+                )
+                default: null
+            )
+        )
+        returnType: null
+        stmts: array(
+            0: Stmt_Expression(
+                expr: Expr_Variable(
+                    name: bar
+                )
+            )
+        )
+    )
+    2: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: foo
+        )
+        params: array(
+            0: Param(
+                type: null
+                byRef: false
+                variadic: true
+                var: Expr_Error(
+                )
+                default: null
+            )
+        )
+        returnType: null
+        stmts: array(
+            0: Stmt_Expression(
+                expr: Expr_Variable(
+                    name: baz
+                )
+            )
+        )
+    )
+    3: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: foo
+        )
+        params: array(
+            0: Param(
+                type: null
+                byRef: true
+                variadic: false
+                var: Expr_Error(
+                )
+                default: null
+            )
+        )
+        returnType: null
+        stmts: array(
+            0: Stmt_Expression(
+                expr: Expr_Variable(
+                    name: qux
+                )
+            )
+        )
+    )
+    4: Stmt_Function(
+        byRef: false
+        name: Identifier(
+            name: foo
+        )
+        params: array(
+            0: Param(
+                type: Name(
+                    parts: array(
+                        0: Bar
+                    )
+                )
+                byRef: false
+                variadic: false
+                var: Expr_Error(
+                )
+                default: null
+            )
+        )
+        returnType: null
+        stmts: array(
+        )
+    )
+    5: Stmt_Class(
+        flags: 0
+        name: Identifier(
+            name: Bar
+        )
+        extends: null
+        implements: array(
+        )
+        stmts: array(
+            0: Stmt_ClassMethod(
+                flags: 0
+                byRef: false
+                name: Identifier(
+                    name: foo
+                )
+                params: array(
+                    0: Param(
+                        type: Name(
+                            parts: array(
+                                0: Baz
+                            )
+                        )
+                        byRef: false
+                        variadic: false
+                        var: Expr_Error(
+                        )
+                        default: null
+                    )
+                )
+                returnType: null
+                stmts: array(
+                )
+            )
+        )
+    )
+    6: Stmt_Expression(
+        expr: Expr_Closure(
+            static: false
+            byRef: false
+            params: array(
+                0: Param(
+                    type: Name(
+                        parts: array(
+                            0: Foo
+                        )
+                    )
+                    byRef: false
+                    variadic: false
+                    var: Expr_Error(
+                    )
+                    default: null
+                )
+            )
+            uses: array(
+            )
+            returnType: null
+            stmts: array(
+            )
+        )
+    )
+)