Release version 0.9.1

This is in anticipation of doing some backwards compatability breaking
changes in the next version.

.travis.yml

@@ -0,0 +1,6 @@
+language: php
+
+php:
+ - 5.2
+ - 5.3
+ - 5.4

CHANGELOG.md

@@ -0,0 +1,34 @@
+Version 0.9.2-dev
+-----------------
+
+Nothing yet.
+
+Version 0.9.1 (24.04.2012)
+--------------------------
+
+* Add ability to add attributes to nodes:
+
+  It is now possible to add attributes to a node using `$node->setAttribute('name', 'value')` and to retrieve them using
+  `$node->getAttribute('name' [, 'default'])`. Additionally the existance of an attribute can be checked with
+  `$node->hasAttribute('name')` and all attributes can be returned using `$node->getAttributes()`.
+
+* Add code generation features: Builders and templates.
+
+  For more infos, see the [code generation documentation][1].
+
+* [BC] Don't traverse nodes merged by another visitor:
+
+  If a NodeVisitor returns an array of nodes to merge, these will no longer be traversed by all other visitors. This
+  behavior only caused problems.
+
+* Fix line numbers for some list structures
+* Fix XML unserialization of empty nodes
+* Fix parsing of integers that overflow into floats
+* Fix emulation of NOWDOC and binary floats
+
+Version 0.9.0 (05.01.2012)
+--------------------------
+
+First version.
+
+ [1]: https://github.com/nikic/PHP-Parser/blob/master/doc/3_Code_generation.markdown

README.md

@@ -6,8 +6,73 @@ manipulation.
 
 Documentation can be found in the [`doc/`][1] directory.
 
-***Note: This project is experimental. There are no known bugs in the parser itself, but the API is
-subject to change.***
+***Note: This project is experimental, so the API is subject to change.***
 
+In a Nutshell
+-------------
+
+Basically, the parser does nothing more than turn some PHP code into an abstract syntax tree. ("nothing
+more" is kind of sarcastic here as PHP has a ... uhm, let's just say "not nice" ... grammar, which makes
+parsing PHP very hard.)
+
+For example, if you stick this code in the parser:
+
+```php
+<?php
+echo 'Hi', 'World';
+hello\world('foo', 'bar' . 'baz');
+```
+
+You'll get a syntax tree looking roughly like this:
+
+```
+array(
+    0: Stmt_Echo(
+        exprs: array(
+            0: Scalar_String(
+                value: Hi
+            )
+            1: Scalar_String(
+                value: World
+            )
+        )
+    )
+    1: Expr_FuncCall(
+        name: Name(
+            parts: array(
+                0: hello
+                1: world
+            )
+        )
+        args: array(
+            0: Arg(
+                value: Scalar_String(
+                    value: foo
+                )
+                byRef: false
+            )
+            1: Arg(
+                value: Expr_Concat(
+                    left: Scalar_String(
+                        value: bar
+                    )
+                    right: Scalar_String(
+                        value: baz
+                    )
+                )
+                byRef: false
+            )
+        )
+    )
+)
+```
+
+You can then work with this syntax tree, for example to statically analyze the code (e.g. to find
+programming errors or security issues).
+
+Additionally, you can convert a syntax tree back to PHP code. This allows you to do code preprocessing
+(like automatedly porting code to older PHP versions).
+
+So, that's it, in a nutshell. You can find everything else in the [docs][1].
 
  [1]: https://github.com/nikic/PHP-Parser/tree/master/doc

doc/0_Introduction.markdown

@@ -26,31 +26,13 @@ programmatic PHP code analysis are incidentially PHP developers, not C developer
 What can it parse?
 ------------------
 
-The parser uses a PHP 5.4 compliant grammar, but lexing is done using the `token_get_all` tokenization
-facility provided by PHP itself. This means that you will be able to parse pretty much any PHP code you
-want, but there are some limitations to keep in mind:
+The parser uses a PHP 5.4 compliant grammar, which is backwards compatible with at least PHP 5.3 and PHP
+5.2 (and maybe older).
 
- * The PHP 5.4 grammar is implemented in such a way that it is backwards compatible. So parsing PHP 5.3
-   and PHP 5.2 is also possible (and maybe older versions). On the other hand this means that the parser
-   will let some code through, which would be invalid in the newest version (for example call time pass
-   by reference will *not* throw an error even though PHP 5.4 doesn't allow it anymore). This shouldn't
-   normally be a problem and if it is strictly required it can be easily implemented in a NodeVisitor.
-
- * Even though the parser supports PHP 5.4 it depends on the internal tokenizer, which only supports
-   the PHP version it runs on. So you will be able parse PHP 5.4 if you are running PHP 5.4. But you
-   wouldn't be able to parse PHP 5.4 code (which uses one of the new features) on PHP 5.3. The support
-   matrix looks roughly like this:
-
-                        | parsing PHP 5.4 | parsing PHP 5.3 | parsing PHP 5.2
-        ---------------------------------------------------------------------
-        running PHP 5.4 | yes             | yes             | yes
-        running PHP 5.3 | no              | yes             | yes
-        running PHP 5.2 | no              | no              | yes
-
- * The parser inherits all bugs of the `token_get_all` function. There are only two which I
-   currently know of, namely lexing of `b"$var"` literals and nested HEREDOC strings. The former
-   bug is circumvented by the `PHPParser_Lexer` wrapper which the parser uses, but the latter remains
-   (though I seriously doublt it will ever occur in practical use.)
+As the parser is based on the tokens returned by `token_get_all` (which is only able to lex the PHP
+version it runs on), additionally a wrapper for emulating new tokens from 5.3 and 5.4 is provided. This
+allows to parse PHP 5.4 source code running on PHP 5.2, for example. This emulation is very hacky and not
+yet perfect, but it should work well on any sane code.
 
 What output does it produce?
 ----------------------------

doc/1_Usage_of_basic_components.markdown

@@ -14,14 +14,6 @@ The library needs to register a class autoloader; you can do this either by incl
 require 'path/to/PHP-Parser/lib/bootstrap.php';
 ```
 
-Or by manually registering the loader:
-
-```php
-<?php
-require 'path/to/PHP-Parser/lib/PHPParser/Autoloader.php';
-PHPParser_Autoloader::register();
-```
-
 Parsing
 -------
 
@@ -45,6 +37,12 @@ try {
 
 The `parse` method will return an array of statement nodes (`$stmts`).
 
+### Emulative lexer
+
+Instead of `PHPParser_Lexer` one can also use `PHPParser_Lexer_Emulative`. This class will emulate tokens
+of newer PHP versions and as such allow parsing PHP 5.4 on PHP 5.2, for example. So if you want to parse
+PHP code of newer versions than the one you are running, you should use the emulative lexer.
+
 Node tree
 ---------
 
@@ -102,6 +100,9 @@ without the `PHPParser_Node_` prefix). Additionally there are `getLine()`, which
 the node startet in, and `getDocComment()`, which returns the doc comment above the node (if there
 is any), and the respective setters `setLine()` and `setDocComment()`.
 
+It is possible to associate custom metadata with a node using the `setAttribute()` method. This data
+can then be retrieved using `hasAttribute()`, `getAttribute()` and `getAttributes()`.
+
 Pretty printer
 --------------
 
@@ -288,7 +289,9 @@ foreach (new RecursiveIteratorIterator(
         $code = file_get_contents($file);
 
         // parse
-        $stmts = $parser->parse(new PHPParser_Lexer($code));
+        // use the emulative lexer here, as we are running PHP 5.2 but want to
+        // parse PHP 5.3
+        $stmts = $parser->parse(new PHPParser_Lexer_Emulative($code));
 
         // traverse
         $stmts = $traverser->traverse($stmts);

doc/3_Code_generation.markdown

@@ -0,0 +1,265 @@
+Code generation
+===============
+
+It is also possible to generate code using the parser, by first creating an Abstract Syntax Tree and then using the
+pretty printer to convert it to PHP code. To simplify code generation, the project comes with a set of builders for
+common structures as well as simple templating support. Both features are described in the following:
+
+Builders
+--------
+
+The project provides builders for classes, methods, functions, parameters and properties, which
+allow creating node trees with a fluid interface, instead of instantiating all nodes manually.
+
+Here is an example:
+
+```php
+<?php
+$factory = new PHPParser_BuilderFactory;
+$node = $factory->class('SomeClass')
+    ->extend('SomeOtherClass')
+    ->implement('A\Few', 'Interfaces')
+    ->makeAbstract() // ->makeFinal()
+
+    ->addStmt($factory->method('someMethod')
+        ->makeAbstract() // ->makeFinal()
+        ->addParam($factory->param('someParam')->setTypeHint('SomeClass'))
+    )
+
+    ->addStmt($factory->method('anotherMethod')
+        ->makeProtected() // ->makePublic() [default], ->makePrivate()
+        ->addParam($factory->param('someParam')->setDefault('test'))
+        // it is possible to add manually created nodes
+        ->addStmt(new PHPParser_Node_Expr_Print(new PHPParser_Node_Expr_Variable('someParam')))
+    )
+
+    // properties will be correctly reordered above the methods
+    ->addStmt($factory->property('someProperty')->makeProtected())
+    ->addStmt($factory->property('anotherProperty')->makePrivate()->setDefault(array(1, 2, 3)))
+
+    ->getNode()
+;
+
+$stmts = array($node);
+echo $prettyPrinter->prettyPrint($stmts);
+```
+
+This will produce the following output with the default pretty printer:
+
+```php
+<?php
+abstract class SomeClass extends SomeOtherClass implements A\Few, Interfaces
+{
+    protected $someProperty;
+    private $anotherProperty = array(1, 2, 3);
+    abstract function someMethod(SomeClass $someParam);
+    protected function anotherMethod($someParam = 'test')
+    {
+        print $someParam;
+    }
+}
+```
+
+Templates
+---------
+
+Additionally it is possible to generate code from reusable templates.
+
+As an example consider the following template, which defines a general getter/setter sceleton in terms of a property
+`__name__` and its `__type__`:
+
+```php
+<?php
+
+class GetterSetterTemplate
+{
+    /**
+     * @var __type__ The __name__
+     */
+    protected $__name__;
+
+    /**
+     * Gets the __name__.
+     *
+     * @return __type__ The __name__
+     */
+    public function get__Name__() {
+        return $this->__name__;
+    }
+
+    /**
+     * Sets the __name__.
+     *
+     * @param __type__ $__name__ The new __name__
+     */
+    public function set__Name__($__name__) {
+        $this->__name__ = $__name__;
+    }
+}
+```
+
+Using this template we can easily create a class with multiple properties and their respective getters and setters:
+
+```php
+<?php
+
+// $templateString contains the above template
+$template = new PHPParser_Template($parser, $templateString);
+
+// We only have to specify the __name__ placeholder, as the
+// capitalized __Name__ placeholder is automatically created
+$properties = [
+    ['name' => 'title',     'type' => 'string'],
+    ['name' => 'body',      'type' => 'string'],
+    ['name' => 'author',    'type' => 'User'],
+    ['name' => 'timestamp', 'type' => 'DateTime'],
+];
+
+$class = $factory->class('BlogPost')->implement('Post');
+
+foreach ($properties as $propertyPlaceholders) {
+    $stmts = $template->getStmts($propertyPlaceholders);
+
+    $class->addStmts(
+        // $stmts contains all statements from the template. So [0] fetches the class statement
+        // and ->stmts retrieves the methods.
+        $stmts[0]->stmts
+    );
+}
+
+echo $prettyPrinter->prettyPrint(array($class->getNode()));
+```
+
+The result would look roughly like this:
+
+```php
+<?php
+
+class BlogPost implements Post
+{
+    /**
+     * @var string The title
+     */
+    protected $title;
+
+    /**
+     * @var string The body
+     */
+    protected $body;
+
+    /**
+     * @var User The author
+     */
+    protected $author;
+
+    /**
+     * @var DateTime The timestamp
+     */
+    protected $timestamp;
+
+    /**
+     * Gets the title.
+     *
+     * @return string The title
+     */
+    public function getTitle()
+    {
+        return $this->title;
+    }
+
+    /**
+     * Sets the title.
+     *
+     * @param string $title The new title
+     */
+    public function setTitle($title)
+    {
+        $this->title = $title;
+    }
+
+    /**
+     * Gets the body.
+     *
+     * @return string The body
+     */
+    public function getBody()
+    {
+        return $this->body;
+    }
+
+    /**
+     * Sets the body.
+     *
+     * @param string $body The new body
+     */
+    public function setBody($body)
+    {
+        $this->body = $body;
+    }
+
+    /**
+     * Gets the author.
+     *
+     * @return User The author
+     */
+    public function getAuthor()
+    {
+        return $this->author;
+    }
+
+    /**
+     * Sets the author.
+     *
+     * @param User $author The new author
+     */
+    public function setAuthor($author)
+    {
+        $this->author = $author;
+    }
+
+    /**
+     * Gets the timestamp.
+     *
+     * @return DateTime The timestamp
+     */
+    public function getTimestamp()
+    {
+        return $this->timestamp;
+    }
+
+    /**
+     * Sets the timestamp.
+     *
+     * @param DateTime $timestamp The new timestamp
+     */
+    public function setTimestamp($timestamp)
+    {
+        $this->timestamp = $timestamp;
+    }
+}
+```
+
+When using multiple templates it is easier to manage them on the filesystem. They can be loaded using the
+`TemplateLoader`:
+
+```php
+<?php
+
+// We'll store our templates in ./templates and give them a .php suffix
+$loader = new PHPParser_TemplateLoader($parser, './templates', '.php');
+
+// loads ./templates/GetterSetter.php
+$getterSetterTemplate = $loader->load('GetterSetter');
+
+// loads ./templates/Collection.php
+$collectionTemplate = $loader->load('Collection');
+
+// The use of a suffix is optional. The following code for example is equivalent:
+$loader = new PHPParser_TemplateLoader($parser, './templates');
+
+// loads ./templates/GetterSetter.php
+$getterSetterTemplate = $loader->load('GetterSetter.php');
+
+// loads ./templates/Collection.php
+$collectionTemplate = $loader->load('Collection.php');
+```

grammar/kmyacc.php.parser

@@ -6,9 +6,14 @@ $meta #
 #semval(%n,%t) $this->yyastk[$this->yysp-(%l-%n)]
 #include;
 
-/* Prototype file of an object oriented PHP parser.
- * Written by Moriyoshi Koizumi, based on the work by Masato Bito.
- * This file is PUBLIC DOMAIN.
+/* This is an automatically GENERATED file, which should not be manually edited.
+ * Instead edit one of the following:
+ *  * the grammar file grammar/zend_language_parser.phpy
+ *  * the parser skeleton grammar/kymacc.php.parser
+ *  * the preprocessing script grammar/rebuildParser.php
+ *
+ * The skeleton for this parser was written by Moriyoshi Koizumi and is based on
+ * the work by Masato Bito and is in the PUBLIC DOMAIN.
  */
 #if -t
 class #(-p)_Debug extends #(-p)

grammar/rebuildParser.php

@@ -138,12 +138,6 @@ function resolveMacros($code) {
                 return 'substr(' . $args[0] . ', 1)';
             }
 
-            if ('parseDNumber' == $name) {
-                assertArgs(1, $args, $name);
-
-                return '(double) ' . $args[0];
-            }
-
             if ('parseEncapsed' == $name) {
                 assertArgs(2, $args, $name);
 

grammar/zend_language_parser.phpy

@@ -279,8 +279,12 @@ declare_statement:
 ;
 
 declare_list:
-      T_STRING '=' static_scalar                            { init(Stmt_DeclareDeclare[$1, $3]); }
-    | declare_list ',' T_STRING '=' static_scalar           { push($1, Stmt_DeclareDeclare[$3, $5]); }
+      declare_list_element                                  { init($1); }
+    | declare_list ',' declare_list_element                 { push($1, $3); }
+;
+
+declare_list_element:
+      T_STRING '=' static_scalar                            { $$ = Stmt_DeclareDeclare[$1, $3]; }
 ;
 
 switch_case_list:
@@ -292,10 +296,12 @@ switch_case_list:
 
 case_list:
       /* empty */                                           { init(); }
-    | case_list T_CASE expr case_separator inner_statement_list
-          { push($1, Stmt_Case[$3, $5]); }
-    | case_list T_DEFAULT case_separator inner_statement_list
-          { push($1, Stmt_Case[null, $4]); }
+    | case_list case                                        { push($1, $2); }
+;
+
+case:
+      T_CASE expr case_separator inner_statement_list       { $$ = Stmt_Case[$2, $4]; }
+    | T_DEFAULT case_separator inner_statement_list         { $$ = Stmt_Case[null, $3]; }
 ;
 
 case_separator:
@@ -309,14 +315,21 @@ while_statement:
 ;
 
 elseif_list:
-      /* empty */                                           { init();}
-    | elseif_list T_ELSEIF '(' expr ')' statement           { push($1, Stmt_ElseIf[$4, toArray($6)]); }
+      /* empty */                                           { init(); }
+    | elseif_list elseif                                    { push($1, $2); }
+;
+
+elseif:
+      T_ELSEIF '(' expr ')' statement                       { $$ = Stmt_ElseIf[$3, toArray($5)]; }
 ;
 
 new_elseif_list:
       /* empty */                                           { init(); }
-    | new_elseif_list T_ELSEIF '(' expr ')' ':' inner_statement_list
-          { push($1, Stmt_ElseIf[$4, $7]); }
+    | new_elseif_list new_elseif                            { push($1, $2); }
+;
+
+new_elseif:
+     T_ELSEIF '(' expr ')' ':' inner_statement_list         { $$ = Stmt_ElseIf[$3, $6]; }
 ;
 
 else_single:
@@ -572,10 +585,12 @@ lexical_vars:
 ;
 
 lexical_var_list:
-      lexical_var_list ',' optional_ref T_VARIABLE
-          { push($1, Expr_ClosureUse[parseVar($4), $3]); }
-    | optional_ref T_VARIABLE
-          { init(Expr_ClosureUse[parseVar($2), $1]); }
+      lexical_var                                           { init($1); }
+    | lexical_var_list ',' lexical_var                      { push($1, $3); }
+;
+
+lexical_var:
+      optional_ref T_VARIABLE                               { $$ = Expr_ClosureUse[parseVar($2), $1]; }
 ;
 
 function_call:
@@ -659,7 +674,7 @@ ctor_arguments:
 
 common_scalar:
       T_LNUMBER                                             { $$ = Scalar_LNumber[Scalar_LNumber::parse($1)]; }
-    | T_DNUMBER                                             { $$ = Scalar_DNumber[parseDNumber($1)]; }
+    | T_DNUMBER                                             { $$ = Scalar_DNumber[Scalar_DNumber::parse($1)]; }
     | T_CONSTANT_ENCAPSED_STRING                            { $$ = Scalar_String::create($1, $line, $docComment); }
     | T_LINE                                                { $$ = Scalar_LineConst[]; }
     | T_FILE                                                { $$ = Scalar_FileConst[]; }

lib/PHPParser/Builder.php

@@ -0,0 +1,11 @@
+<?php
+
+interface PHPParser_Builder
+{
+    /**
+     * Returns the built node.
+     *
+     * @return PHPParser_Node The built node
+     */
+    public function getNode();
+}

lib/PHPParser/Builder/Class.php

@@ -0,0 +1,137 @@
+<?php
+
+class PHPParser_Builder_Class extends PHPParser_BuilderAbstract
+{
+    protected $name;
+
+    protected $extends;
+    protected $implements;
+    protected $type;
+
+    protected $uses;
+    protected $constants;
+    protected $properties;
+    protected $methods;
+
+    /**
+     * Creates a class builder.
+     *
+     * @param string $name Name of the class
+     */
+    public function __construct($name) {
+        $this->name = $name;
+
+        $this->type = 0;
+        $this->extends = null;
+        $this->implements = array();
+
+        $this->uses = $this->constants = $this->properties = $this->methods = array();
+    }
+
+    /**
+     * Extends a class.
+     *
+     * @param PHPParser_Node_Name|string $class Name of class to extend
+     *
+     * @return PHPParser_Builder_Class The builder instance (for fluid interface)
+     */
+    public function extend($class) {
+        $this->extends = $this->normalizeName($class);
+
+        return $this;
+    }
+
+    /**
+     * Implements one or more interfaces.
+     *
+     * @param PHPParser_Node_Name|string $interface Name of interface to implement
+     * @param PHPParser_Node_Name|string $...       More interfaces to implement
+     *
+     * @return PHPParser_Builder_Class The builder instance (for fluid interface)
+     */
+    public function implement() {
+        foreach (func_get_args() as $interface) {
+            $this->implements[] = $this->normalizeName($interface);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Makes the class abstract.
+     *
+     * @return PHPParser_Builder_Class The builder instance (for fluid interface)
+     */
+    public function makeAbstract() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_ABSTRACT);
+
+        return $this;
+    }
+
+    /**
+     * Makes the class final.
+     *
+     * @return PHPParser_Builder_Class The builder instance (for fluid interface)
+     */
+    public function makeFinal() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_FINAL);
+
+        return $this;
+    }
+
+    /**
+     * Adds a statement.
+     *
+     * @param PHPParser_Node_Stmt|PHPParser_Builder $stmt The statement to add
+     *
+     * @return PHPParser_Builder_Class The builder instance (for fluid interface)
+     */
+    public function addStmt($stmt) {
+        $stmt = $this->normalizeNode($stmt);
+
+        $targets = array(
+            'Stmt_TraitUse'    => &$this->uses,
+            'Stmt_ClassConst'  => &$this->constants,
+            'Stmt_Property'    => &$this->properties,
+            'Stmt_ClassMethod' => &$this->methods,
+        );
+
+        $type = $stmt->getType();
+        if (!isset($targets[$type])) {
+            throw new LogicException(sprintf('Unexpected node of type "%s"', $type));
+        }
+
+        $targets[$type][] = $stmt;
+
+        return $this;
+    }
+
+    /**
+     * Adds multiple statements.
+     *
+     * @param array $stmts The statements to add
+     *
+     * @return PHPParser_Builder_Class The builder instance (for fluid interface)
+     */
+    public function addStmts(array $stmts) {
+        foreach ($stmts as $stmt) {
+            $this->addStmt($stmt);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Returns the built class node.
+     *
+     * @return PHPParser_Node_Stmt_Class The built class node
+     */
+    public function getNode() {
+        return new PHPParser_Node_Stmt_Class($this->name, array(
+            'type' => $this->type,
+            'extends' => $this->extends,
+            'implements' => $this->implements,
+            'stmts' => array_merge($this->uses, $this->constants, $this->properties, $this->methods),
+        ));
+    }
+}

lib/PHPParser/Builder/Function.php

@@ -0,0 +1,109 @@
+<?php
+
+class PHPParser_Builder_Function extends PHPParser_BuilderAbstract
+{
+    protected $name;
+
+    protected $returnByRef;
+    protected $params;
+    protected $stmts;
+
+    /**
+     * Creates a function builder.
+     *
+     * @param string $name Name of the function
+     */
+    public function __construct($name) {
+        $this->name = $name;
+
+        $this->returnByRef = false;
+        $this->params = array();
+        $this->stmts = array();
+    }
+
+    /**
+     * Make the function return by reference.
+     *
+     * @return PHPParser_Builder_Function The builder instance (for fluid interface)
+     */
+    public function makeReturnByRef() {
+        $this->returnByRef = true;
+
+        return $this;
+    }
+
+    /**
+     * Adds a parameter.
+     *
+     * @param PHPParser_Node_Param|PHPParser_Builder_Param $param The parameter to add
+     *
+     * @return PHPParser_Builder_Function The builder instance (for fluid interface)
+     */
+    public function addParam($param) {
+        $param = $this->normalizeNode($param);
+
+        if (!$param instanceof PHPParser_Node_Param) {
+            throw new LogicException(sprintf('Expected parameter node, got "%s"', $param->getType()));
+        }
+
+        $this->params[] = $param;
+
+        return $this;
+    }
+
+    /**
+     * Adds multiple parameters.
+     *
+     * @param array $params The parameters to add
+     *
+     * @return PHPParser_Builder_Function The builder instance (for fluid interface)
+     */
+    public function addParams(array $params) {
+        foreach ($params as $param) {
+            $this->addParam($param);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Adds a statement.
+     *
+     * @param PHPParser_Node|PHPParser_Builder $stmt The statement to add
+     *
+     * @return PHPParser_Builder_Function The builder instance (for fluid interface)
+     */
+    public function addStmt($stmt) {
+        $this->stmts[] = $this->normalizeNode($stmt);
+
+        return $this;
+    }
+
+    /**
+     * Adds multiple statements.
+     *
+     * @param array $stmts The statements to add
+     *
+     * @return PHPParser_Builder_Function The builder instance (for fluid interface)
+     */
+    public function addStmts(array $stmts) {
+        foreach ($stmts as $stmt) {
+            $this->addStmt($stmt);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Returns the built function node.
+     *
+     * @return PHPParser_Node_Stmt_Function The built function node
+     */
+    public function getNode() {
+        return new PHPParser_Node_Stmt_Function($this->name, array(
+            'byRef'  => $this->returnByRef,
+            'params' => $this->params,
+            'stmts'  => $this->stmts,
+        ));
+    }
+}

lib/PHPParser/Builder/Method.php

@@ -0,0 +1,187 @@
+<?php
+
+class PHPParser_Builder_Method extends PHPParser_BuilderAbstract
+{
+    protected $name;
+
+    protected $type;
+    protected $returnByRef;
+    protected $params;
+    protected $stmts;
+
+    /**
+     * Creates a method builder.
+     *
+     * @param string $name Name of the method
+     */
+    public function __construct($name) {
+        $this->name = $name;
+
+        $this->type = 0;
+        $this->returnByRef = false;
+        $this->params = array();
+        $this->stmts = array();
+    }
+
+    /**
+     * Makes the method public.
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function makePublic() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_PUBLIC);
+
+        return $this;
+    }
+
+    /**
+     * Makes the method protected.
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function makeProtected() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_PROTECTED);
+
+        return $this;
+    }
+
+    /**
+     * Makes the method private.
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function makePrivate() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_PRIVATE);
+
+        return $this;
+    }
+
+    /**
+     * Makes the method static.
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function makeStatic() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_STATIC);
+
+        return $this;
+    }
+
+    /**
+     * Makes the method abstract.
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function makeAbstract() {
+        if (!empty($this->stmts)) {
+            throw new LogicException('Cannot make method with statements abstract');
+        }
+
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_ABSTRACT);
+        $this->stmts = null; // abstract methods don't have statements
+
+        return $this;
+    }
+
+    /**
+     * Makes the method final.
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function makeFinal() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_FINAL);
+
+        return $this;
+    }
+
+    /**
+     * Make the method return by reference.
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function makeReturnByRef() {
+        $this->returnByRef = true;
+
+        return $this;
+    }
+
+    /**
+     * Adds a parameter.
+     *
+     * @param PHPParser_Node_Param|PHPParser_Builder_Param $param The parameter to add
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function addParam($param) {
+        $param = $this->normalizeNode($param);
+
+        if (!$param instanceof PHPParser_Node_Param) {
+            throw new LogicException(sprintf('Expected parameter node, got "%s"', $param->getType()));
+        }
+
+        $this->params[] = $param;
+
+        return $this;
+    }
+
+    /**
+     * Adds multiple parameters.
+     *
+     * @param array $params The parameters to add
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function addParams(array $params) {
+        foreach ($params as $param) {
+            $this->addParam($param);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Adds a statement.
+     *
+     * @param PHPParser_Node|PHPParser_Builder $stmt The statement to add
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function addStmt($stmt) {
+        if (null === $this->stmts) {
+            throw new LogicException('Cannot add statements to an abstract method');
+        }
+
+        $this->stmts[] = $this->normalizeNode($stmt);
+
+        return $this;
+    }
+
+    /**
+     * Adds multiple statements.
+     *
+     * @param array $stmts The statements to add
+     *
+     * @return PHPParser_Builder_Method The builder instance (for fluid interface)
+     */
+    public function addStmts(array $stmts) {
+        foreach ($stmts as $stmt) {
+            $this->addStmt($stmt);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Returns the built method node.
+     *
+     * @return PHPParser_Node_Stmt_ClassMethod The built method node
+     */
+    public function getNode() {
+        return new PHPParser_Node_Stmt_ClassMethod($this->name, array(
+            'type'   => $this->type !== 0 ? $this->type : PHPParser_Node_Stmt_Class::MODIFIER_PUBLIC,
+            'byRef'  => $this->returnByRef,
+            'params' => $this->params,
+            'stmts'  => $this->stmts,
+        ));
+    }
+}

lib/PHPParser/Builder/Param.php

@@ -0,0 +1,75 @@
+<?php
+
+class PHPParser_Builder_Param extends PHPParser_BuilderAbstract
+{
+    protected $name;
+
+    protected $default;
+    protected $type;
+    protected $byRef;
+
+    /**
+     * Creates a parameter builder.
+     *
+     * @param string $name Name of the parameter
+     */
+    public function __construct($name) {
+        $this->name = $name;
+
+        $this->default = null;
+        $this->type = null;
+        $this->byRef = false;
+    }
+
+    /**
+     * Sets default value for the parameter.
+     *
+     * @param mixed $value Default value to use
+     *
+     * @return PHPParser_Builder_Param The builder instance (for fluid interface)
+     */
+    public function setDefault($value) {
+        $this->default = $this->normalizeValue($value);
+
+        return $this;
+    }
+
+    /**
+     * Sets type hint for the parameter.
+     *
+     * @param string|PHPParser_Node_Name $type Type hint to use
+     *
+     * @return PHPParser_Builder_Param The builder instance (for fluid interface)
+     */
+    public function setTypeHint($type) {
+        if ($type === 'array' || $type === 'callable') {
+            $this->type = $type;
+        } else {
+            $this->type = $this->normalizeName($type);
+        }
+
+        return $this;
+    }
+
+    /**
+     * Make the parameter accept the value by reference.
+     *
+     * @return PHPParser_Builder_Param The builder instance (for fluid interface)
+     */
+    public function makeByRef() {
+        $this->byRef = true;
+
+        return $this;
+    }
+
+    /**
+     * Returns the built parameter node.
+     *
+     * @return PHPParser_Node_Param The built parameter node
+     */
+    public function getNode() {
+        return new PHPParser_Node_Param(
+            $this->name, $this->default, $this->type, $this->byRef
+        );
+    }
+}

lib/PHPParser/Builder/Property.php

@@ -0,0 +1,92 @@
+<?php
+
+class PHPParser_Builder_Property extends PHPParser_BuilderAbstract
+{
+    protected $name;
+
+    protected $type;
+    protected $default;
+
+    /**
+     * Creates a property builder.
+     *
+     * @param string $name Name of the property
+     */
+    public function __construct($name) {
+        $this->name = $name;
+
+        $this->type = 0;
+        $this->default = null;
+    }
+
+    /**
+     * Makes the property public.
+     *
+     * @return PHPParser_Builder_Property The builder instance (for fluid interface)
+     */
+    public function makePublic() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_PUBLIC);
+
+        return $this;
+    }
+
+    /**
+     * Makes the property protected.
+     *
+     * @return PHPParser_Builder_Property The builder instance (for fluid interface)
+     */
+    public function makeProtected() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_PROTECTED);
+
+        return $this;
+    }
+
+    /**
+     * Makes the property private.
+     *
+     * @return PHPParser_Builder_Property The builder instance (for fluid interface)
+     */
+    public function makePrivate() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_PRIVATE);
+
+        return $this;
+    }
+
+    /**
+     * Makes the property static.
+     *
+     * @return PHPParser_Builder_Property The builder instance (for fluid interface)
+     */
+    public function makeStatic() {
+        $this->setModifier(PHPParser_Node_Stmt_Class::MODIFIER_STATIC);
+
+        return $this;
+    }
+
+    /**
+     * Sets default value for the property.
+     *
+     * @param mixed $value Default value to use
+     *
+     * @return PHPParser_Builder_Property The builder instance (for fluid interface)
+     */
+    public function setDefault($value) {
+        $this->default = $this->normalizeValue($value);
+
+        return $this;
+    }
+
+    /**
+     * Returns the built class node.
+     *
+     * @return PHPParser_Node_Stmt_Property The built property node
+     */
+    public function getNode() {
+        return new PHPParser_Node_Stmt_Property(
+            $this->type !== 0 ? $this->type : PHPParser_Node_Stmt_Class::MODIFIER_PUBLIC,
+            array(
+                new PHPParser_Node_Stmt_PropertyProperty($this->name, $this->default)
+            )
+        );
+    }
+}

lib/PHPParser/BuilderAbstract.php

@@ -0,0 +1,94 @@
+<?php
+
+abstract class PHPParser_BuilderAbstract implements PHPParser_Builder {
+    /**
+     * Normalizes a node: Converts builder objects to nodes.
+     *
+     * @param PHPParser_Node|PHPParser_Builder $node The node to normalize
+     *
+     * @return PHPParser_Node The normalized node
+     */
+    protected function normalizeNode($node) {
+        if ($node instanceof PHPParser_Builder) {
+            return $node->getNode();
+        } elseif ($node instanceof PHPParser_Node) {
+            return $node;
+        }
+
+        throw new LogicException('Expected node or builder object');
+    }
+
+    /**
+     * Normalizes a name: Converts plain string names to PHPParser_Node_Name.
+     *
+     * @param PHPParser_Node_Name|string $name The name to normalize
+     *
+     * @return PHPParser_Node_Name The normalized name
+     */
+    protected function normalizeName($name) {
+        if ($name instanceof PHPParser_Node_Name) {
+            return $name;
+        } else {
+            return new PHPParser_Node_Name($name);
+        }
+    }
+
+    /**
+     * Normalizes a value: Converts nulls, booleans, integers,
+     * floats and strings into their respective nodes
+     *
+     * @param mixed $value The value to normalize
+     *
+     * @return PHPParser_Node_Expr The normalized value
+     */
+    protected function normalizeValue($value) {
+        if ($value instanceof PHPParser_Node) {
+            return $value;
+        } elseif (is_null($value)) {
+            return new PHPParser_Node_Expr_ConstFetch(
+                new PHPParser_Node_Name('null')
+            );
+        } elseif (is_bool($value)) {
+            return new PHPParser_Node_Expr_ConstFetch(
+                new PHPParser_Node_Name($value ? 'true' : 'false')
+            );
+        } elseif (is_int($value)) {
+            return new PHPParser_Node_Scalar_LNumber($value);
+        } elseif (is_float($value)) {
+            return new PHPParser_Node_Scalar_DNumber($value);
+        } elseif (is_string($value)) {
+            return new PHPParser_Node_Scalar_String($value);
+        } elseif (is_array($value)) {
+            $items = array();
+            $lastKey = -1;
+            foreach ($value as $itemKey => $itemValue) {
+                // for consecutive, numeric keys don't generate keys
+                if (null !== $lastKey && ++$lastKey === $itemKey) {
+                    $items[] = new PHPParser_Node_Expr_ArrayItem(
+                        $this->normalizeValue($itemValue)
+                    );
+                } else {
+                    $lastKey = null;
+                    $items[] = new PHPParser_Node_Expr_ArrayItem(
+                        $this->normalizeValue($itemValue),
+                        $this->normalizeValue($itemKey)
+                    );
+                }
+            }
+
+            return new PHPParser_Node_Expr_Array($items);
+        } else {
+            throw new LogicException('Invalid value');
+        }
+    }
+
+    /**
+     * Sets a modifier in the $this->type property.
+     *
+     * @param int $modifier Modifier to set
+     */
+    protected function setModifier($modifier) {
+        PHPParser_Node_Stmt_Class::verifyModifier($this->type, $modifier);
+        $this->type |= $modifier;
+    }
+}

lib/PHPParser/BuilderFactory.php

@@ -0,0 +1,77 @@
+<?php
+
+/**
+ * "class" and "function" are reserved keywords, so the methods are defined as _class()
+ * and _function() in the class and are made available as class() and function() through
+ * __call() magic.
+ *
+ * @method PHPParser_Builder_Class    class(string $name)    Creates a class builder.
+ * @method PHPParser_Builder_Function function(string $name) Creates a function builder
+ */
+class PHPParser_BuilderFactory
+{
+    /**
+     * Creates a class builder.
+     *
+     * @param string $name Name of the class
+     *
+     * @return PHPParser_Builder_Class The created class builder
+     */
+    protected function _class($name) {
+        return new PHPParser_Builder_Class($name);
+    }
+
+    /**
+     * Creates a method builder.
+     *
+     * @param string $name Name of the method
+     *
+     * @return PHPParser_Builder_Method The created method builder
+     */
+    public function method($name) {
+        return new PHPParser_Builder_Method($name);
+    }
+
+    /**
+     * Creates a parameter builder.
+     *
+     * @param string $name Name of the parameter
+     *
+     * @return PHPParser_Builder_Param The created parameter builder
+     */
+    public function param($name) {
+        return new PHPParser_Builder_Param($name);
+    }
+
+    /**
+     * Creates a property builder.
+     *
+     * @param string $name Name of the property
+     *
+     * @return PHPParser_Builder_Property The created property builder
+     */
+    public function property($name) {
+        return new PHPParser_Builder_Property($name);
+    }
+
+    /**
+     * Creates a function builder.
+     *
+     * @param string $name Name of the function
+     *
+     * @return PHPParser_Builder_Property The created function builder
+     */
+    protected function _function($name) {
+        return new PHPParser_Builder_Function($name);
+    }
+
+    public function __call($name, array $args) {
+        if ('class' === $name) {
+            return call_user_func_array(array($this, '_class'), $args);
+        } elseif ('function' === $name) {
+            return call_user_func_array(array($this, '_function'), $args);
+        }
+
+        throw new LogicException(sprintf('Method "%s" does not exist', $name));
+    }
+}

lib/PHPParser/Lexer.php

@@ -22,32 +22,34 @@ class PHPParser_Lexer
     public function __construct($code) {
         self::initTokenMap();
 
-        // Reset the error message in error_get_last()
-        // Still hoping for a better solution to be found.
-        @$errorGetLastResetUndefinedVariable;
-
-        $this->code   = $code;
+        $this->resetErrors();
         $this->tokens = @token_get_all($code);
+        $this->handleErrors();
+
+        $this->code = $code; // keep the code around for __halt_compiler() handling
         $this->pos  = -1;
         $this->line =  1;
+    }
 
+    protected function resetErrors() {
+        // clear error_get_last() by forcing an undefined variable error
+        @$undefinedVariable;
+    }
+
+    protected function handleErrors() {
         $error = error_get_last();
 
         if (preg_match(
             '~^Unterminated comment starting line ([0-9]+)$~',
-                $error['message'],
-                $matches
-            )
-        ) {
+            $error['message'], $matches
+        )) {
             throw new PHPParser_Error('Unterminated comment', $matches[1]);
         }
 
         if (preg_match(
             '~^Unexpected character in input:  \'(.)\' \(ASCII=([0-9]+)\)~s',
-                $error['message'],
-                $matches
-            )
-        ) {
+            $error['message'], $matches
+        )) {
             throw new PHPParser_Error(sprintf(
                 'Unexpected character "%s" (ASCII %d)',
                 $matches[1], $matches[2]

lib/PHPParser/Lexer/Emulative.php

@@ -23,20 +23,39 @@ class PHPParser_Lexer_Emulative extends PHPParser_Lexer
     public function __construct($code) {
         $this->inObjectAccess = false;
 
-        if (version_compare(PHP_VERSION, '5.4.0RC1', '<')) {
-            // binary notation
-            $code = preg_replace('(\b0b[01]+\b)', '~__EMU__BINARY__$0__~', $code);
+        // on PHP 5.4 don't do anything
+        if (version_compare(PHP_VERSION, '5.4.0RC1', '>=')) {
+            parent::__construct($code);
+        } else {
+            $code = $this->preprocessCode($code);
+            parent::__construct($code);
+            $this->postprocessTokens();
+        }
     }
 
-        if (version_compare(PHP_VERSION, '5.3.0', '<')) {
-            // namespace separator
-            $code = preg_replace('(\\\\(?!["\'`$\\\\]))', '~__EMU__NS__~', $code);
+    /*
+     * Replaces new features in the code by ~__EMU__{NAME}__{DATA}__~ sequences.
+     * ~LABEL~ is never valid PHP code, that's why we can (to some degree) safely
+     * use it here.
+     * Later when preprocessing the tokens these sequences will either be replaced
+     * by real tokens or replaced with their original content (e.g. if they occured
+     * inside a string, i.e. a place where they don't have a special meaning).
+     */
+    protected function preprocessCode($code) {
+        // binary notation (0b010101101001...)
+        $code = preg_replace('(\b0b[01]+\b)', '~__EMU__BINARY__$0__~', $code);
 
-            // nowdoc
+        if (version_compare(PHP_VERSION, '5.3.0', '<')) {
+            // namespace separator (backslash not followed by some special characters,
+            // which are not valid after a NS separator, but would cause problems with
+            // escape sequence parsing if one would replace the backslash there)
+            $code = preg_replace('(\\\\(?!["\'`${\\\\]))', '~__EMU__NS__~', $code);
+
+            // nowdoc (<<<'ABC'\ncontent\nABC;)
             $code = preg_replace_callback(
-                '((*BSR_ANYCRLF)    # set \R to (\r|\n|\r\n)
+                '((*BSR_ANYCRLF)        # set \R to (?>\r\n|\r|\n)
                   (b?<<<[\t ]*\'([a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*)\'\R) # opening token
-                  ((?:(?!\2).*\R)*) # content
+                  ((?:(?!\2;?\R).*\R)*) # content
                   (\2)                  # closing token
                   (?=;?\R)              # must be followed by newline (with optional semicolon)
                  )x',
@@ -45,9 +64,31 @@ class PHPParser_Lexer_Emulative extends PHPParser_Lexer
             );
         }
 
-        parent::__construct($code);
+        return $code;
+    }
 
+    /*
+     * As nowdocs can have arbitrary content but LABELs can only contain a certain
+     * range of characters, the nowdoc content is encoded as hex and separated by
+     * 'x' tokens. So the result of the encoding will look like this:
+     * ~__EMU__NOWDOC__{HEX(START_TOKEN)}x{HEX(CONTENT)}x{HEX(END_TOKEN)}~
+     */
+    public function encodeNowdocCallback(array $matches) {
+        return '~__EMU__NOWDOC__'
+                . bin2hex($matches[1]) . 'x' . bin2hex($matches[3]) . 'x' . bin2hex($matches[4])
+                . '__~';
+    }
+
+    /*
+     * Replaces the ~__EMU__...~ sequences with real tokens or their original
+     * value.
+     */
+    protected function postprocessTokens() {
+        // we need to manually iterate and manage a count because we'll change
+        // the tokens array on the way
         for ($i = 0, $c = count($this->tokens); $i < $c; ++$i) {
+            // first check that the following tokens are form ~LABEL~,
+            // then match the __EMU__... sequence.
             if ('~' === $this->tokens[$i]
                 && isset($this->tokens[$i + 2])
                 && '~' === $this->tokens[$i + 2]
@@ -55,10 +96,18 @@ class PHPParser_Lexer_Emulative extends PHPParser_Lexer
                 && preg_match('(^__EMU__([A-Z]++)__(?:([A-Za-z0-9]++)__)?$)', $this->tokens[$i + 1][1], $matches)
             ) {
                 if ('BINARY' === $matches[1]) {
-                    $replace = array(array(T_LNUMBER, $matches[2], $this->tokens[$i + 1][2]));
+                    // the binary number can either be an integer or a double, so return a LNUMBER
+                    // or DNUMBER respectively
+                    $replace = array(
+                        array(is_int(bindec($matches[2])) ? T_LNUMBER : T_DNUMBER, $matches[2], $this->tokens[$i + 1][2])
+                    );
                 } elseif ('NS' === $matches[1]) {
+                    // a \ single char token is returned here and replaced by a
+                    // PHPParser_Parser::T_NS_SEPARATOR token in ->lex(). This hacks around the
+                    // limitations arising from T_NS_SEPARATOR not being defined on 5.3
                     $replace = array('\\');
                 } elseif ('NOWDOC' === $matches[1]) {
+                    // decode the encoded nowdoc payload; pack('H*' is bin2hex( for 5.3
                     list($start, $content, $end) = explode('x', $matches[2]);
                     list($start, $content, $end) = array(pack('H*', $start), pack('H*', $content), pack('H*', $end));
 
@@ -69,11 +118,14 @@ class PHPParser_Lexer_Emulative extends PHPParser_Lexer
                     }
                     $replace[] = array(T_END_HEREDOC, $end, -1);
                 } else {
+                    // just ignore all other __EMU__ sequences
                     continue;
                 }
 
                 array_splice($this->tokens, $i, 3, $replace);
                 $c -= 3 - count($replace);
+            // for multichar tokens (e.g. strings) replace any ~__EMU__...~ sequences
+            // in their content with the original character sequence
             } elseif (is_array($this->tokens[$i])
                       && 0 !== strpos($this->tokens[$i][1], '__EMU__')
             ) {
@@ -86,12 +138,10 @@ class PHPParser_Lexer_Emulative extends PHPParser_Lexer
         }
     }
 
-    public function encodeNowdocCallback(array $matches) {
-        return '~__EMU__NOWDOC__'
-             . bin2hex($matches[1]) . 'x' . bin2hex($matches[3]) . 'x' . bin2hex($matches[4])
-             . '__~';
-    }
-
+    /*
+     * This method is a callback for restoring EMU sequences in
+     * multichar tokens (like strings) to their original value.
+     */
     public function restoreContentCallback(array $matches) {
         if ('BINARY' === $matches[1]) {
             return $matches[2];
@@ -108,12 +158,17 @@ class PHPParser_Lexer_Emulative extends PHPParser_Lexer
     public function lex(&$value = null, &$line = null, &$docComment = null) {
         $token = parent::lex($value, $line, $docComment);
 
+        // replace new keywords by their respective tokens. This is not done
+        // if we currently are in an object access (e.g. in $obj->namespace
+        // "namespace" stays a T_STRING tokens and isn't converted to T_NAMESPACE)
         if (PHPParser_Parser::T_STRING === $token && !$this->inObjectAccess) {
             if (isset(self::$keywords[strtolower($value)])) {
                 return self::$keywords[strtolower($value)];
             }
+        // backslashes are replaced by T_NS_SEPARATOR tokens
         } elseif (92 === $token) { // ord('\\')
             return PHPParser_Parser::T_NS_SEPARATOR;
+        // keep track of whether we currently are in an object access (after ->)
         } elseif (PHPParser_Parser::T_OBJECT_OPERATOR === $token) {
             $this->inObjectAccess = true;
         } else {

lib/PHPParser/Node.php

@@ -43,4 +43,38 @@ interface PHPParser_Node
      * @param null|string $docComment Nearest doc comment or null
      */
     public function setDocComment($docComment);
+
+    /**
+     * Sets an attribute on a node.
+     *
+     * @param string $key
+     * @param mixed  $value
+     */
+    public function setAttribute($key, $value);
+
+    /**
+     * Returns whether an attribute exists.
+     *
+     * @param string $key
+     *
+     * @return bool
+     */
+    public function hasAttribute($key);
+
+    /**
+     * Returns the value of an attribute.
+     *
+     * @param string $key
+     * @param mixed  $default
+     *
+     * @return mixed
+     */
+    public function getAttribute($key, $default = null);
+
+    /**
+     * Returns all attributes for the given node.
+     *
+     * @return array
+     */
+    public function getAttributes();
 }

lib/PHPParser/Node/Scalar/DNumber.php

@@ -20,4 +20,40 @@ class PHPParser_Node_Scalar_DNumber extends PHPParser_Node_Scalar
             $line, $docComment
         );
     }
+
+    /**
+     * Parses a DNUMBER token like PHP would.
+     *
+     * @param string $str A string number
+     *
+     * @return float The parsed number
+     */
+    public static function parse($str) {
+        // if string contains any of .eE just cast it to float
+        if (false !== strpbrk($str, '.eE')) {
+            return (float) $str;
+        }
+
+        // otherwise it's an integer notation that overflowed into a float
+        // if it starts with 0 it's one of the special integer notations
+        if ('0' === $str[0]) {
+            // hex
+            if ('x' === $str[1] || 'X' === $str[1]) {
+                return hexdec($str);
+            }
+
+            // bin
+            if ('b' === $str[1] || 'B' === $str[1]) {
+                return bindec($str);
+            }
+
+            // oct
+            // substr($str, 0, strcspn($str, '89')) cuts the string at the first invalid digit (8 or 9)
+            // so that only the digits before that are used
+            return octdec(substr($str, 0, strcspn($str, '89')));
+        }
+
+        // dec
+        return (float) $str;
+    }
 }

lib/PHPParser/Node/Stmt/Class.php

@@ -79,7 +79,7 @@ class PHPParser_Node_Stmt_Class extends PHPParser_Node_Stmt
         }
 
         if ($a & 48 && $b & 48) {
-            throw new PHPParser_Error('Cannot use the final modifier on an abstract class member');
+            throw new PHPParser_Error('Cannot use the final and abstract modifier at the same time');
         }
     }
 }

lib/PHPParser/NodeAbstract.php

@@ -5,6 +5,7 @@ abstract class PHPParser_NodeAbstract implements PHPParser_Node, IteratorAggrega
     protected $subNodes;
     protected $line;
     protected $docComment;
+    protected $attributes;
 
     /**
      * Creates a Node.
@@ -17,6 +18,7 @@ abstract class PHPParser_NodeAbstract implements PHPParser_Node, IteratorAggrega
         $this->subNodes   = $subNodes;
         $this->line       = $line;
         $this->docComment = $docComment;
+        $this->attributes = array();
     }
 
     /**
@@ -73,6 +75,34 @@ abstract class PHPParser_NodeAbstract implements PHPParser_Node, IteratorAggrega
         $this->docComment = $docComment;
     }
 
+    /**
+     * {@inheritDoc}
+     */
+    public function setAttribute($key, $value) {
+        $this->attributes[$key] = $value;
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    public function hasAttribute($key) {
+        return array_key_exists($key, $this->attributes);
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    public function getAttribute($key, $default = null) {
+        return array_key_exists($key, $this->attributes) ? $this->attributes[$key] : $default;
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    public function getAttributes() {
+        return $this->attributes;
+    }
+
     /* Magic interfaces */
 
     public function &__get($name) {

lib/PHPParser/NodeTraverser.php

@@ -91,19 +91,13 @@ class PHPParser_NodeTraverser
 
                 $node = $this->traverseNode($node);
 
-                foreach ($this->visitors as $j => $visitor) {
+                foreach ($this->visitors as $visitor) {
                     $return = $visitor->leaveNode($node);
 
                     if (false === $return) {
                         $doNodes[] = array($i, array());
                         break;
                     } elseif (is_array($return)) {
-                        // traverse replacement nodes using all visitors apart from the one that
-                        // did the change
-                        unset($this->visitors[$j]);
-                        $return = $this->traverseArray($return);
-                        $this->visitors[$j] = $visitor;
-
                         $doNodes[] = array($i, $return);
                         break;
                     } elseif (null !== $return) {

lib/PHPParser/Serializer/XML.php

@@ -29,7 +29,7 @@ class PHPParser_Serializer_XML implements PHPParser_Serializer
         return $this->writer->outputMemory();
     }
 
-    public function _serialize($node) {
+    protected function _serialize($node) {
          if ($node instanceof PHPParser_Node) {
             $this->writer->startElement('node:' . $node->getType());
 

lib/PHPParser/Template.php

@@ -0,0 +1,84 @@
+<?php
+
+class PHPParser_Template
+{
+    protected $parser;
+    protected $template;
+
+    /**
+     * Creates a new code template from a template string.
+     *
+     * @param PHPParser_Parser $parser   A parser instance
+     * @param string           $template The template string
+     */
+    public function __construct(PHPParser_Parser $parser, $template) {
+        $this->parser   = $parser;
+        $this->template = $template;
+    }
+
+    /**
+     * Get the statements of the template with the passed in placeholders
+     * replaced.
+     *
+     * @param array $placeholders Placeholders
+     *
+     * @return PHPParser_Node[] Statements
+     */
+    public function getStmts(array $placeholders) {
+        /*
+         * TODO This is evil.
+         * The lexer shouldn't be created in here, instead it should be a dependency, which
+         * basically means that we'd need to have a LexerFactory (which seems strange).
+         * An alternative solution would be to make the lexer work similar to how the parser
+         * works. I.e. one would instantiate the Lexer only once and then pass the results
+         * of ->lex() to the parser (which would then be the full tokens array). This design
+         * seems cleaner, but comes at the expense of higher memory consumption, as the token
+         * array can be quite large.
+         */
+        return $this->parser->parse(
+            new PHPParser_Lexer_Emulative(
+                $this->getTemplateWithPlaceholdersReplaced($placeholders)
+            )
+        );
+    }
+
+    protected function getTemplateWithPlaceholdersReplaced(array $placeholders) {
+        if (empty($placeholders)) {
+            return $this->template;
+        }
+
+        return strtr($this->template, $this->preparePlaceholders($placeholders));
+    }
+
+    /*
+     * Prepare the placeholders for replacement. This means that
+     * a) all placeholders will be surrounded with __.
+     * b) ucfirst/lcfirst variations of the placeholders are generated.
+     *
+     * E.g. for an input array of ['foo' => 'bar'] the result will be
+     * ['__foo__' => 'bar', '__Foo__' => 'Bar'].
+     */
+    protected function preparePlaceholders(array $placeholders) {
+        $preparedPlaceholders = array();
+
+        foreach ($placeholders as $name => $value) {
+            $preparedPlaceholders['__' . $name . '__'] = $value;
+
+            if (ctype_lower($name[0])) {
+                $ucfirstName = ucfirst($name);
+                if (!isset($placeholders[$ucfirstName])) {
+                    $preparedPlaceholders['__' . $ucfirstName . '__'] = ucfirst($value);
+                }
+            }
+
+            if (ctype_upper($name[0])) {
+                $lcfirstName = lcfirst($name);
+                if (!isset($placeholders[$lcfirstName])) {
+                    $preparedPlaceholders['__' . $lcfirstName . '__'] = lcfirst($value);
+                }
+            }
+        }
+
+        return $preparedPlaceholders;
+    }
+}

lib/PHPParser/TemplateLoader.php

@@ -0,0 +1,48 @@
+<?php
+
+class PHPParser_TemplateLoader
+{
+    protected $parser;
+    protected $baseDir;
+    protected $suffix;
+
+    /**
+     * Constructs a filesystem template loader.
+     *
+     * The templates are loaded from {baseDir}/{name}{suffix}.
+     *
+     * @param PHPParser_Parser $parser  A PHP parser instance
+     * @param string           $baseDir The base directory to load templates from
+     * @param string           $suffix  An optional suffix to append after the template name
+     */
+    public function __construct(PHPParser_Parser $parser, $baseDir, $suffix = '') {
+        if (!is_dir($baseDir)) {
+            throw new InvalidArgumentException(
+                sprintf('The specified base directory "%s" does not exist', $baseDir)
+            );
+        }
+
+        $this->parser  = $parser;
+        $this->baseDir = $baseDir;
+        $this->suffix  = $suffix;
+    }
+
+    /**
+     * Loads the template with the specified name.
+     *
+     * @param string $name The name of template
+     *
+     * @return PHPParser_Template The loaded template
+     */
+    public function load($name) {
+        $file = $this->baseDir . '/' . $name . $this->suffix;
+
+        if (!is_file($file)) {
+            throw new InvalidArgumentException(
+                sprintf('The file "%s" does not exist', $file)
+            );
+        }
+
+        return new PHPParser_Template($this->parser, file_get_contents($file));
+    }
+}

lib/PHPParser/Unserializer/XML.php

@@ -47,7 +47,10 @@ class PHPParser_Unserializer_XML implements PHPParser_Unserializer
 
         // create the node without calling it's constructor
         $node = unserialize(
-            sprintf('O:%d:"%s":0:{}', strlen($className), $className)
+            sprintf(
+                "O:%d:\"%s\":2:{s:11:\"\0*\0subNodes\";a:0:{}s:13:\"\0*\0attributes\";a:0:{}}",
+                strlen($className), $className
+            )
         );
 
         $line = $this->reader->getAttribute('line');

lib/bootstrap.php

@@ -2,3 +2,13 @@
 
 require dirname(__FILE__) . '/PHPParser/Autoloader.php';
 PHPParser_Autoloader::register();
+
+/*
+ * lcfirst() was added in PHP 5.3, so we have to emulate it for PHP 5.3.
+ */
+if (!function_exists('lcfirst')) {
+    function lcfirst($string) {
+        $string[0] = strtolower($string[0]);
+        return $string;
+    }
+}

phpunit.xml.dist

@@ -9,7 +9,7 @@
          processIsolation="false"
          stopOnFailure="false"
          syntaxCheck="false"
-         bootstrap="./test/bootstrap.php">
+         bootstrap="./lib/bootstrap.php">
     <testsuites>
         <testsuite name="PHPParser Test Suite">
             <directory>./test/</directory>

test/PHPParser/Tests/Builder/ClassTest.php

@@ -0,0 +1,91 @@
+<?php
+
+class PHPParser_Tests_Builder_ClassTest extends PHPUnit_Framework_TestCase
+{
+    protected function createClassBuilder($class) {
+        return new PHPParser_Builder_Class($class);
+    }
+
+    public function testExtendsImplements() {
+        $node = $this->createClassBuilder('SomeLogger')
+            ->extend('BaseLogger')
+            ->implement('Namespaced\Logger', new PHPParser_Node_Name('SomeInterface'))
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Class('SomeLogger', array(
+                'extends' => new PHPParser_Node_Name('BaseLogger'),
+                'implements' => array(
+                    new PHPParser_Node_Name('Namespaced\Logger'),
+                    new PHPParser_Node_Name('SomeInterface')
+                ),
+            )),
+            $node
+        );
+    }
+
+    public function testAbstract() {
+        $node = $this->createClassBuilder('Test')
+            ->makeAbstract()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Class('Test', array(
+                'type' => PHPParser_Node_Stmt_Class::MODIFIER_ABSTRACT
+            )),
+            $node
+        );
+    }
+
+    public function testFinal() {
+        $node = $this->createClassBuilder('Test')
+            ->makeFinal()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Class('Test', array(
+                'type' => PHPParser_Node_Stmt_Class::MODIFIER_FINAL
+            )),
+            $node
+        );
+    }
+
+    public function testStatementOrder() {
+        $method = new PHPParser_Node_Stmt_ClassMethod('testMethod');
+        $property = new PHPParser_Node_Stmt_Property(
+            PHPParser_Node_Stmt_Class::MODIFIER_PUBLIC,
+            array(new PHPParser_Node_Stmt_PropertyProperty('testProperty'))
+        );
+        $const = new PHPParser_Node_Stmt_ClassConst(array(
+            new PHPParser_Node_Const('TEST_CONST', new PHPParser_Node_Scalar_String('ABC'))
+        ));
+        $use = new PHPParser_Node_Stmt_TraitUse(array(new PHPParser_Node_Name('SomeTrait')));
+
+        $node = $this->createClassBuilder('Test')
+            ->addStmt($method)
+            ->addStmt($property)
+            ->addStmts(array($const, $use))
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Class('Test', array(
+                'stmts' => array($use, $const, $property, $method)
+            )),
+            $node
+        );
+    }
+
+    /**
+     * @expectedException LogicException
+     * @expectedExceptionMessage Unexpected node of type "Stmt_Echo"
+     */
+    public function testInvalidStmtError() {
+        $this->createClassBuilder('Test')
+            ->addStmt(new PHPParser_Node_Stmt_Echo(array()))
+        ;
+    }
+}

test/PHPParser/Tests/Builder/FunctionTest.php

@@ -0,0 +1,70 @@
+<?php
+
+class PHPParser_Tests_Builder_FunctionTest extends PHPUnit_Framework_TestCase
+{
+    public function createFunctionBuilder($name) {
+        return new PHPParser_Builder_Function($name);
+    }
+
+    public function testReturnByRef() {
+        $node = $this->createFunctionBuilder('test')
+            ->makeReturnByRef()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Function('test', array(
+                'byRef' => true
+            )),
+            $node
+        );
+    }
+
+    public function testParams() {
+        $param1 = new PHPParser_Node_Param('test1');
+        $param2 = new PHPParser_Node_Param('test2');
+        $param3 = new PHPParser_Node_Param('test3');
+
+        $node = $this->createFunctionBuilder('test')
+            ->addParam($param1)
+            ->addParams(array($param2, $param3))
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Function('test', array(
+                'params' => array($param1, $param2, $param3)
+            )),
+            $node
+        );
+    }
+
+    public function testStmts() {
+        $stmt1 = new PHPParser_Node_Expr_Print(new PHPParser_Node_Scalar_String('test1'));
+        $stmt2 = new PHPParser_Node_Expr_Print(new PHPParser_Node_Scalar_String('test2'));
+        $stmt3 = new PHPParser_Node_Expr_Print(new PHPParser_Node_Scalar_String('test3'));
+
+        $node = $this->createFunctionBuilder('test')
+            ->addStmt($stmt1)
+            ->addStmts(array($stmt2, $stmt3))
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Function('test', array(
+                'stmts' => array($stmt1, $stmt2, $stmt3)
+            )),
+            $node
+        );
+    }
+
+    /**
+     * @expectedException LogicException
+     * @expectedExceptionMessage Expected parameter node, got "Name"
+     */
+    public function testInvalidParamError() {
+        $this->createFunctionBuilder('test')
+            ->addParam(new PHPParser_Node_Name('foo'))
+        ;
+    }
+}

test/PHPParser/Tests/Builder/MethodTest.php

@@ -0,0 +1,137 @@
+<?php
+
+class PHPParser_Tests_Builder_MethodTest extends PHPUnit_Framework_TestCase
+{
+    public function createMethodBuilder($name) {
+        return new PHPParser_Builder_Method($name);
+    }
+
+    public function testModifiers() {
+        $node = $this->createMethodBuilder('test')
+            ->makePublic()
+            ->makeAbstract()
+            ->makeStatic()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_ClassMethod('test', array(
+                'type' => PHPParser_Node_Stmt_Class::MODIFIER_PUBLIC
+                        | PHPParser_Node_Stmt_Class::MODIFIER_ABSTRACT
+                        | PHPParser_Node_Stmt_Class::MODIFIER_STATIC,
+                'stmts' => null,
+            )),
+            $node
+        );
+
+        $node = $this->createMethodBuilder('test')
+            ->makeProtected()
+            ->makeFinal()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_ClassMethod('test', array(
+                'type' => PHPParser_Node_Stmt_Class::MODIFIER_PROTECTED
+                        | PHPParser_Node_Stmt_Class::MODIFIER_FINAL
+            )),
+            $node
+        );
+
+        $node = $this->createMethodBuilder('test')
+            ->makePrivate()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_ClassMethod('test', array(
+                'type' => PHPParser_Node_Stmt_Class::MODIFIER_PRIVATE
+            )),
+            $node
+        );
+    }
+
+    public function testReturnByRef() {
+        $node = $this->createMethodBuilder('test')
+            ->makeReturnByRef()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_ClassMethod('test', array(
+                'byRef' => true
+            )),
+            $node
+        );
+    }
+
+    public function testParams() {
+        $param1 = new PHPParser_Node_Param('test1');
+        $param2 = new PHPParser_Node_Param('test2');
+        $param3 = new PHPParser_Node_Param('test3');
+
+        $node = $this->createMethodBuilder('test')
+            ->addParam($param1)
+            ->addParams(array($param2, $param3))
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_ClassMethod('test', array(
+                'params' => array($param1, $param2, $param3)
+            )),
+            $node
+        );
+    }
+
+    public function testStmts() {
+        $stmt1 = new PHPParser_Node_Expr_Print(new PHPParser_Node_Scalar_String('test1'));
+        $stmt2 = new PHPParser_Node_Expr_Print(new PHPParser_Node_Scalar_String('test2'));
+        $stmt3 = new PHPParser_Node_Expr_Print(new PHPParser_Node_Scalar_String('test3'));
+
+        $node = $this->createMethodBuilder('test')
+            ->addStmt($stmt1)
+            ->addStmts(array($stmt2, $stmt3))
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_ClassMethod('test', array(
+                'stmts' => array($stmt1, $stmt2, $stmt3)
+            )),
+            $node
+        );
+    }
+
+    /**
+     * @expectedException LogicException
+     * @expectedExceptionMessage Cannot add statements to an abstract method
+     */
+    public function testAddStmtToAbstractMethodError() {
+        $this->createMethodBuilder('test')
+            ->makeAbstract()
+            ->addStmt(new PHPParser_Node_Expr_Print(new PHPParser_Node_Scalar_String('test')))
+        ;
+    }
+
+    /**
+     * @expectedException LogicException
+     * @expectedExceptionMessage Cannot make method with statements abstract
+     */
+    public function testMakeMethodWithStmtsAbstractError() {
+        $this->createMethodBuilder('test')
+            ->addStmt(new PHPParser_Node_Expr_Print(new PHPParser_Node_Scalar_String('test')))
+            ->makeAbstract()
+        ;
+    }
+
+    /**
+     * @expectedException LogicException
+     * @expectedExceptionMessage Expected parameter node, got "Name"
+     */
+    public function testInvalidParamError() {
+        $this->createMethodBuilder('test')
+            ->addParam(new PHPParser_Node_Name('foo'))
+        ;
+    }
+}

test/PHPParser/Tests/Builder/ParamTest.php

@@ -0,0 +1,118 @@
+<?php
+
+class PHPParser_Tests_Builder_ParamTest extends PHPUnit_Framework_TestCase
+{
+    public function createParamBuilder($name) {
+        return new PHPParser_Builder_Param($name);
+    }
+
+    /**
+     * @dataProvider provideTestDefaultValues
+     */
+    public function testDefaultValues($value, $expectedValueNode) {
+        $node = $this->createParamBuilder('test')
+            ->setDefault($value)
+            ->getNode()
+        ;
+
+        $this->assertEquals($expectedValueNode, $node->default);
+    }
+
+    public function provideTestDefaultValues() {
+        return array(
+            array(
+                null,
+                new PHPParser_Node_Expr_ConstFetch(new PHPParser_Node_Name('null'))
+            ),
+            array(
+                true,
+                new PHPParser_Node_Expr_ConstFetch(new PHPParser_Node_Name('true'))
+            ),
+            array(
+                false,
+                new PHPParser_Node_Expr_ConstFetch(new PHPParser_Node_Name('false'))
+            ),
+            array(
+                31415,
+                new PHPParser_Node_Scalar_LNumber(31415)
+            ),
+            array(
+                3.1415,
+                new PHPParser_Node_Scalar_DNumber(3.1415)
+            ),
+            array(
+                'Hallo World',
+                new PHPParser_Node_Scalar_String('Hallo World')
+            ),
+            array(
+                array(1, 2, 3),
+                new PHPParser_Node_Expr_Array(array(
+                    new PHPParser_Node_Expr_ArrayItem(new PHPParser_Node_Scalar_LNumber(1)),
+                    new PHPParser_Node_Expr_ArrayItem(new PHPParser_Node_Scalar_LNumber(2)),
+                    new PHPParser_Node_Expr_ArrayItem(new PHPParser_Node_Scalar_LNumber(3)),
+                ))
+            ),
+            array(
+                array('foo' => 'bar', 'bar' => 'foo'),
+                new PHPParser_Node_Expr_Array(array(
+                    new PHPParser_Node_Expr_ArrayItem(
+                        new PHPParser_Node_Scalar_String('bar'),
+                        new PHPParser_Node_Scalar_String('foo')
+                    ),
+                    new PHPParser_Node_Expr_ArrayItem(
+                        new PHPParser_Node_Scalar_String('foo'),
+                        new PHPParser_Node_Scalar_String('bar')
+                    ),
+                ))
+            ),
+            array(
+                new PHPParser_Node_Scalar_DirConst,
+                new PHPParser_Node_Scalar_DirConst
+            )
+        );
+    }
+
+    public function testTypeHints() {
+        $node = $this->createParamBuilder('test')
+            ->setTypeHint('array')
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Param('test', null, 'array'),
+            $node
+        );
+
+        $node = $this->createParamBuilder('test')
+            ->setTypeHint('callable')
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Param('test', null, 'callable'),
+            $node
+        );
+
+        $node = $this->createParamBuilder('test')
+            ->setTypeHint('Some\Class')
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Param('test', null, new PHPParser_Node_Name('Some\Class')),
+            $node
+        );
+    }
+
+    public function testByRef() {
+        $node = $this->createParamBuilder('test')
+            ->makeByRef()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Param('test', null, null, true),
+            $node
+        );
+    }
+}

test/PHPParser/Tests/Builder/PropertyTest.php

@@ -0,0 +1,123 @@
+<?php
+
+class PHPParser_Tests_Builder_PropertyTest extends PHPUnit_Framework_TestCase
+{
+    public function createPropertyBuilder($name) {
+        return new PHPParser_Builder_Property($name);
+    }
+
+    public function testModifiers() {
+        $node = $this->createPropertyBuilder('test')
+            ->makePrivate()
+            ->makeStatic()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Property(
+                PHPParser_Node_Stmt_Class::MODIFIER_PRIVATE
+              | PHPParser_Node_Stmt_Class::MODIFIER_STATIC,
+                array(
+                    new PHPParser_Node_Stmt_PropertyProperty('test')
+                )
+            ),
+            $node
+        );
+
+        $node = $this->createPropertyBuilder('test')
+            ->makeProtected()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Property(
+                PHPParser_Node_Stmt_Class::MODIFIER_PROTECTED,
+                array(
+                    new PHPParser_Node_Stmt_PropertyProperty('test')
+                )
+            ),
+            $node
+        );
+
+        $node = $this->createPropertyBuilder('test')
+            ->makePublic()
+            ->getNode()
+        ;
+
+        $this->assertEquals(
+            new PHPParser_Node_Stmt_Property(
+                PHPParser_Node_Stmt_Class::MODIFIER_PUBLIC,
+                array(
+                    new PHPParser_Node_Stmt_PropertyProperty('test')
+                )
+            ),
+            $node
+        );
+    }
+
+    /**
+     * @dataProvider provideTestDefaultValues
+     */
+    public function testDefaultValues($value, $expectedValueNode) {
+        $node = $this->createPropertyBuilder('test')
+            ->setDefault($value)
+            ->getNode()
+        ;
+
+        $this->assertEquals($expectedValueNode, $node->props[0]->default);
+    }
+
+    public function provideTestDefaultValues() {
+        return array(
+            array(
+                null,
+                new PHPParser_Node_Expr_ConstFetch(new PHPParser_Node_Name('null'))
+            ),
+            array(
+                true,
+                new PHPParser_Node_Expr_ConstFetch(new PHPParser_Node_Name('true'))
+            ),
+            array(
+                false,
+                new PHPParser_Node_Expr_ConstFetch(new PHPParser_Node_Name('false'))
+            ),
+            array(
+                31415,
+                new PHPParser_Node_Scalar_LNumber(31415)
+            ),
+            array(
+                3.1415,
+                new PHPParser_Node_Scalar_DNumber(3.1415)
+            ),
+            array(
+                'Hallo World',
+                new PHPParser_Node_Scalar_String('Hallo World')
+            ),
+            array(
+                array(1, 2, 3),
+                new PHPParser_Node_Expr_Array(array(
+                    new PHPParser_Node_Expr_ArrayItem(new PHPParser_Node_Scalar_LNumber(1)),
+                    new PHPParser_Node_Expr_ArrayItem(new PHPParser_Node_Scalar_LNumber(2)),
+                    new PHPParser_Node_Expr_ArrayItem(new PHPParser_Node_Scalar_LNumber(3)),
+                ))
+            ),
+            array(
+                array('foo' => 'bar', 'bar' => 'foo'),
+                new PHPParser_Node_Expr_Array(array(
+                    new PHPParser_Node_Expr_ArrayItem(
+                        new PHPParser_Node_Scalar_String('bar'),
+                        new PHPParser_Node_Scalar_String('foo')
+                    ),
+                    new PHPParser_Node_Expr_ArrayItem(
+                        new PHPParser_Node_Scalar_String('foo'),
+                        new PHPParser_Node_Scalar_String('bar')
+                    ),
+                ))
+            ),
+            array(
+                new PHPParser_Node_Scalar_DirConst,
+                new PHPParser_Node_Scalar_DirConst
+            )
+        );
+    }
+}

test/PHPParser/Tests/BuilderFactoryTest.php

@@ -0,0 +1,29 @@
+<?php
+
+class PHPParser_Tests_BuilderFactoryTest extends PHPUnit_Framework_TestCase
+{
+    public function testCreateClassBuilder() {
+        $factory = new PHPParser_BuilderFactory;
+        $this->assertInstanceOf('PHPParser_Builder_Class', $factory->class('Test'));
+    }
+
+    public function testCreateMethodBuilder() {
+        $factory = new PHPParser_BuilderFactory;
+        $this->assertInstanceOf('PHPParser_Builder_Method', $factory->method('test'));
+    }
+
+    public function testCreateParamBuilder() {
+        $factory = new PHPParser_BuilderFactory;
+        $this->assertInstanceOf('PHPParser_Builder_Param', $factory->param('test'));
+    }
+
+    public function testCreatePropertyBuilder() {
+        $factory = new PHPParser_BuilderFactory;
+        $this->assertInstanceOf('PHPParser_Builder_Property', $factory->property('test'));
+    }
+
+    public function testCreateFunctionBuilder() {
+        $factory = new PHPParser_BuilderFactory;
+        $this->assertInstanceOf('PHPParser_Builder_Function', $factory->function('test'));
+    }
+}

test/PHPParser/Tests/Lexer/EmulativeTest.php

@@ -0,0 +1,89 @@
+<?php
+
+class PHPParser_Tests_Lexer_EmulativeTest extends PHPUnit_Framework_TestCase
+{
+    /**
+     * @dataProvider provideTestReplaceKeywords
+     */
+    public function testReplaceKeywords($keyword, $expectedToken) {
+        $lexer = new PHPParser_Lexer_Emulative('<?php ' . $keyword);
+
+        $this->assertEquals($expectedToken, $lexer->lex());
+        $this->assertEquals(0, $lexer->lex());
+    }
+
+    /**
+     * @dataProvider provideTestReplaceKeywords
+     */
+    public function testNoReplaceKeywordsAfterObjectOperator($keyword) {
+        $lexer = new PHPParser_Lexer_Emulative('<?php ->' . $keyword);
+
+        $this->assertEquals(PHPParser_Parser::T_OBJECT_OPERATOR, $lexer->lex());
+        $this->assertEquals(PHPParser_Parser::T_STRING, $lexer->lex());
+        $this->assertEquals(0, $lexer->lex());
+    }
+
+    public function provideTestReplaceKeywords() {
+        return array(
+            array('callable',      PHPParser_Parser::T_CALLABLE),
+            array('insteadof',     PHPParser_Parser::T_INSTEADOF),
+            array('trait',         PHPParser_Parser::T_TRAIT),
+            array('__TRAIT__',     PHPParser_Parser::T_TRAIT_C),
+            array('__DIR__',       PHPParser_Parser::T_DIR),
+            array('goto',          PHPParser_Parser::T_GOTO),
+            array('namespace',     PHPParser_Parser::T_NAMESPACE),
+            array('__NAMESPACE__', PHPParser_Parser::T_NS_C),
+        );
+    }
+
+    /**
+     * @dataProvider provideTestLexNewFeatures
+     */
+    public function testLexNewFeatures($code, array $expectedTokens) {
+        $lexer = new PHPParser_Lexer_Emulative('<?php ' . $code);
+
+        foreach ($expectedTokens as $expectedToken) {
+            list($expectedTokenType, $expectedTokenText) = $expectedToken;
+            $this->assertEquals($expectedTokenType, $lexer->lex($text));
+            $this->assertEquals($expectedTokenText, $text);
+        }
+        $this->assertEquals(0, $lexer->lex());
+    }
+
+    /**
+     * @dataProvider provideTestLexNewFeatures
+     */
+    public function testLeaveStuffAloneInStrings($code) {
+        $stringifiedToken = '"' . addcslashes($code, '"\\') . '"';
+        $lexer = new PHPParser_Lexer_Emulative('<?php ' . $stringifiedToken);
+
+        $this->assertEquals(PHPParser_Parser::T_CONSTANT_ENCAPSED_STRING, $lexer->lex($text));
+        $this->assertEquals($stringifiedToken, $text);
+        $this->assertEquals(0, $lexer->lex());
+    }
+
+    public function provideTestLexNewFeatures() {
+        return array(
+            array('0b1010110', array(
+                array(PHPParser_Parser::T_LNUMBER, '0b1010110'),
+            )),
+            array('0b10110101010010101101010100101010110101010101011010110', array(
+                array(PHPParser_Parser::T_DNUMBER, '0b10110101010010101101010100101010110101010101011010110'),
+            )),
+            array('\\', array(
+                array(PHPParser_Parser::T_NS_SEPARATOR, '\\'),
+            )),
+            array("<<<'NOWDOC'\nNOWDOC;\n", array(
+                array(PHPParser_Parser::T_START_HEREDOC, "<<<'NOWDOC'\n"),
+                array(PHPParser_Parser::T_END_HEREDOC, 'NOWDOC'),
+                array(ord(';'), ';'),
+            )),
+            array("<<<'NOWDOC'\nFoobar\nNOWDOC;\n", array(
+                array(PHPParser_Parser::T_START_HEREDOC, "<<<'NOWDOC'\n"),
+                array(PHPParser_Parser::T_ENCAPSED_AND_WHITESPACE, "Foobar\n"),
+                array(PHPParser_Parser::T_END_HEREDOC, 'NOWDOC'),
+                array(ord(';'), ';'),
+            )),
+        );
+    }
+}

test/PHPParser/Tests/NodeAbstractTest.php

@@ -21,6 +21,7 @@ class PHPParser_Tests_NodeAbstractTest extends PHPUnit_Framework_TestCase
         $this->assertEquals('/** doc comment */', $node->getDocComment());
         $this->assertEquals('value', $node->subNode);
         $this->assertTrue(isset($node->subNode));
+        $this->assertEmpty($node->getAttributes());
 
         return $node;
     }
@@ -50,4 +51,32 @@ class PHPParser_Tests_NodeAbstractTest extends PHPUnit_Framework_TestCase
         unset($node->subNode);
         $this->assertFalse(isset($node->subNode));
     }
+
+    /**
+     * @depends testConstruct
+     */
+    public function testAttributes(PHPParser_NodeAbstract $node) {
+        $this->assertEmpty($node->getAttributes());
+
+        $node->setAttribute('key', 'value');
+        $this->assertTrue($node->hasAttribute('key'));
+        $this->assertEquals('value', $node->getAttribute('key'));
+
+        $this->assertFalse($node->hasAttribute('doesNotExist'));
+        $this->assertNull($node->getAttribute('doesNotExist'));
+        $this->assertEquals('default', $node->getAttribute('doesNotExist', 'default'));
+
+        $node->setAttribute('null', null);
+        $this->assertTrue($node->hasAttribute('null'));
+        $this->assertNull($node->getAttribute('null'));
+        $this->assertNull($node->getAttribute('null', 'default'));
+
+        $this->assertEquals(
+            array(
+                'key' => 'value',
+                'null' => null,
+            ),
+            $node->getAttributes()
+        );
+    }
 }

test/PHPParser/Tests/NodeVisitor/NameResolverTest.php

@@ -78,7 +78,7 @@ EOC;
         $traverser     = new PHPParser_NodeTraverser;
         $traverser->addVisitor(new PHPParser_NodeVisitor_NameResolver);
 
-        $stmts = $parser->parse(new PHPParser_Lexer($code));
+        $stmts = $parser->parse(new PHPParser_Lexer_Emulative($code));
         $stmts = $traverser->traverse($stmts);
 
         $this->assertEquals($expectedCode, $prettyPrinter->prettyPrint($stmts));
@@ -134,7 +134,7 @@ EOC;
         $traverser     = new PHPParser_NodeTraverser;
         $traverser->addVisitor(new PHPParser_NodeVisitor_NameResolver);
 
-        $stmts = $parser->parse(new PHPParser_Lexer($code));
+        $stmts = $parser->parse(new PHPParser_Lexer_Emulative($code));
         $stmts = $traverser->traverse($stmts);
 
         $this->assertEquals($expectedCode, $prettyPrinter->prettyPrint($stmts));
@@ -182,10 +182,6 @@ EOC;
     }
 
     public function testAddTraitNamespacedName() {
-        if (!version_compare(PHP_VERSION, '5.4.0RC1', '>=')) {
-            $this->markTestSkipped('The test requires PHP 5.4');
-        }
-
         $stmts = $this->createNamespacedAndNonNamespaced(array(
             new PHPParser_Node_Stmt_Trait('A')
         ));

test/PHPParser/Tests/TemplateLoaderTest.php

@@ -0,0 +1,44 @@
+<?php
+
+class PHPParser_Tests_TemplateLoaderTest extends PHPUnit_Framework_TestCase
+{
+    public function testLoadWithoutSuffix() {
+        $templateLoader = new PHPParser_TemplateLoader(
+            new PHPParser_Parser, dirname(__FILE__)
+        );
+
+        // load this file as a template, as we don't really care about the contents
+        $template = $templateLoader->load('TemplateLoaderTest.php');
+        $this->assertInstanceOf('PHPParser_Template', $template);
+    }
+
+    public function testLoadWithSuffix() {
+        $templateLoader = new PHPParser_TemplateLoader(
+            new PHPParser_Parser, dirname(__FILE__), '.php'
+        );
+
+        // load this file as a template, as we don't really care about the contents
+        $template = $templateLoader->load('TemplateLoaderTest');
+        $this->assertInstanceOf('PHPParser_Template', $template);
+    }
+
+    /**
+     * @expectedException InvalidArgumentException
+     */
+    public function testNonexistentBaseDirectoryError() {
+        new PHPParser_TemplateLoader(
+            new PHPParser_Parser, dirname(__FILE__) . '/someDirectoryThatDoesNotExist'
+        );
+    }
+
+    /**
+     * @expectedException InvalidArgumentException
+     */
+    public function testNonexistentFileError() {
+        $templateLoader = new PHPParser_TemplateLoader(
+            new PHPParser_Parser, dirname(__FILE__)
+        );
+
+        $templateLoader->load('SomeTemplateThatDoesNotExist');
+    }
+}

test/PHPParser/Tests/TemplateTest.php

@@ -0,0 +1,54 @@
+<?php
+
+class PHPParser_Tests_TemplateTest extends PHPUnit_Framework_TestCase
+{
+    /**
+     * @dataProvider provideTestPlaceholderReplacement
+     * @covers PHPParser_Template
+     */
+    public function testPlaceholderReplacement($templateCode, $placeholders, $expectedPrettyPrint) {
+        $parser = new PHPParser_Parser;
+        $prettyPrinter = new PHPParser_PrettyPrinter_Zend;
+
+        $template = new PHPParser_Template($parser, $templateCode);
+        $this->assertEquals(
+            $expectedPrettyPrint,
+            $prettyPrinter->prettyPrint($template->getStmts($placeholders))
+        );
+    }
+
+    public function provideTestPlaceholderReplacement() {
+        return array(
+            array(
+                '<?php $__name__ + $__Name__;',
+                array('name' => 'foo'),
+                '$foo + $Foo;'
+            ),
+            array(
+                '<?php $__name__ + $__Name__;',
+                array('Name' => 'Foo'),
+                '$foo + $Foo;'
+            ),
+            array(
+                '<?php $__name__ + $__Name__;',
+                array('name' => 'foo', 'Name' => 'Bar'),
+                '$foo + $Bar;'
+            ),
+            array(
+                '<?php $__name__ + $__Name__;',
+                array('Name' => 'Bar', 'name' => 'foo'),
+                '$foo + $Bar;'
+            ),
+            array(
+                '<?php $prefix__Name__Suffix;',
+                array('name' => 'infix'),
+                '$prefixInfixSuffix;'
+            ),
+            array(
+                '<?php $___name___;',
+                array('name' => 'foo'),
+                '$_foo_;'
+            )
+        );
+    }
+}

test/PHPParser/Tests/Unserializer/XMLTest.php

@@ -21,6 +21,21 @@ XML;
         );
     }
 
+    public function testEmptyNode() {
+        $xml = <<<XML
+<?xml version="1.0" encoding="UTF-8"?>
+<AST xmlns:node="http://nikic.github.com/PHPParser/XML/node" xmlns:subNode="http://nikic.github.com/PHPParser/XML/subNode" xmlns:scalar="http://nikic.github.com/PHPParser/XML/scalar">
+ <node:Scalar_ClassConst />
+</AST>
+XML;
+
+        $unserializer  = new PHPParser_Unserializer_XML;
+        $this->assertEquals(
+            new PHPParser_Node_Scalar_ClassConst,
+            $unserializer->unserialize($xml)
+        );
+    }
+
     public function testScalars() {
         $xml = <<<XML
 <?xml version="1.0" encoding="UTF-8"?>

test/PHPParser/Tests/codeTest.php

@@ -51,7 +51,11 @@ class PHPParser_Tests_codeTest extends PHPUnit_Framework_TestCase
             $fileContents = file_get_contents($file);
 
             // evaluate @@{expr}@@ expressions
-            $fileContents = preg_replace('/@@\{(.*?)\}@@/e', '$1', $fileContents);
+            $fileContents = preg_replace_callback(
+                '/@@\{(.*?)\}@@/',
+                array($this, 'evalCallback'),
+                $fileContents
+            );
 
             // parse sections
             $parts = array_map('trim', explode('-----', $fileContents));
@@ -78,4 +82,8 @@ class PHPParser_Tests_codeTest extends PHPUnit_Framework_TestCase
         // trim right side of all lines
         return implode("\n", array_map('rtrim', explode("\n", $str)));
     }
+
+    protected function evalCallback($matches) {
+        return eval('return ' . $matches[1] . ';');
+    }
 }

test/bootstrap.php

@@ -1,4 +0,0 @@
-<?php
-
-require_once dirname(__FILE__) . '/../lib/PHPParser/Autoloader.php';
-PHPParser_Autoloader::register();

test/code/scalar/constantString.test

@@ -45,7 +45,7 @@ array(
     9: Scalar_String(
         value: !"!\!$!
     !
-!@@{ chr(9) }@@!@@{ chr(12) }@@!@@{ chr(11) }@@!@@{ chr(27) }@@!\a
+!@@{ "\t" }@@!@@{ "\f" }@@!@@{ "\v" }@@!@@{ chr(27) /* "\e" */ }@@!\a
     )
     10: Scalar_String(
         value: !@@{ chr(255) }@@!@@{ chr(255) }@@!@@{ chr(0) }@@!@@{ chr(0) }@@!

test/code/scalar/float.test

@@ -12,6 +12,13 @@ Different float syntaxes
 30.20e10;
 300.200e100;
 1e10000;
+
+// various integer -> float overflows
+9999999999999999999;
+0xFFFFFFFFFFFFFFFF;
+07777777777777777777777;
+0777777777777777777777787;
+0b1111111111111111111111111111111111111111111111111111111111111111;
 -----
 array(
     0: Scalar_DNumber(
@@ -44,4 +51,19 @@ array(
     9: Scalar_DNumber(
         value: INF
     )
+    10: Scalar_DNumber(
+        value: 1.0E+19
+    )
+    11: Scalar_DNumber(
+        value: 1.844674407371E+19
+    )
+    12: Scalar_DNumber(
+        value: 7.3786976294838E+19
+    )
+    13: Scalar_DNumber(
+        value: 7.3786976294838E+19
+    )
+    14: Scalar_DNumber(
+        value: 1.844674407371E+19
+    )
 )

test/code/stmt/blocklessStatement.test

@@ -0,0 +1,112 @@
+Blockless statements for if/for/etc
+-----
+<?php
+
+if ($a) $A;
+elseif ($b) $B;
+else $C;
+
+for (;;) $foo;
+
+foreach ($a as $b) $AB;
+
+while ($a) $A;
+
+do $A; while ($a);
+
+declare (a='b') $C;
+-----
+array(
+    0: Stmt_If(
+        stmts: array(
+            0: Expr_Variable(
+                name: A
+            )
+        )
+        elseifs: array(
+            0: Stmt_ElseIf(
+                cond: Expr_Variable(
+                    name: b
+                )
+                stmts: array(
+                    0: Expr_Variable(
+                        name: B
+                    )
+                )
+            )
+        )
+        else: Stmt_Else(
+            stmts: array(
+                0: Expr_Variable(
+                    name: C
+                )
+            )
+        )
+        cond: Expr_Variable(
+            name: a
+        )
+    )
+    1: Stmt_For(
+        init: array(
+        )
+        cond: array(
+        )
+        loop: array(
+        )
+        stmts: array(
+            0: Expr_Variable(
+                name: foo
+            )
+        )
+    )
+    2: Stmt_Foreach(
+        keyVar: null
+        byRef: false
+        stmts: array(
+            0: Expr_Variable(
+                name: AB
+            )
+        )
+        expr: Expr_Variable(
+            name: a
+        )
+        valueVar: Expr_Variable(
+            name: b
+        )
+    )
+    3: Stmt_While(
+        cond: Expr_Variable(
+            name: a
+        )
+        stmts: array(
+            0: Expr_Variable(
+                name: A
+            )
+        )
+    )
+    4: Stmt_Do(
+        cond: Expr_Variable(
+            name: a
+        )
+        stmts: array(
+            0: Expr_Variable(
+                name: A
+            )
+        )
+    )
+    5: Stmt_Declare(
+        declares: array(
+            0: Stmt_DeclareDeclare(
+                key: a
+                value: Scalar_String(
+                    value: b
+                )
+            )
+        )
+        stmts: array(
+            0: Expr_Variable(
+                name: C
+            )
+        )
+    )
+)

test/code/stmt/class/modifier.test-fail

@@ -22,4 +22,8 @@ Multiple final modifiers are not allowed on line 1
 -----
 <?php class A { abstract final a(); }
 -----
-Cannot use the final modifier on an abstract class member on line 1
+Cannot use the final and abstract modifier at the same time on line 1
+-----
+<?php abstract final class A { }
+-----
+Unexpected token T_FINAL on line 1

test_old/run.php

@@ -17,11 +17,11 @@ $TEST_TYPE = $argv[1];
 $DIR = $argv[2];
 
 if ('Symfony' === $TEST_TYPE) {
-    $FILTER_FUNC = function ($path) {
+    function filter_func($path) {
         return preg_match('~\.php(?:\.cache)?$~', $path) && false === strpos($path, 'skeleton');
     };
 } elseif ('PHP' === $TEST_TYPE) {
-    $FILTER_FUNC = function ($path) {
+    function filter_func($path) {
         return preg_match('~\.phpt$~', $path);
     };
 } else {
@@ -44,7 +44,7 @@ foreach (new RecursiveIteratorIterator(
              new RecursiveDirectoryIterator($DIR),
              RecursiveIteratorIterator::LEAVES_ONLY)
          as $file) {
-    if (!$FILTER_FUNC($file)) {
+    if (!filter_func($file)) {
         continue;
     }
 
@@ -87,7 +87,7 @@ foreach (new RecursiveIteratorIterator(
 
     try {
         $startTime = microtime(true);
-        $stmts = $parser->parse(new PHPParser_Lexer($code));
+        $stmts = $parser->parse(new PHPParser_Lexer_Emulative($code));
         $parseTime += microtime(true) - $startTime;
 
         $startTime = microtime(true);
@@ -96,7 +96,7 @@ foreach (new RecursiveIteratorIterator(
 
         try {
             $startTime = microtime(true);
-            $ppStmts = $parser->parse(new PHPParser_Lexer($code));
+            $ppStmts = $parser->parse(new PHPParser_Lexer_Emulative($code));
             $reparseTime += microtime(true) - $startTime;
 
             $startTime = microtime(true);