🧩 JSON Parser

Zero-dependencies pull parser to read large JSON from any source in a memory-efficient way.

📦 Install

Via Composer:

composer require cerbero/json-parser

🔮 Usage

• Basics
• Sources
• Pointers
• Decoders
• Progress

Basics

JSON Parser provides a minimal API to read large JSON from any source:

use Cerbero\JsonParser\JsonParser;

// the JSON source in this example is an API endpoint
$source = 'https://randomuser.me/api/1.4?seed=json-parser&results=5';

foreach (new JsonParser($source) as $key => $value) {
    // instead of loading the whole JSON, we keep in memory only one key and value at a time
}

Depending on our taste, we can instantiate the parser in 3 different ways:

use Cerbero\JsonParser\JsonParser;

// classic object instantiation
new JsonParser($source);

// static instantiation, facilitates methods chaining
JsonParser::parse($source);

// namespaced function
use function Cerbero\JsonParser\parseJson;

parseJson($source);

If we don't want to use foreach() to loop through each key and value, we can chain the traverse() method:

JsonParser::parse($source)->traverse(function (mixed $value, string|int $key, JsonParser $parser) {
    // lazily load one key and value at a time, we can also access the parser if needed
});

// no foreach needed

⚠️ Please note the parameters order of the callback: the value is passed before the key.

Sources

A wide range of JSON sources is supported, here is the full list:

• strings, e.g. {"foo":"bar"}
• iterables, i.e. arrays or instances of Traversable
• files, e.g. /path/to/large_file.json
• resources, e.g. streams
• API endpoint URLs, e.g. https://endpoint.json or any instance of Psr\Http\Message\UriInterface
• PSR-7 requests, i.e. any instance of Psr\Http\Message\RequestInterface
• PSR-7 messages, i.e. any instance of Psr\Http\Message\MessageInterface
• PSR-7 streams, i.e. any instance of Psr\Http\Message\StreamInterface
• Laravel HTTP client responses, i.e. any instance of Illuminate\Http\Client\Response
• user-defined sources, i.e. any instance of Cerbero\JsonParser\Sources\Source

If the source we need to parse is not supported by default, we can implement our own custom source.

Click here to see how to implement a custom source.

To implement a custom source, we need to extend Source and implement 3 methods:

use Cerbero\JsonParser\Sources\Source;
use Traversable;

class CustomSource extends Source
{
    public function getIterator(): Traversable
    {
        // return a Traversable holding the JSON source, e.g. a Generator yielding chunks of JSON
    }

    public function matches(): bool
    {
        // return TRUE if this class can handle the JSON source
    }

    protected function calculateSize(): ?int
    {
        // return the size of the JSON in bytes or NULL if it can't be calculated
    }
}

The parent class Source gives us access to 2 properties:

• $source: the JSON source we pass to the parser, i.e.: new JsonParser($source)
• $config: the configuration we set by chaining methods, e.g.: $parser->pointer('/foo')

The method getIterator() defines the logic to read the JSON source in a memory-efficient way. It feeds the parser with small pieces of JSON. Please refer to the already existing sources to see some implementations.

The method matches() determines whether the JSON source passed to the parser can be handled by our custom implementation. In other words, we are telling the parser if it should use our class for the JSON to parse.

Finally, calculateSize() computes the whole size of the JSON source. It's used to track the parsing progress, however it's not always possible to know the size of a JSON source. In this case, or if we don't need to track the progress, we can return null.

Now that we have implemented our custom source, we can pass it to the parser:

$json = JsonParser::parse(new CustomSource($source));

foreach ($json as $key => $value) {
    // process one key and value of $source at a time
}

If you find yourself implementing the same custom source in different projects, feel free to send a PR and we will consider to support your custom source by default. Thank you in advance for any contribution!

Pointers

A JSON pointer is a standard used to point to nodes within a JSON. This package leverages JSON pointers to extract only some sub-trees from large JSONs.

Consider this JSON for example. To extract only the first gender and avoid parsing the rest of the JSON, we can set the /0/gender pointer:

$json = JsonParser::parse($source)->pointer('/0/gender');

foreach ($json as $key => $value) {
    // 1st and only iteration: $key === 'gender', $value === 'female'
}

JSON Parser takes advantage of the - character to point to any array index, so we can extract all the genders with the /-/gender pointer:

$json = JsonParser::parse($source)->pointer('/-/gender');

foreach ($json as $key => $value) {
    // 1st iteration: $key === 'gender', $value === 'female'
    // 2nd iteration: $key === 'gender', $value === 'female'
    // 3rd iteration: $key === 'gender', $value === 'male'
    // and so on for all the objects in the array...
}

If we want to extract more sub-trees, we can set multiple pointers. Let's extract all genders and countries:

$json = JsonParser::parse($source)->pointers(['/-/gender', '/-/location/country']);

foreach ($json as $key => $value) {
    // 1st iteration: $key === 'gender', $value === 'female'
    // 2nd iteration: $key === 'country', $value === 'Germany'
    // 3rd iteration: $key === 'gender', $value === 'female'
    // 4th iteration: $key === 'country', $value === 'Mexico'
    // and so on for all the objects in the array...
}

⚠️ Please avoid intersecting pointers (e.g. setting both /foo and /foo/bar) as the deeper pointer won't be found and will force the parser to parse the whole JSON.

We can also specify a callback to execute when JSON pointers are found. This is handy when we have different pointers and we need to run custom logic for each of them:

$json = JsonParser::parse($source)->pointers([
    '/-/gender' => fn (string $gender, string $key) => new Gender($gender),
    '/-/location/country' => fn (string $country, string $key) => new Country($country),
]);

foreach ($json as $key => $value) {
    // 1st iteration: $key === 'gender', $value instanceof Gender
    // 2nd iteration: $key === 'country', $value instanceof Country
    // and so on for all the objects in the array...
}

⚠️ Please note the parameters order of the callbacks: the value is passed before the key.

The same can also be achieved by chaining the method pointer() multiple times:

$json = JsonParser::parse($source)
    ->pointer('/-/gender', fn (string $gender, string $key) => new Gender($gender))
    ->pointer('/-/location/country', fn (string $country, string $key) => new Country($country));

foreach ($json as $key => $value) {
    // 1st iteration: $key === 'gender', $value instanceof Gender
    // 2nd iteration: $key === 'country', $value instanceof Country
    // and so on for all the objects in the array...
}

If the callbacks are enough to handle the pointers and we don't need to run any common logic for all pointers, we can avoid to manually call foreach() by chaining the method traverse():

JsonParser::parse($source)
    ->pointer('/-/gender', $this->handleGender(...))
    ->pointer('/-/location/country', $this->handleCountry(...))
    ->traverse();

// no foreach needed

Otherwise if some common logic for all pointers is needed but we prefer methods chaining to manual loops, we can pass a callback to the traverse() method:

JsonParser::parse($source)
    ->pointer('/-/gender', fn (string $gender, string $key) => new Gender($gender))
    ->pointer('/-/location/country', fn (string $country, string $key) => new Country($country))
    ->traverse(function (Gender|Country $value, string $key, JsonParser $parser) {
        // 1st iteration: $key === 'gender', $value instanceof Gender
        // 2nd iteration: $key === 'country', $value instanceof Country
        // and so on for all the objects in the array...
    });

// no foreach needed

⚠️ Please note the parameters order of the callbacks: the value is passed before the key.

Decoders

By default JSON Parser uses the built-in PHP function json_decode() to decode one key and value at a time.

Normally it decodes values to associative arrays but, if we prefer to decode values to objects, we can set a custom decoder:

use Cerbero\JsonParser\Decoders\JsonDecoder;

JsonParser::parse($source)->decoder(new JsonDecoder(decodesToArray: false));

JSON Parser also provides a convenient method to set the simdjson decoder:

JsonParser::parse($source)->simdjson(); // decode JSON to associative arrays using simdjson

JsonParser::parse($source)->simdjson(decodesToArray: false); // decode JSON to objects using simdjson

Simdjson is faster than json_decode() and can be installed via pecl install simdjson if your server satisfies the requirements.

If we need a decoder that is not supported by default, we can implement our custom one.

Click here to see how to implement a custom decoder.

To create a custom decoder, we need to implement the Decoder interface and implement 1 method:

use Cerbero\JsonParser\Decoders\Decoder;
use Cerbero\JsonParser\Decoders\DecodedValue;

class CustomDecoder implements Decoder
{
    public function decode(string $json): DecodedValue
    {
        // return an instance of DecodedValue both in case of success or failure
    }
}

The method decode() defines the logic to decode the given JSON value and it needs to return an instance of DecodedValue both in case of success or failure.

To make custom decoder implementations even easier, JSON Parser provides an abstract decoder that hydrates DecodedValue for us so that we just need to define how a JSON value should be decoded:

use Cerbero\JsonParser\Decoders\AbstractDecoder;

class CustomDecoder extends AbstractDecoder
{
    protected function decodeJson(string $json): mixed
    {
        // decode the given JSON or throw an exception on failure
        return json_decode($json, flags: JSON_THROW_ON_ERROR);
    }
}

⚠️ Please make sure to throw an exception in decodeJson() if the decoding process fails.

Now that we have implemented our custom decoder, we can set it like this:

JsonParser::parse($source)->decoder(new CustomDecoder());

To see some implementation examples, please refer to the already existing decoders.

If you find yourself implementing the same custom decoder in different projects, feel free to send a PR and we will consider to support your custom decoder by default. Thank you in advance for any contribution!

Progress

When processing large JSONs, we may need to know the parsing progress. JSON Parser offers convenient methods to access all the progress details:

$json = new JsonParser($source);

$json->progress(); // <Cerbero\JsonParser\Progress>
$json->progress()->current(); // the already parsed bytes e.g. 86759341
$json->progress()->total(); // the total bytes to parse e.g. 182332642
$json->progress()->fraction(); // the completed fraction e.g. 0.47583
$json->progress()->percentage(); // the completed percentage e.g. 47.583
$json->progress()->format(); // the formatted progress e.g. 47.5%

The total size of a JSON is calculated differently depending on the source. It is not always possible to determine how large a JSON is, in these cases only the current progress is known:

$json->progress()->current(); // 86759341
$json->progress()->total(); // null
$json->progress()->fraction(); // null
$json->progress()->percentage(); // null
$json->progress()->format(); // null

📆 Change log

Please see CHANGELOG for more information on what has changed recently.

🧪 Testing

composer test

💞 Contributing

Please see CONTRIBUTING and CODE_OF_CONDUCT for details.

🧯 Security

If you discover any security related issues, please email andrea.marco.sartori@gmail.com instead of using the issue tracker.

🏅 Credits

• Andrea Marco Sartori
• All Contributors

⚖️ License

The MIT License (MIT). Please see License File for more information.