Create LazyCollection from API resource

Laravel 6 introduced the LazyCollection, a different kind of collection which exploits php generators to save memory usage.

While data fetching from database is natively managed by eloquent, Web API-based data fetching is not provided.

This article describes how to use a paginated api endpoint to fetch data to a LazyCollection instance.

With this approach, client is not required to deal with page numbers or subsequent api calls. In addition, by using generators, api calls are made transparently behind the scenes only when they are actually required, resulting in memory and time savings.

Steps

• Create test application
• Create dummy data
• Create an API endpoint
• Create API client

Create test application

Let's start by creating a blank laravel application:

laravel new TestGenerators

Then, since we will use an http client, include the GuzzleHttp library:

composer require guzzlehttp/guzzle

Make test model

This step is required to create test data. Create a new database for the test application:

create database TestGenerators;

Generate a test Model along with migrations:

php artisan make:model Test -m

Create Test model migration as follows

    public function up()
    {
        Schema::create('tests', function (Blueprint $table) {
            $table->bigIncrements('id');
            $table->string('value');
            $table->timestamps();
        });
    }

Run migrations:

php artisan migrate

Now, generate a factory for the Test model:

php artisan make:factory TestFactory --model=Test

and define TestFactory as follows:

 $factory->define(Test::class, function (Faker $faker) {
    return [
        'value' => $faker->word,
    ];
});

In order not to incur the MassAssignmentException error, open App\Test.php and make all the field mass assignable:

class Test extends Model
{
    protected $guarded = [];
}

Finally create a lot of dummy data:

php artisan tinker

factory('App\Test', 100000)->create();

Create an API endpoint

This step is required to create an endpoint for fetching dummy data.

Open routes/api.php and add the following endpoint:

Route::get('/test', function () {

    return \App\Test::paginate();

});

this will create an api endpoint for retrieving data.

Note: you would need to disable the throttle middleware from app/Http/Kernel.php to avoid endpoint protection from too many requests:

        'api' => [
//            'throttle:60,1',
            'bindings',
        ],

To test the endpoint, simply point your browser to http://<your-host>/api/test, you should see the first page of data.

Create Repository interface and implementations

Before moving to generators, let's create a normal collection. The idea is to load all the api pages, one after the other and merge all the results in a single array.

So, we will implement an API repository, with the following interface:

<?php

namespace App;


interface TestRepositoryInterface
{
    public function cursor($url, $options = null);
}

Create a class with a single public method cursor($url) which will return a Collection instance:

class TestRepository implements TestRepositoryInterface
{

    public function cursor($url, $options = null)
    {
        if ($options == null) {
            $options = [
                'timeout' => 2.0,
            ];
        }

        $nextPage = 1;
        $lastPage = 1;

        $result = [];

        $client = new Client($options);

        while ($nextPage <= $lastPage) {
            list($data, $nextPage, $lastPage) = $this->getNextPage($client, $nextPage, $url);
            $result = array_merge($result, $data);
        }

        return Collection::make($result);
    }


    private function getNextPage(Client $client, int $nextPage, string $url): array
    {
        $response = $client->request('GET', $url . '?page=' . $nextPage);
        $data = json_decode($response->getBody());

        $nextPage = $data->current_page + 1;
        $lastPage = $data->last_page;

        return array($data->data, $nextPage, $lastPage);
    }
}

The core of the class is the while loop, which gets data from one page and merge to create a final $result array.

Data are thus stored in memory and this will become easily unmanageable whene there are lot of data.

In order to exploit generators features, the only modifications required are these:

    public function cursor($url)
    {
        return LazyCollection::make(function () use ($url) {
            $nextPage = 1;
            $lastPage = 1;

            $client = new Client([
                'base_uri' => url('api') . '/',
                'timeout' => 2.0,
            ]);

            while ($nextPage <= $lastPage) {
                list($data, $nextPage, $lastPage) = $this->getNextPage($client, $nextPage, $url);

                yield from $data;
            }
        });
    }

The logic is wrapped in a callback for a LazyCollection generation, and, instead of merging data and returning a final value, each page is yielded into the generator. Now, you can assume you have the full dataset. However, what's most important is that data pages are loaded only if they are actually required.

Testing

For testing this approach, create a new command:

php artisan make:command TestApiCommand

<?php

namespace App\Console\Commands;

use App\TestRepositoryGenerators;
use Illuminate\Console\Command;

class TestApiCommand extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'test:api';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Tests the generator api repository';

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {
        parent::__construct();
    }

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {
        $iterator = new TestRepository();
        $data = $iterator->cursor(url('api/test'));

        dump($data->first());
        dump($data->skip(1000)->first());
    }
}

You should see in your console the first and tenth element. Note that unused pages would never be loaded.

Final notes

If you have doubts or request, please leave a comment below.