Create LazyCollection from API resource
Laravel 6 introduced the LazyCollection, a different kind of collection which exploits php generators to save memory usage. While data fetching from database is natively managed by eloquent, Web API-based data fetching is not provided. This article describes how to use a paginated api endpoint to fetch data to a LazyCollection instance.
Create LazyCollection from API resource
Laravel 6 introduced the LazyCollection
, a different kind of collection which exploits php generators to save memory usage.
While data fetching from database is natively managed by eloquent, Web API-based data fetching is not provided.
This article describes how to use a paginated api endpoint to fetch data to a LazyCollection
instance.
With this approach, client is not required to deal with page numbers or subsequent api calls. In addition, by using generators, api calls are made transparently behind the scenes only when they are actually required, resulting in memory and time savings.
Steps
- Create test application
- Create dummy data
- Create an API endpoint
- Create API client
Create test application
Let's start by creating a blank laravel application:
laravel new TestGenerators
Then, since we will use an http client, include the GuzzleHttp
library:
composer require guzzlehttp/guzzle
Make test model
This step is required to create test data. Create a new database for the test application:
create database TestGenerators;
Generate a test Model along with migrations:
php artisan make:model Test -m
Create Test
model migration as follows
public function up()
{
Schema::create('tests', function (Blueprint $table) {
$table->bigIncrements('id');
$table->string('value');
$table->timestamps();
});
}
Run migrations:
php artisan migrate
Now, generate a factory for the Test
model:
php artisan make:factory TestFactory --model=Test
and define TestFactory
as follows:
$factory->define(Test::class, function (Faker $faker) {
return [
'value' => $faker->word,
];
});
In order not to incur the MassAssignmentException
error, open App\Test.php
and make all the field mass assignable:
class Test extends Model
{
protected $guarded = [];
}
Finally create a lot of dummy data:
php artisan tinker
factory('App\Test', 100000)->create();
Create an API endpoint
This step is required to create an endpoint for fetching dummy data.
Open routes/api.php
and add the following endpoint:
Route::get('/test', function () {
return \App\Test::paginate();
});
this will create an api endpoint for retrieving data.
Note: you would need to disable the throttle
middleware from app/Http/Kernel.php to avoid endpoint protection from too many requests:
'api' => [
// 'throttle:60,1',
'bindings',
],
To test the endpoint, simply point your browser to http://<your-host>/api/test
, you should see the first page of data.
Create Repository interface and implementations
Before moving to generators, let's create a normal collection. The idea is to load all the api pages, one after the other and merge all the results in a single array.
So, we will implement an API repository, with the following interface:
<?php
namespace App;
interface TestRepositoryInterface
{
public function cursor($url, $options = null);
}
Create a class with a single public method cursor($url)
which will return a Collection
instance:
class TestRepository implements TestRepositoryInterface
{
public function cursor($url, $options = null)
{
if ($options == null) {
$options = [
'timeout' => 2.0,
];
}
$nextPage = 1;
$lastPage = 1;
$result = [];
$client = new Client($options);
while ($nextPage <= $lastPage) {
list($data, $nextPage, $lastPage) = $this->getNextPage($client, $nextPage, $url);
$result = array_merge($result, $data);
}
return Collection::make($result);
}
private function getNextPage(Client $client, int $nextPage, string $url): array
{
$response = $client->request('GET', $url . '?page=' . $nextPage);
$data = json_decode($response->getBody());
$nextPage = $data->current_page + 1;
$lastPage = $data->last_page;
return array($data->data, $nextPage, $lastPage);
}
}
The core of the class is the while
loop, which gets data from one page and merge to create a final $result
array.
Data are thus stored in memory and this will become easily unmanageable whene there are lot of data.
In order to exploit generators features, the only modifications required are these:
public function cursor($url)
{
return LazyCollection::make(function () use ($url) {
$nextPage = 1;
$lastPage = 1;
$client = new Client([
'base_uri' => url('api') . '/',
'timeout' => 2.0,
]);
while ($nextPage <= $lastPage) {
list($data, $nextPage, $lastPage) = $this->getNextPage($client, $nextPage, $url);
yield from $data;
}
});
}
The logic is wrapped in a callback for a LazyCollection
generation, and, instead of merging data and returning a final value, each page is yield
ed into the generator. Now, you can assume you have the full dataset. However, what's most important is that data pages are loaded only if they are actually required.
Testing
For testing this approach, create a new command:
php artisan make:command TestApiCommand
<?php
namespace App\Console\Commands;
use App\TestRepositoryGenerators;
use Illuminate\Console\Command;
class TestApiCommand extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'test:api';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Tests the generator api repository';
/**
* Create a new command instance.
*
* @return void
*/
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* @return mixed
*/
public function handle()
{
$iterator = new TestRepository();
$data = $iterator->cursor(url('api/test'));
dump($data->first());
dump($data->skip(1000)->first());
}
}
You should see in your console the first and tenth element. Note that unused pages would never be loaded.
Final notes
If you have doubts or request, please leave a comment below.