How to recursively get results per page and finally return them into one result set.
The Challenge
Most of the time while working on a JavaScript project, I write some code to call REST APIs. This is normal.
However, sometimes a REST API could have restrictions on the number of items it returns per request. In other words, while calling the API to get some items, these items would be divided into indexed pages/groups and you have to provide the page/group index. This would return only the items in this page/group. To get the rest, you would need to repeat the same API call but now with a different index, and so on…
How To Do It?
This is what I am going to show you in this article. We are going to use a simple example of an API which returns items into pages and see how to consume this API and get all items.
We will simulate the API call with a function which returns a promise and a timeout call. The items would be retrieved from a static array of items which we would define.
Also, important to note that the logic of splitting the items of the static array into pages is based on the analysis, equations, and code provided on the article Paging/Partitioning — Learn the Main Equations to Make it Easy.
Here, in this article, I would just include the code for brevity.
Let’s Write Some Code
Array Page Function
Refer to this article for details.
Array.prototype.page = function(pageSize) {
const self = this;
let actualPageSize = pageSize;
if (actualPageSize <= 0) {
actualPageSize = self.length;
}
let maxNumberOfPages = Math.max(1, Math.ceil(parseFloat(parseFloat(self.length) / parseFloat(actualPageSize))));
let pagesBoundries = {};
for (let pageZeroIndex = 0; pageZeroIndex < maxNumberOfPages; pageZeroIndex++) {
pagesBoundries[pageZeroIndex] = {
firstItemZeroIndex: (pageZeroIndex * actualPageSize),
lastItemZeroIndex: Math.min((pageZeroIndex * actualPageSize) + (actualPageSize - 1), self.length - 1)
};
}
return {
actualPageSize: actualPageSize,
numberOfPages: maxNumberOfPages,
pagesBoundries: pagesBoundries,
find: function(itemZeroIndex) {
return {
pageZeroIndex: parseInt(parseInt(itemZeroIndex) / parseInt(actualPageSize)),
itemZeroIndexInsidePage: parseInt(parseInt(itemZeroIndex) % parseInt(actualPageSize))
};
}
};
};Static Items Array and Default Page-Size
const pageSize = 3;
const items = [{
name: "Item1",
id: 1
}, {
name: "Item2",
id: 2
}, {
name: "Item3",
id: 3
}, {
name: "Item4",
id: 4
}, {
name: "Item5",
id: 5
}, {
name: "Item6",
id: 6
}, {
name: "Item7",
id: 7
}, {
name: "Item8",
id: 8
}, {
name: "Item9",
id: 9
}, {
name: "Item10",
id: 10
}];Execute API Call Function
const executeApiCall = function(pageZeroIndex) {
const pagingResult = items.page(pageSize);
const pageBoundries = pagingResult.pagesBoundries[pageZeroIndex];
const itemsToReturn = items.slice(pageBoundries.firstItemZeroIndex, pageBoundries.lastItemZeroIndex + 1);
const result = {
items: itemsToReturn,
paging: {
pageZeroIndex: pageZeroIndex,
totalPagesCount: pagingResult.numberOfPages
}
};
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve(result);
}, 1000);
});
};What we can notice here:
This is the function which simulates the API call.
First it divides the static items array into pages using the static pageSize variable. Depending on the passed in pageZeroIndex parameter, it sets the itemsToReturn internal array.
Then it wraps this into a final object which has a paging.pageZeroIndex and page.totalPagesCount members. These should be used by the API caller to know if there are other pages or not.
Finally, this final object is resolved in a promise after one second.
Getting Results Using Old-fashioned Promises
const getResults = function() {
console.log('Started getting results.');
return getResultsRecursively(0);
};
const getResultsRecursively = function(pageIndex) {
console.log(`Getting results of page ${pageIndex} started.`);
let foundItems = [];
return new Promise((resolve, reject) => {
try {
executeApiCall(pageIndex)
.then((response) => {
if (response.items && response.items.length > 0) {
foundItems = foundItems.concat(response.items);
}
console.log(`Getting results of page ${pageIndex} ended.`);
if ((pageIndex + 1) < response.paging.totalPagesCount) {
getResultsRecursively(pageIndex + 1)
.then(nextBatch => {
if (nextBatch && nextBatch.length > 0) {
foundItems = foundItems.concat(nextBatch);
}
resolve(foundItems);
})
.catch((error) => {
console.log(`Error while getting results of ${pageIndex + 1}`);
console.log(error);
reject(new Error(error));
});
} else {
resolve(foundItems);
}
})
.catch((error) => {
console.log(`Error while getting results of ${pageIndex}`);
console.log(error);
reject(new Error(error));
});
} catch (error) {
console.log(error);
reject(new Error(error));
}
});
};What we can notice here:
We are calling the API in recursive manner but only resolving at the end of the items.
We are using page.totalPagesCount to decide if there could be another page or not.
Testing Calling the Old-fashioned Code
getResults()
.then((results) => {
console.log('results', JSON.stringify(results));
alert('done');
})
.catch((error) => {
console.log(error);
});And the result would be the following:
Getting Results Using async/await
const getResults = async function() {
console.log('Started getting results.');
let foundItems = [];
let loop = true;
let pageIndex = 0;
do {
console.log(`Getting results of page ${pageIndex} started.`);
var response = await executeApiCall(pageIndex);
console.log(`Getting results of page ${pageIndex} ended.`);
if (response.items && response.items.length > 0) {
foundItems = foundItems.concat(response.items);
}
loop = ((pageIndex + 1) < response.paging.totalPagesCount);
pageIndex++;
}
while (loop);
return foundItems;
};What we can notice here is that we are doing the same as in the old-fashioned code but here it is simpler.
Testing Calling the async/await Code
const results = await getResults();
console.log('results', JSON.stringify(results));
alert('done');And the result would be the following:
That’s it, hope you found reading this story as interesting as I found writing it.
Comments