Integrate Polling Based API In Node/Express Js

At times we need to integrate in our Node/Express application third party APIs that have implemented polling mechanism i.e. they only send back results once available; the client must keep on attempting until the service has put together all the results to return. Usually data providers of hotel, travel, tourism, property have polling mechanism in place for their APIs.

In Node js application it’s fairly easy to implement. There might be multiple ways but here I’ll be using async library and its whilst control flow method. Depending on the goal, this piece of code can be used as a stand alone script (to run in background for instance) or placed in an Express controller/route. Here I’m adding just bare bones in the form of Node js script (call it polling.js).

First, install and save several npm packages:

npm install async request query-string moment ip default-user-agent --save

request: for calling third party API

async: we’re interested in the whilst method it contains

query-string: a cool library to convert json object to query parameters and vice versa

moment: not actually required but using to pass a session id in date format

ip: some APIs require ip

default-user-agent: some APIs ask for user agent.

Then require the packages and add some constants:

let request = require('request');
const async = require('async');
const queryString = require('query-string');
const ip = require("ip");
const moment = require("moment");
const ua = require('default-user-agent');
const baseUrl = "https://api.example.com/"
const apiKey = "xxxx-xxxx-xxxx-xxxx"
const pageSize = 10

We declare request with ‘let’ to reassign itself after defaults are added like baseUrl and headers (so as not to add them repeatedly):

request = request.defaults({
    baseUrl,
    headers: {
        'accept': 'application/json',
        'User-Agent': ua()
    }
})

Now add some more variables including queryParams object which contains fields such as sessionID, onlyIfComplete, pageSize, apiKey, clientIp etc. required by some APIs:

let whileLoop = true, // to switch on and off async whilst
    whileTries = 40, // how many tries before giving up further polling?
    pageIndex = 0, //starting with 0. For future pagination this should increase by 1 per page
    queryParams = {
        sessionID: `my-app${moment().format('YYYY-MM-DD-h')}`, //add if required. Using moment to give it a date format and an hourly uniqueness
        onlyIfComplete: true, // if true response status code is 202 till complete result is available
        pageSize, //results to include in response
        apiKey, // almost always required
        clientIp: ip.address() // required by some providers
    }

Moving on, include async whilst block. It takes three argument functions. First returns true or false, based on which whilst continues or stops. Second is meant to run the actual stuff of the while loop and pass error or result once done to the argument function usually named callback. And third is the final callback function that is called when the whilst loop ends, either with error or result.

console.log("*** API Polling Started ***")

async.whilst(
    function () { return whileLoop; },
    function (callback) {
        whileProcess(callback)
    },
    function (err, result) {
        if (err) {
            console.log(err);
        }
        else {
            console.log("*** API Polling Ended ***")
            // process 'result' further
        }
    }
);

To keep the code more streamlined, I’ve taken out the code of whilst into a function whileProcess, which is passed the whilstCallback function in argument


function whileProcess(whilstCallback) {
    request({
        method: 'GET',
        uri: `endpoint?${queryString.stringify(queryParams)}`
    }, function (error, response, body) {
        if (error) {
            return whilstCallback(error)
        }
        else if (response.statusCode < 200 || response.statusCode > 299) {
            return whilstCallback({ status: response.statusCode, message: body })
        }
        else if (response.statusCode === 202) {
            whileTries -= 1;
            if (whileTries === 0) {
                return whilstCallback({ status: 504, message: "Tries limit reached" }); //Gateway Timeout
            }
            else {
                console.log("trying again, attempts left: ", whileTries);
                return whilstCallback(null); //try again. Can also be wrapped in setTimeout to delay a bit before attempting again
            }
        }
        else { // status 200
            whileLoop = false; //discontinue while loop by switching it off instead of passing error in argument
            return whilstCallback(null, body)
        }
    });
}

Here’s what happens in whileProcess:

A GET call is made to the endpoint of the API with query parameters (remember baseUrl and headers were already added as defaults at the top). queryString.stringify converts json object to proper query parameter format
Whilst loop is immediately cut short by sending an error object in first argument to the callback function if status code is not 2XX or error is received
If status code is 202*, decrement *whileTries* and invoke whilstCallback with null so that the loop should continue**. If tries limit has reached, stop the whilst. 504 can be used to inform of gateway timeout in latter case
Finally we have else which means everything went fine and we got the result. Assign false to whileLoop to stop the whilst and return body as second argument

Different providers use different strategies and returning 202 status code for incompleteness is one of them. Some respond with 200 along with another field that has ‘COMPLETE’ or ‘INCOMPLETE’ value

** Though while loop alone takes considerable time between tries, but whilstCallback can also be wrapped in setTimeout to put further delay between them

Complete Code

polling.js

let request = require('request');
const async = require('async');
const queryString = require('query-string');
const ip = require("ip");
const moment = require("moment");
const ua = require('default-user-agent');
const baseUrl = "https://api.example.com/"
const apiKey = "xxxx-xxxx-xxxx-xxxx"
const pageSize = 10

request = request.defaults({
    baseUrl,
    headers: {
        'accept': 'application/json',
        'User-Agent': ua()
    }
})

let whileLoop = true, 
    whileTries = 40, 
    pageIndex = 0, 
    queryParams = {
        sessionID: `my-app${moment().format('YYYY-MM-DD-h')}`, 
        onlyIfComplete: true,
        pageSize,
        apiKey,
        clientIp: ip.address()
    }

console.log("*** API Polling Started ***")

async.whilst(
    function () { return whileLoop; },
    function (callback) {
        whileProcess(callback)
    },
    function (err, result) {
        if (err) {
            console.log(err);
        }
        else {
            console.log("*** API Polling Ended ***")
            // process 'result' further
        }
    }
);

function whileProcess(whilstCallback) {
    request({
        method: 'GET',
        uri: `endpoint?${queryString.stringify(queryParams)}`
    }, function (error, response, body) {
        if (error) {
            return whilstCallback(error)
        }
        else if (response.statusCode < 200 || response.statusCode > 299) {
            return whilstCallback({ status: response.statusCode, message: body })
        }
        else if (response.statusCode === 202) {
            whileTries -= 1;
            if (whileTries === 0) {
                return whilstCallback({ status: 504, message: "Tries limit reached" }); //Gateway Timeout
            }
            else {
                console.log("trying again, attempts left: ", whileTries);
                return whilstCallback(null); 
            }
        }
        else { // status 200
            whileLoop = false; 
            return whilstCallback(null, body)
        }
    });
}

nodejs expressjs polling

Complete Code

See also