Make concurrent requests in Node.js

Scraping 100 URLs one at a time might take several minutes, but processing URLs concurrently can reduce total scraping time when network responses are the main bottleneck.

The ScrapingBee API supports multiple concurrent operations. However, your ScrapingBee plan determines how many requests your Node.js script can run concurrently, and exceeding that number returns a 429 error response. You can verify your current limit in your ScrapingBee dashboard, and the ScrapingBee concurrency guidelines explain how it works in more detail.

Node.js offers multiple ways to handle concurrency, depending on whether your workload is I/O-bound or CPU-bound. Web scraping is I/O-bound: your app spends most of its time waiting for external resources, and Node's event loop keeps the main thread free to handle other requests while it waits.

You can take advantage of this architecture using the built-in Promise.all() method to run requests concurrently. This approach is easy to set up and uses fewer resources since you don't need to spin up independent operating system (OS) processes. Remember to cap the number of requests running at once, because calling Promise.all() on the entire URL array will fire all requests immediately.

To implement this, install the ScrapingBee client first:

npm install scrapingbee

Then structure your scraping logic around bounded batches using Promise.all(), as shown in the code snippet below:

const scrapingbee = require('scrapingbee');
const fs = require('fs');
const crypto = require('crypto');

const client = new scrapingbee.ScrapingBeeClient('YOUR-API-KEY');

// Set this to match or stay below your account's concurrency limit
const CONCURRENCY_LIMIT = 5;

async function scrapeUrl(url) {
  try {
    const response = await client.htmlApi({ url });
    const path = 'spb' + crypto.randomBytes(4).readUInt32LE(0) + '.html';
    await fs.promises.writeFile(path, response.data);
    console.log(`Done: ${url} -> ${path}`);
  } catch (e) {
    console.log(`A problem occurred scraping ${url}: ${e.response?.data ?? e.message}`);
  }
}

async function scrapeInBatches(urls, limit) {
  for (let i = 0; i < urls.length; i += limit) {
    const batch = urls.slice(i, i + limit);
    await Promise.all(batch.map(scrapeUrl));
  }
}

const urls = [
  "https://scrapingbee.com/blog/",
  "https://scrapingbee.com/documentation/",
];

scrapeInBatches(urls, CONCURRENCY_LIMIT);

This script divides the URL array into chunks based on the defined concurrency limit. Within these chunks, the Promise.all() method runs every request concurrently. The next chunk doesn't run until the current one finishes, keeping things within your account's concurrency limit.

You can scrape as many pages as your project requires by extending the URL array, since scrapeInBatches() handles any array length automatically. One trade-off: a slow URL holds up its entire batch. If that's a problem, a concurrency control library that starts a new request as soon as a slot frees up can make things more efficient.

The approach above suits scraping because the work is almost entirely waiting. If your app also does CPU-intensive work, such as parsing large HTML responses, consider worker_threads or multiple processes. Whichever processing model you choose, keep the number of simultaneous ScrapingBee requests within your account's concurrency limit.