Python concurrent requests let you send multiple HTTP requests at the same time. This can help you scrape hundreds, thousands, or even millions of pages per day, depending on your plan.
In this tutorial we will use ThreadPoolExecutor and the ScrapingBee Python SDK to run Python requests in parallel and save the returned screenshots.
Prerequisites
Before you start, make sure you have:
- A ScrapingBee API key
- The ScrapingBee Python SDK
- Active Python Virtual Environment
Run Python Requests in Parallel
We will use the code below to run concurrent Python requests:
import concurrent.futures
import time
from pathlib import Path
from scrapingbee import ScrapingBeeClient
# Add your ScrapingBee API key.
client = ScrapingBeeClient(api_key="YOUR_API_KEY")
# Set the maximum number of retry attempts.
MAX_RETRIES = 5
# Set the number of requests that can run at the same time.
MAX_THREADS = 4
# Create a folder for the screenshots.
OUTPUT_DIR = Path("screenshots")
OUTPUT_DIR.mkdir(exist_ok=True)
# Add the pages you want to scrape.
URLS = [
"https://www.scrapingbee.com/blog/",
"https://www.scrapingbee.com/documentation/",
]
def scrape(url: str) -> None:
"""Capture and save a screenshot of one URL."""
for attempt in range(1, MAX_RETRIES + 1):
try:
# Send the request and return a screenshot.
response = client.get(
url,
params={
"screenshot": True,
},
)
except Exception as error:
print(f"{url}: request error on attempt {attempt}: {error}")
else:
if response.ok:
# Create a unique filename from the current timestamp.
filename = OUTPUT_DIR / f"screenshot-{time.time_ns()}.png"
# Save the returned screenshot.
filename.write_bytes(response.content)
print(f"{url}: saved as {filename}")
return
print(
f"{url}: HTTP {response.status_code} "
f"on attempt {attempt}"
)
# Wait before retrying the failed request.
if attempt < MAX_RETRIES:
delay = 2 ** (attempt - 1)
print(f"{url}: retrying in {delay} seconds")
time.sleep(delay)
print(f"{url}: failed after {MAX_RETRIES} attempts")
# Create a pool of worker threads.
with concurrent.futures.ThreadPoolExecutor(
max_workers=MAX_THREADS
) as executor:
# Run the scrape function for every URL.
executor.map(scrape, URLS)
When we run our code, we can see that there are two concurrent HTTP requests sent. The script creates a screenshots folder and saves one PNG screenshot for each successful request, as shown below:
How Python Concurrent API Calls Work
ThreadPoolExecutor creates a pool of worker threads. Each worker is tasked with running the scrape() function for each URL.
In the code, there is also this line, which controls the number of Python requests that can run in parallel:
MAX_THREADS = 4
This simply means that the script can process a maximum of 4 URLs at one time. So if you want to run more URLs, you can change this value according to the concurrent request limit available on your ScrapingBee plan.
Handle Failed Concurrent Requests with Exponential Backoff
Each URL can be attempted up to five times as shown below:
MAX_RETRIES = 5
When a request fails, the script waits before trying again:
delay = 2 ** (attempt - 1)
time.sleep(delay)
This section of our code executes a delay of 1, 2, 4, and 8 seconds between retry attempts.
Scale Your Python Parallel Requests: Adding More URLs
You can add more pages to the URL list in this section of our code:
URLS = [
"https://www.scrapingbee.com/blog/",
"https://www.scrapingbee.com/documentation/",
"https://www.scrapingbee.com/tutorials/",
]
Bottom line
Set MAX_THREADS within your ScrapingBee plan's concurrency limit. Add more URLs when you are ready to process a larger batch.