How to scrape all text from a website for LLM training

If you’re building AI applications, you need LLM-ready text pulled from web pages. Whether you’re feeding a RAG retrieval pipeline, fine-tuning, or continuing pretraining an existing model, the first step is web scraping for LLMs.

In this article, we’ll show you how to scrape a website for LLM training data by automating text collection across an entire site. We’ll build a custom Python LLM training data scraper that extracts, parses, and saves website text in a clean format.

TL;DR

To scrape website text for LLM training, discover URLs from the site’s sitemap, fetch each page, and extract the main content with a boilerplate remover like trafilatura, instead of a raw BeautifulSoup get_text() call, which pulls in nav menus and footers along with the content.

Save the output as markdown, then deduplicate before use in RAG, fine-tuning, or pretraining.

Key takeaways

• Extraction quality is the variable that changes your output most: A fast scraper feeding a raw get_text() call produces a worse dataset than a slower one paired with a proper boilerplate remover like trafilatura.
• Markdown is the format most LLM pipelines want: Plain text loses heading structure, and raw HTML wastes tokens on markup that carries no meaning for the model.
• RAG and fine-tuning need far less data than pretraining: And most scraping projects for an LLM are doing one of these two, not training a model from scratch.
• robots.txt and TDM opt-out signals now carry legal weight in the EU: This is no longer just etiquette. The AI Act ties them directly to compliance obligations for models placed on the EU market.
• Deduplicate before you index or train: Near-duplicate pages are the most common defect in a scraped corpus, and the easiest to miss if you only check for exact matches.

What LLM-ready text actually means: RAG, fine-tuning, or continued pretraining

“LLM-ready text” doesn’t mean the same thing for every AI project:

• If you’re building a RAG system, you need small, well-chunked, current documents.
• If you’re fine-tuning, you need a moderate, consistently formatted dataset.
• If you’re continuing pretraining on a domain corpus, you need a very large volume of clean, deduplicated text.

The right scraping approach depends on which path you’re on:

A quick way to think about it:

• RAG treats scraped text as a lookup index, not training material. The model never “learns” it, so freshness and precise retrieval boundaries matter more than volume.
• Fine-tuning bakes patterns into the model’s weights. This means consistency across examples matters more than raw size.
• Continued pretraining is closer to how base models are originally trained, but on a narrower domain. The priorities shift toward corpus-level hygiene: removing duplicates, boilerplate, and low-quality pages before they dilute the signal.

Preparing to write a custom script to load training data

So, where does the training data actually come from? One reliable option is scraping content from websites relevant to your task. This is exactly what we’ll do next, using a custom Python script.

Prerequisites

Before we dive into the code, let’s quickly cover the prerequisites. I expect that you have:

• Basic knowledge of Python and a general understanding of Python web scraping.
• Already installed Python 3 on your machine.
• Installed your favorite code editor or IDE.

That’s basically it!

Setting up your environment

When you’re ready to proceed, create a new folder for your Python project. Inside, add two files:

• find_links.py — we’ll use this to find all website URLs.
• extract_data.py — this will contain the main script to download website data.

Next, initialize the project with uv and create a virtual environment:

uv init

This command might create a main.py file, but we won’t be using it.

Now install the required dependencies:

uv add beautifulsoup4 requests lxml scrapingbee pandas trafilatura

Let me briefly cover the tools we’re going to use:

• requests — library to make HTTP requests easily.
• beautifulsoup4 — enables us to perform data parsing of HTML content.
• lxml — fast XML/HTML parser used by BeautifulSoup.
• pandas — data analysis and manipulation tool.
• scrapingbee — ScrapingBee Python client used to route requests through proxies and avoid getting blocked, with extra features like JS rendering and screenshots.
• trafilatura — boilerplate remover that pulls just the main content out of a page, stripping nav bars, ads, and footers so what’s left is clean, LLM-ready text.

Alright, at this point we’re ready to go!

Respecting robots.txt and scraping policies

This section is general information, not legal advice. If scraping for LLM training is central to your business, have this reviewed by a lawyer before you rely on it.

Before scraping any website, check its robots.txt file and consider that:

• robots.txt is a voluntary standard, not a lock: Formalized as RFC 9309, robots.txt tells crawlers what a site owner would prefer they access, but it carries no independent legal force. However, it still matters in practice. Courts have treated compliance (or disregard) with the robots.txt file as evidence of good or bad faith. Ignoring it is not by itself a DMCA circumvention violation, but respecting it costs you little and works in your favor if a dispute ever arises.
• AI crawlers now split training from search: When robots.txt directives were simpler, one crawler line covered a provider. That’s no longer true. Most major AI providers now run separate agents for training versus live search/answer citations. This split matters operationally because a site can block its content from being used as LLM training data while still allowing it to be indexed and cited in AI-generated answers. If you’re scraping a website for training data, check which specific agent names are disallowed. For example, a blanket “AI bot” assumption can miss this distinction, and so can a script that only checks for a generic Disallow: /.
• The EU AI Act is the real 2026 development: Under the EU AI Act, providers of general-purpose AI models must publish a sufficiently detailed summary of the content used to train their models and maintain a copyright policy that respects machine-readable rights reservations under the EU’s text-and-data-mining exception. In practice, this means honoring opt-outs signaled via robots.txt and similar mechanisms. These obligations began applying in August 2025, with enforcement provisions taking effect in August 2026.
• llms.txt is not a substitute for robots.txt: These two files do different jobs. robots.txt governs access. llms.txt is a proposed convention for guidance. It points AI systems toward the content a site considers most relevant or useful, typically for retrieval and citation contexts. Adding an llms.txt file doesn’t grant scraping permission, and respecting robots.txt doesn’t mean you can ignore llms.txt where it exists. Treat them as complementary signals, not interchangeable ones.
• Screen for PII before it enters your dataset: Avoid scraping Personally Identifiable Information (PII) like names, emails, or other users’ private data, and filter it out if it slips through during collection. If any of the data you’re collecting involves EU residents’ personal data, note that GDPR obligations apply in addition to, not instead of, AI Act requirements.

Beyond these points, the fundamentals still apply: review the target site’s Terms of Service and confirm your use case is permitted, and be mindful of server load. In other words, send requests at a reasonable rate and avoid overwhelming the site you’re scraping.

Finding all website pages

Now, before we can download any data, there’s another important task to solve: we need to understand what pages the target website actually contains. This can be a challenge on its own, so I have prepared an article showing a few solutions for how to find all URLs on a domain.

Today, we’re going to use a solid but simple approach: scanning the website’s sitemap and extracting links from it. If a site has no sitemap, or an incomplete one, fall back to crawling internal links from the homepage, or check robots.txt for a Sitemap: line pointing to a sitemap index you might otherwise miss.

Let’s open the find_links.py file and import the necessary dependencies:

import csv
from pathlib import Path
import requests
from bs4 import BeautifulSoup as Soup

We’ll save the URLs into a CSV file containing the actual link, last modification date, and priority, in case you need that extra data later:

from typing import Final

# Constants for the attributes to be extracted from the sitemap.
ATTRS: Final[tuple[str, ...]] = ("loc", "lastmod", "priority")

Now, let’s code the main function:

def parse_sitemap(
    url: str,
    csv_filename: str = "urls.csv",
    visited: set[str] | None = None,
) -> bool:
    """Parse the sitemap at the given URL and append the data to a CSV file."""
    if not url:
        print("No sitemap URL provided.")
        return False

    if visited is None:
        visited = set()

    url = url.strip()

    # Avoid processing the same sitemap more than once.
    if url in visited:
        return True

    visited.add(url)

    try:
        response = requests.get(url, timeout=30)
        response.raise_for_status()
    except requests.RequestException as e:
        print(f"Failed to fetch sitemap {url}: {e}")
        return False

    soup = Soup(response.content, "xml")

    success = True

    # Recursively parse nested sitemaps.
    for sitemap in soup.find_all("sitemap"):
        loc = sitemap.find("loc")

        if loc and loc.text:
            success = parse_sitemap(
                loc.text.strip(),
                csv_filename,
                visited,
            ) and success

    # Find all URL entries in the sitemap.
    urls = soup.find_all("url")

    rows: list[list[str]] = []
    for url_entry in urls:
        row = []

        for attr in ATTRS:
            found_attr = url_entry.find(attr)
            row.append(found_attr.text.strip() if found_attr else "n/a")

        rows.append(row)

    if not rows:
        return success

    # Save the CSV file in the same directory as the script.
    csv_path = Path(__file__).resolve().parent / csv_filename
    file_exists = csv_path.exists()

    try:
        with csv_path.open("a", newline="", encoding="utf-8") as csvfile:
            writer = csv.writer(csvfile)

            if not file_exists:
                writer.writerow(ATTRS)

            writer.writerows(rows)
    except OSError as e:
        print(f"Failed to write sitemap data to {csv_path}: {e}")
        return False

    return success


if __name__ == "__main__":
    parse_sitemap("https://example.com/sitemap.xml")

This code is pretty straightforward:

1. We fetch the given sitemap URL.
2. Parse the response with BeautifulSoup.
3. Look for any nested sitemaps and process them recursively.
4. Find every url tag in the sitemap.
5. Extract the required attributes from each found URL.
6. Save the data to a CSV file.

That’s it! If you have issues processing a sitemap because your request is being blocked, you can use the ScrapingBee client, as I’ll show in the section below. Some websites protect themselves from automated web scraping and web scrapers, so routing requests through a proxy API can help avoid unnecessary blocks.

To run the script with uv, use:

uv run python find_links.py

Fetching website data from every page

At this point, you should have a urls.csv file with all website links ready for data extraction. Open the extract_data.py file, and let’s get down to business.

Simple script to load website data

Let’s code a first version of the script. This is a naive baseline we’ll measure against and improve in the next section. Start by importing the necessary libraries:

from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import pandas as pd
import requests
from bs4 import BeautifulSoup

Next, define a few constants that we’ll use throughout the script:

INPUT_CSV = Path("urls.csv")
OUTPUT_FILE = Path("extracted_texts.txt")
URL_COLUMN = "loc"
REQUEST_TIMEOUT = 30

Add a helper function to read URLs from the CSV file:

def load_urls(csv_path: Path) -> list[str]:
    """Load URLs from the sitemap CSV file."""
    try:
        df = pd.read_csv(csv_path)
    except FileNotFoundError:
        print(f"Input file not found: {csv_path}")
        return []
    except pd.errors.EmptyDataError:
        print(f"Input file is empty: {csv_path}")
        return []
    except pd.errors.ParserError as e:
        print(f"Failed to parse CSV file {csv_path}: {e}")
        return []

    if URL_COLUMN not in df.columns:
        print(f"Missing required column: {URL_COLUMN}")
        return []

    return [
        str(url).strip()
        for url in df[URL_COLUMN].dropna()
        if str(url).strip() and str(url).strip().lower() != "n/a"
    ]

Then, a function to fetch and extract text from a single page:

def fetch_page_text(url: str) -> str | None:
    """Fetch a page and extract readable text from its HTML."""
    try:
        response = requests.get(url, timeout=REQUEST_TIMEOUT)
        response.raise_for_status()
    except requests.RequestException as e:
        print(f"Error fetching {url}: {e}")
        return None

    content_type = response.headers.get("Content-Type", "")

    if content_type and "text/html" not in content_type:
        print(f"Skipping non-HTML page: {url}")
        return None

    soup = BeautifulSoup(response.text, "html.parser")

    # Remove elements that usually do not contain useful training text.
    for tag in soup(["script", "style", "noscript"]):
        tag.decompose()

    return soup.get_text(separator="\n", strip=True)

And finally, a function to save the scraped data to a UTF-8 encoded text file:

def save_texts(texts: list[str], output_path: Path) -> bool:
    """Save extracted texts to a UTF-8 encoded text file."""
    try:
        with output_path.open("w", encoding="utf-8") as file:
            for text in texts:
                file.write(text + "\n\n")
    except OSError as e:
        print(f"Failed to write output file {output_path}: {e}")
        return False

    return True

Wire it together with a main() function that loads the URLs, fetches each page, and saves the results:

def main() -> None:
    urls = load_urls(INPUT_CSV)

    if not urls:
        print("No URLs found. Nothing to extract.")
        return

    all_texts: list[str] = []

    for url in urls:
        text = fetch_page_text(url)

        if text:
            all_texts.append(text)

    if save_texts(all_texts, OUTPUT_FILE):
        print("Text extraction completed successfully!")


if __name__ == "__main__":
    main()

This baseline runs and produces output, but it has two problems we’ll fix as we go:

1. Requests may get blocked, since many sites protect themselves from automated crawlers.
2. soup.get_text() pulls text from the entire page. This matters particularly for getting LLM-ready data.

Extracting clean text: why soup.get_text() is not enough

Running soup.get_text() on the full HTML document extracts almost everything on the page. For LLM training data, that’s a problem of “garbage in, garbage out”. It’s worth being precise about how much of a problem this is, rather than just asserting it.

The Zyte article-extraction benchmark scores article-body extraction quality across common libraries on a shared dataset. The relevant results are:

• BeautifulSoup scores F1 0.665 with precision 0.499. This means that nearly half of what a plain get_text() call returns is boilerplate text.
• Trafilatura scores F1 0.958 with 0.938 precision.

That’s the gap between “usable training data” and “data you’ll spend as much time cleaning as you spent collecting”.

These numbers are measured mostly on article-style pages, and extraction quality drops on other page types. This is why the WCXB benchmark — 2,008 annotated pages spanning seven page types — shows trafilatura falling from roughly 0.92 on articles down to 0.55–0.58 on forums and listing pages. On the practical side, this means that:

• If you’re scraping docs sites or blogs, expect numbers close to the article benchmark.
• If you’re scraping forums, product listings, or other non-article layouts, spot-check the output yourself before trusting it at scale. Don’t assume the article-page numbers transfer.

If you’re wondering why trafilatura gets these results, it’s because of how it was built. It originated from corpus-linguistics work at the Berlin-Brandenburg Academy of Sciences, and it’s used in data pipelines at organizations including Hugging Face and the Allen Institute for AI.

Here’s the corrected fetch_page_text(), swapping the soup.get_text() call for trafilatura.extract() with markdown output:

from pathlib import Path
import pandas as pd
import requests
import trafilatura

def fetch_page_text(url: str) -> str | None:
    """Fetch a page and extract clean main-content text as markdown."""
    try:
        response = requests.get(url, timeout=REQUEST_TIMEOUT)
        response.raise_for_status()
    except requests.RequestException as e:
        print(f"Error fetching {url}: {e}")
        return None

    content_type = response.headers.get("Content-Type", "")

    if content_type and "text/html" not in content_type:
        print(f"Skipping non-HTML page: {url}")
        return None

    extracted = trafilatura.extract(response.text, output_format="markdown")

    if not extracted:
        print(f"No main content found: {url}")
        return None

    return extracted

The rest of the script remains the same. Only the extraction step inside fetch_page_text() changes. This isn’t a rewrite of the whole pipeline, just a swap of the noisiest step for a tool built to do that job well.

Benchmark: get_text() versus trafilatura versus markdown output

The Zyte and WCXB numbers above are useful for the general case, but they’re measured on someone else’s dataset. Here we present a first-hand benchmark on real pages, run specifically for this article.

The code ran three extraction paths against 20 pages: 10 from ScrapingBee’s documentation site and 10 from the blog. Below is the list of the target pages stored in the benchmark_urls.txt file:

# --- Documentation pages (10) ---
https://www.scrapingbee.com/documentation/
https://www.scrapingbee.com/documentation/proxy-mode/
https://www.scrapingbee.com/documentation/data-extraction/
https://www.scrapingbee.com/documentation/js-scenario/
https://www.scrapingbee.com/documentation/cli/
https://www.scrapingbee.com/documentation/remote-mcp/
https://www.scrapingbee.com/documentation/n8n/
https://www.scrapingbee.com/documentation/make/
https://www.scrapingbee.com/documentation/zapier/
https://www.scrapingbee.com/documentation/langchain/
 
# --- Blog posts (10) ---
https://www.scrapingbee.com/blog/classic-proxy-geolocation/
https://www.scrapingbee.com/blog/best-sneaker-proxies/
https://www.scrapingbee.com/blog/how-to-scrape-tcgplayer/
https://www.scrapingbee.com/blog/ruflo-ai-agent-orchestration/
https://www.scrapingbee.com/blog/web-scraping-with-cloudproxy/
https://www.scrapingbee.com/blog/mcp-servers-web-scraping/
https://www.scrapingbee.com/blog/agent-skills-ai-coding-agents/
https://www.scrapingbee.com/blog/how-to-build-a-job-aggregator/
https://www.scrapingbee.com/blog/how-to-build-python-flight-scraper/
https://www.scrapingbee.com/blog/how-to-scrape-uber-eats-data/

First, register on ScrapingBee for free. You’ll receive 1000 free credits to test the API. After logging in, copy your API token from the dashboard. Store the token in an environment variable rather than hardcoding it:

export SCRAPINGBEE_API_KEY="YOUR_TOKEN"

On Windows PowerShell, use:

$env:SCRAPINGBEE_API_KEY="YOUR_TOKEN"

Also, you need to add the following library to the ones already added previously:

uv add lxml_html_clean

The code for this benchmark is the following:

from __future__ import annotations
import csv
import os
import sys
from dataclasses import dataclass, asdict
from pathlib import Path
import requests
import trafilatura
from bs4 import BeautifulSoup
from scrapingbee import ScrapingBeeClient

URLS_FILE = Path("benchmark_urls.txt")
OUTPUT_CSV = Path("benchmark_results.csv")
REQUEST_TIMEOUT = 30
CHARS_PER_TOKEN = 4  # Rough estimate; swap for tiktoken for exact counts
SCRAPINGBEE_API_KEY = os.getenv("SCRAPINGBEE_API_KEY")

@dataclass
class PageResult:
    url: str
    method: str
    chars: int
    est_tokens: int
    excerpt: str

def load_urls(path: Path) -> list[str]:
    if not path.exists():
        print(f"URL file not found: {path}")
        return []
    urls = [line.strip() for line in path.read_text(encoding="utf-8").splitlines()]
    return [u for u in urls if u and not u.startswith("#")]

def fetch_html(url: str) -> str | None:
    try:
        response = requests.get(url, timeout=REQUEST_TIMEOUT)
        response.raise_for_status()
    except requests.RequestException as e:
        print(f"Failed to fetch {url}: {e}")
        return None
    return response.text

def extract_get_text(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "noscript"]):
        tag.decompose()
    return soup.get_text(separator="\n", strip=True)

def extract_trafilatura(html: str) -> str:
    extracted = trafilatura.extract(html, output_format="markdown")
    return extracted or ""

def extract_scrapingbee_markdown(client: ScrapingBeeClient, url: str) -> str:
    """Method C: fetch through ScrapingBee and return markdown_relevant output."""

    extract_rules = {
        "content": {
            "selector": "body",
            "output": "markdown_relevant",
        }
    }

    try:
        response = client.html_api(
            url,
            params={
                "extract_rules": extract_rules,
                "render_js": False,
            }
        )
    
        if not response.ok:
            print(f"ScrapingBee returned {response.status_code} for {url}: {response.text[:500]}")
            return ""
        
        # The response from the client library supports .json()
        data = response.json()
        return data.get("content", "")

    except Exception as e:
        print(f"ScrapingBee client request failed for {url}: {e}")
        return ""

def char_and_token_count(text: str) -> tuple[int, int]:
    chars = len(text)
    est_tokens = round(chars / CHARS_PER_TOKEN)
    return chars, est_tokens

def make_excerpt(text: str, length: int = 200) -> str:
    flat = " ".join(text.split())
    return flat[:length] + ("..." if len(flat) > length else "")

def run_benchmark(urls: list[str]) -> list[PageResult]:
    results: list[PageResult] = []

    # Initialize ScrapingBee client once if key is present
    sb_client = None
    if SCRAPINGBEE_API_KEY and SCRAPINGBEE_API_KEY != "api-key":
        sb_client = ScrapingBeeClient(api_key=SCRAPINGBEE_API_KEY)

    for i, url in enumerate(urls, start=1):
        print(f"[{i}/{len(urls)}] {url}")
        html = fetch_html(url)
        if not html:
            continue

        # Method A
        text_a = extract_get_text(html)
        chars_a, tokens_a = char_and_token_count(text_a)
        results.append(PageResult(url, "get_text", chars_a, tokens_a, make_excerpt(text_a)))

        # Method B
        text_b = extract_trafilatura(html)
        chars_b, tokens_b = char_and_token_count(text_b)
        results.append(PageResult(url, "trafilatura", chars_b, tokens_b, make_excerpt(text_b)))

        # Method C
        if sb_client:
            text_c = extract_scrapingbee_markdown(sb_client, url)
            chars_c, tokens_c = char_and_token_count(text_c)
            results.append(PageResult(url, "scrapingbee_markdown", chars_c, tokens_c, make_excerpt(text_c)))

    return results

def save_csv(results: list[PageResult], path: Path) -> None:
    if not results:
        return
    with path.open("w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(asdict(results[0]).keys()))
        writer.writeheader()
        for r in results:
            writer.writerow(asdict(r))

def print_summary(results: list[PageResult]) -> None:
    methods = sorted(set(r.method for r in results))

    print("\n--- Summary (averages across pages) ---")
    print(f"{'Method':<22}{'Avg chars':<14}{'Avg est. tokens':<18}{'Pages':<8}")

    for method in methods:
        subset = [r for r in results if r.method == method]
        if not subset:
            continue
        avg_chars = round(sum(r.chars for r in subset) / len(subset))
        avg_tokens = round(sum(r.est_tokens for r in subset) / len(subset))
        print(f"{method:<22}{avg_chars:<14}{avg_tokens:<18}{len(subset):<8}")

    if "get_text" in methods and "trafilatura" in methods:
        get_text_subset = [r for r in results if r.method == "get_text"]
        traf_subset = [r for r in results if r.method == "trafilatura"]
    
        if get_text_subset and traf_subset:
            get_text_avg = sum(r.chars for r in get_text_subset) / len(get_text_subset)
            traf_avg = sum(r.chars for r in traf_subset) / len(traf_subset)
        
            if get_text_avg > 0:
                reduction = 100 * (1 - traf_avg / get_text_avg)
                print(f"\ntrafilatura output is ~{reduction:.0f}% smaller than get_text() output on average.")

    if not SCRAPINGBEE_API_KEY or SCRAPINGBEE_API_KEY == "api-key":
        print("\nNote: SCRAPINGBEE_API_KEY not set correctly, so the ScrapingBee column was skipped.")

def main() -> None:
    urls = load_urls(URLS_FILE)
    if not urls:
        print(f"No URLs loaded from {URLS_FILE}. Add one URL per line and re-run.")
        sys.exit(1)

    results = run_benchmark(urls)
    if not results:
        print("No results collected -- check that the URLs are reachable.")
        sys.exit(1)

    save_csv(results, OUTPUT_CSV)
    print_summary(results)
    print(f"\nPer-page results saved to {OUTPUT_CSV}")

if __name__ == "__main__":
    main()

The resulting summary table is the following:

--- Summary (averages across pages) ---
Method                Avg chars     Avg est. tokens   Pages 
get_text              34504         8626              20    
scrapingbee_markdown  36782         9196              20    
trafilatura           33445         8361              20    

trafilatura output is ~3% smaller than get_text() output on average.

On this example, trafilatura’s output was only about 3% smaller than plain get_text(), for both characters and tokens. That is nowhere near the reduction the Zyte benchmark would predict for article-style pages. So, take this result as an illustration of how extraction results behave on a real, imperfect sample (including the ways such a comparison can go sideways). This is a small real-world comparison, not a general benchmark. For the general case, the Zyte and WCXB benchmarks above remain the better reference.

Choosing an output format: markdown, JSON, or plain text

The output format you save extracted text in should follow directly from which of the three paths you’re on. The general guidelines are:

• Markdown is the default recommendation for most LLM ingestion: It’s the format trafilatura produces above, and for good reason. It keeps heading structure (#, ##) and link context intact. This matters because most RAG chunkers split documents on headings. Markdown also costs far fewer tokens than raw HTML for the same content, since there’s no tag overhead to carry through embedding or context windows.
• JSON or JSONL is what fine-tuning pipelines usually expect: Most fine-tuning frameworks want examples in a structured, per-record format. Typically, one JSON object per line. If you’re heading toward fine-tuning, plan to convert your extracted markdown into JSONL records rather than saving raw text files.
• Plain text is fine for large-scale pretraining corpora: This is where structure is discarded anyway during tokenization and training. This happens because, at that volume, the priority is deduplication and filtering, not preserving heading hierarchy.

In short:

• If you’re building for RAG, save markdown.
• If you’re fine-tuning, save markdown now and convert to JSONL when you assemble training examples.
• If you’re doing continued pretraining, plain text at scale is sufficient, and structure isn’t worth preserving.

Using proxies to avoid getting blocked

As we’ve already discussed in one of the previous articles, scraping without getting blocked can be tricky. Here’s a simple way to use premium proxies without managing them manually:

import os
from scrapingbee import ScrapingBeeClient
import trafilatura

api_key = os.getenv("SCRAPINGBEE_API_KEY")

if not api_key:
    raise RuntimeError("Missing SCRAPINGBEE_API_KEY environment variable.")

client = ScrapingBeeClient(api_key=api_key)

Find the line where you send a regular request:

response = requests.get(url, timeout=REQUEST_TIMEOUT)

Replace it with:

response = client.html_api(
    url,
    params={
        # Use premium proxies for tougher websites.
        "premium_proxy": True,
        "country_code": "gb",
        "render_js": False,
        "device": "desktop",
    },
)

If you’re using the fetch_page_text() function from the previous section, it will look like this:

def fetch_page_text(url: str) -> str | None:
    """Fetch a page through ScrapingBee and extract clean main-content text as markdown."""
    try:
        response = client.html_api(
            url,
            params={
                "premium_proxy": True,
                "country_code": "gb",
                "render_js": False,
                "device": "desktop",
            },
        )
        response.raise_for_status()
    except requests.RequestException as e:
        print(f"Error fetching {url}: {e}")
        return None

    content_type = response.headers.get("Content-Type", "")

    if content_type and "text/html" not in content_type:
        print(f"Skipping non-HTML page: {url}")
        return None

    extracted = trafilatura.extract(response.text, output_format="markdown")

    if not extracted:
        print(f"No main content found: {url}")
        return None

    return extracted

ScrapingBee handles the proxy setup for you, so you can focus on the data instead of fighting IP blocks. Worth noting: the API can also return markdown or clean text directly, via the markdown_relevant extraction output shown in the benchmark above. This removes the separate trafilatura step entirely if you’d rather not run your own extraction pass. To learn more about other features, refer to the ScrapingBee Python client documentation.

Using multiple threads

Visiting one page after another is not very efficient, especially when you have hundreds or thousands of pages to process. Since most of the time is spent waiting for HTTP responses, we can speed things up by using multiple threads:

from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import pandas as pd
import requests
import trafilatura

Next, add a constant to control how many pages we process at the same time:

MAX_WORKERS = 5

Now update the main() function:

def main() -> None:
    urls = load_urls(INPUT_CSV)

    if not urls:
        print("No URLs found. Nothing to extract.")
        return

    all_texts: list[str] = []

    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        futures = {
            executor.submit(fetch_page_text, url): url
            for url in urls
        }

        for future in as_completed(futures):
            url = futures[future]

            try:
                text = future.result()
            except Exception as e:
                print(f"Unexpected error processing {url}: {e}")
                continue

            if text:
                all_texts.append(text)

    if save_texts(all_texts, OUTPUT_FILE):
        print("Text extraction completed successfully!")

Here we set up five workers to process pages concurrently. Each worker calls the existing fetch_page_text() function, so we don’t have to rewrite our scraping logic. Keep MAX_WORKERS reasonable. Setting it too high may overload the target website or trigger rate limits.

Implementing retries for additional robustness

When scraping a website, you may run into temporary server errors, timeouts, or network hiccups. Instead of failing immediately, we can retry the request a few times before giving up.

We’ll use the tenacity library for this. Install it with uv:

uv add tenacity

Then import it in your Python script:

from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential

Now extract the request logic into a separate function and wrap it with a retry decorator:

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10),
    retry=retry_if_exception_type(requests.RequestException),
    reraise=True,
)
def fetch_with_retry(url: str) -> requests.Response:
    """Fetch a URL with retries."""
    response = requests.get(url, timeout=REQUEST_TIMEOUT)
    response.raise_for_status()
    return response

This tells tenacity to retry failed requests up to three times, using exponential backoff between attempts.

Now update fetch_page_text() to use this new helper:

def fetch_page_text(url: str) -> str | None:
    """Fetch a page and extract clean main-content text as markdown."""
    try:
        response = fetch_with_retry(url)
    except requests.RequestException as e:
        print(f"Error fetching {url}: {e}")
        return None

    content_type = response.headers.get("Content-Type", "")

    if content_type and "text/html" not in content_type:
        print(f"Skipping non-HTML page: {url}")
        return None

    extracted = trafilatura.extract(response.text, output_format="markdown")

    if not extracted:
        print(f"No main content found: {url}")
        return None

    return extracted

All other helper functions stay the same. Here’s the updated version of the full script. The example below uses the basic requests version of the script. If you’ve already switched to ScrapingBee, you can apply the same retry pattern to the client.html_api() call:

from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import pandas as pd
import requests
import trafilatura
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential

INPUT_CSV = Path("urls.csv")
OUTPUT_FILE = Path("extracted_texts.txt")
URL_COLUMN = "loc"
REQUEST_TIMEOUT = 30
MAX_WORKERS = 5

def load_urls(csv_path: Path) -> list[str]:
    """Load URLs from the sitemap CSV file."""
    try:
        df = pd.read_csv(csv_path)
    except FileNotFoundError:
        print(f"Input file not found: {csv_path}")
        return []
    except pd.errors.EmptyDataError:
        print(f"Input file is empty: {csv_path}")
        return []
    except pd.errors.ParserError as e:
        print(f"Failed to parse CSV file {csv_path}: {e}")
        return []

    if URL_COLUMN not in df.columns:
        print(f"Missing required column: {URL_COLUMN}")
        return []

    return [
        str(url).strip()
        for url in df[URL_COLUMN].dropna()
        if str(url).strip() and str(url).strip().lower() != "n/a"
    ]

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10),
    retry=retry_if_exception_type(requests.RequestException),
    reraise=True,
)
def fetch_with_retry(url: str) -> requests.Response:
    """Fetch a URL with retries."""
    response = requests.get(url, timeout=REQUEST_TIMEOUT)
    response.raise_for_status()
    return response

def fetch_page_text(url: str) -> str | None:
    """Fetch a page and extract clean main-content text as markdown."""
    try:
        response = fetch_with_retry(url)
    except requests.RequestException as e:
        print(f"Error fetching {url}: {e}")
        return None

    content_type = response.headers.get("Content-Type", "")

    if content_type and "text/html" not in content_type:
        print(f"Skipping non-HTML page: {url}")
        return None

    extracted = trafilatura.extract(response.text, output_format="markdown")

    if not extracted:
        print(f"No main content found: {url}")
        return None

    return extracted

def save_texts(texts: list[str], output_path: Path) -> bool:
    """Save extracted texts to a UTF-8 encoded text file."""
    try:
        with output_path.open("w", encoding="utf-8") as file:
            for text in texts:
                file.write(text + "\n\n")
    except OSError as e:
        print(f"Failed to write output file {output_path}: {e}")
        return False

    return True

def main() -> None:
    urls = load_urls(INPUT_CSV)

    if not urls:
        print("No URLs found. Nothing to extract.")
        return

    all_texts: list[str] = []

    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        futures = {
            executor.submit(fetch_page_text, url): url
            for url in urls
        }

        for future in as_completed(futures):
            url = futures[future]

            try:
                text = future.result()
            except Exception as e:
                print(f"Unexpected error processing {url}: {e}")
                continue

            if text:
                all_texts.append(text)

    if save_texts(all_texts, OUTPUT_FILE):
        print("Text extraction completed successfully!")

if __name__ == "__main__":
    main()

Now the script can process multiple pages concurrently and handle temporary request failures more gracefully.

Cleaning the dataset: deduplication, PII, and chunking

Saving extracted text to a file isn’t the finish line. A raw scrape typically contains duplicate pages, boilerplate that survived extraction, PII you didn’t mean to collect, and no metadata to tell you where any of it came from later. Let’s cover what to do about each, in the order it matters most.

Deduplication is the highest-value cleaning step

Start with exact deduplication. To do so, hash each document’s text (a simple SHA-256 of the normalized content) and drop exact matches. This catches identical pages reachable through multiple URLs, which is more common than it sounds. Here’s a simple script to do so:

import hashlib

def content_hash(text: str) -> str:
    normalized = " ".join(text.split()).lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

Note that exact hashing won’t catch near-duplicates, which are usually the larger share. For those, use a near-duplicate technique like MinHash or SimHash, which compare documents by similarity rather than exact match. The datasketch library implements MinHash cheaply enough to run over a large corpus.

Before using it, add the datasketch :

uv add datasketch

Then, use the following snippet:

from datasketch import MinHash, MinHashLSH

def get_minhash(text: str, num_perm: int = 128) -> MinHash:
    m = MinHash(num_perm=num_perm)
    for word in set(text.split()):
        m.update(word.encode("utf-8"))
    return m

lsh = MinHashLSH(threshold=0.85, num_perm=128)
# lsh.insert(doc_id, get_minhash(text)) for each doc, then
# lsh.query(get_minhash(new_text)) to find near-duplicates before adding it

Note: the 0.85 threshold above assumes n-gram shingles (sequences of consecutive words). This snippet shingles on single words (unigrams) instead, which is a coarser similarity signal (meaning unrelated documents can share many individual words just by topic overlap). If you keep unigram shingling, you may need to retune the threshold against your own corpus (likely higher than 0.85) to avoid over-flagging genuinely distinct pages as near-duplicates.

Boilerplate that survives extraction

Even a good extractor like trafilatura occasionally lets through cookie banners, related-post blocks, or repeated CTAs, especially on page types outside its strong suit. The practical way to catch these is to look for strings that repeat verbatim across many otherwise-different documents. A short line appearing in 40 of 50 scraped pages is very unlikely to be page-specific content. A simple frequency count across your corpus surfaces these candidates for a stopword-style removal list.

Quality filtering

Drop documents that are too short to be useful (a reasonable floor is a few hundred characters, adjusted to your domain), pages that are navigation-only (high link-to-text ratio, low prose content), and pages that are mostly code or markup when that isn’t what you’re collecting. These are cheap, rule-based checks worth running before anything else, since they remove the most obviously unusable rows first.

PII screening

This must happen before the data lands in your store, as covered above. Screen for names, emails, phone numbers, and other user-generated personal data using a pattern-matching pass. For anything higher-stakes, use a dedicated PII-detection library rather than relying on regex alone.

What to store alongside the text

The text itself isn’t enough. Store at minimum the following:

• Source URL — so you can trace any document back to its origin
• Fetch date — so you know how current the content was when collected
• Content hash — the same hash used for deduplication, which doubles as a stable identifier for that exact version of the page

Without these three fields, you can’t refresh a stale document, audit where a piece of training data came from, or respond to a takedown request — all of which come up in practice once a dataset is in active use, not hypothetically.

Chunking for RAG

This is the payoff for keeping markdown output instead of plain text. Split documents on heading boundaries (#, ##, ###) rather than fixed character counts. Fixed-size chunking cuts sentences and sections arbitrarily, which damages retrieval quality by splitting related content across chunks. Heading-based chunking respects the document’s own structure, so each chunk is a coherent unit. This is the exact reason why the earlier decision table recommends markdown for RAG in the first place.

Recrawl cadence

Finally, decide a re-crawl cadence up front. A corpus with no refresh plan rots. Whether that’s weekly, monthly, or tied to a site’s own lastmod values from its sitemap (which find_links.py already collects), pick a cadence before you start relying on the data.

Why train your own model?

There are a few reasons teams reach for custom-trained models instead of using an off-the-shelf one as-is:

• Customization: Adapting a model to specific tasks, industries, or datasets that general-purpose models don’t cover well.
• Privacy: Keeping sensitive data inside your own environment rather than sending it to third-party APIs.
• Performance: A model tuned to a narrow domain can outperform a general-purpose one on that domain’s specific data and queries.

That said, most scraping projects for LLM data should focus on fine-tuning an existing model or building a RAG pipeline rather than training a model from scratch, as discussed above.

How much data is needed?

Data needs scale sharply with what you’re building:

• Continued pretraining: Rough estimates run from small models needing around 1–5 million words (the entire Harry Potter series is a little over 1 million words) up to frontier-scale models trained on billions or even trillions of words. 1 billion words is roughly 900 times the entire Harry Potter series.
• RAG: Needs far less. Often just a few hundred to a few thousand documents, since the model retrieves relevant passages at query time rather than learning the content.
• Fine-tuning: Typically sits in the low thousands to tens of thousands of examples, depending on task complexity. If you’re following the decision table earlier in this article, the data volume you actually need is probably much smaller than the Harry Potter comparisons above might suggest.

When to use a scraping API instead of your own script

Everything in this article works with a self-hosted requests + trafilatura pipeline, and for a lot of projects, that’s the right answer. That approach is free, and it’s genuinely enough for a few thousand pages on cooperative sites that don’t actively fight scrapers.

Reach for a managed web scraping API when one of a few specific things becomes true from the following list:

• You’re hitting blocks: Sites that rate-limit, fingerprint, or actively challenge automated requests will eventually stop your requests.get() calls cold, no matter how polite your crawl rate is. A managed API absorbs that problem with rotating proxies and browser fingerprinting handled for you.
• The pages are JS-rendered: If content only appears after JavaScript runs, a plain HTTP client fetching static HTML will get an empty shell. You need either a headless browser you manage yourself, or a service that renders pages for you.
• You want markdown back without running your own extraction step: As shown earlier, AI-assisted data extraction can return clean, boilerplate-trimmed markdown directly from the API. This allows you to skip the trafilatura step entirely if you don’t want to maintain that part of the pipeline yourself.
• The crawl needs to be reliable enough to schedule: A one-off script you babysit is very different from a recurring job that has to succeed unattended on a cadence, as covered in the freshness point in the data-cleaning section above. Managed infrastructure is built for the second case. A personal script usually isn’t, without real engineering investment.

Beyond ScrapingBee, other tools solving overlapping parts of this problem include:

• Firecrawl: Scraping with LLM-ready markdown output as a core feature.
• Apify: A broader actor/automation marketplace with many pre-built scrapers.
• ScrapeGraphAI: An LLM-driven extraction layer for structured data.

Which one fits depends on your volume, budget, and how much of the pipeline you want to own versus delegate. Our own comparison of the best AI web scrapers goes deeper into the tradeoffs if you want to evaluate them side by side.

If you’d rather not manage proxies, rendering, and extraction yourself, ScrapingBee gives you 1,000 free API credits to test the approach shown in this article without writing the block-avoidance and extraction layers from scratch.

Finally, the source code you’ve seen in this article can be found on GitHub.

How to scrape text from a website for LLM training: FAQ

Why scrape website text for LLM training?

Scraping gives you domain-specific text at scale. This is the raw material for three different jobs: building a retrieval corpus for RAG, fine-tuning an existing model, or assembling a pretraining dataset. If you want a model or retrieval system to understand a specific industry, product, or documentation site, scraping relevant pages is how you collect that material.

What is the best way to get clean text from a URL for an LLM?

Fetch the page, then run it through a boilerplate remover like trafilatura instead of taking everything with a raw get_text() call. On the Zyte extraction benchmark, plain BeautifulSoup scores F1 0.665 versus trafilatura’s 0.958. Request markdown output so heading structure survives for chunking later.

Can I use scraped web text for RAG instead of training a model?

Yes, and for most teams that’s the better option. RAG needs far less data, no GPU training run, and updates as soon as you re-crawl. Keep markdown headings so chunks split on real section boundaries, and store the source URL with every chunk so answers can cite it.

Can I scrape any website for LLM training?

Not always. Check the site’s robots.txt and Terms of Service first. The robots.txt has no independent legal force, but disregarding it can count against you. Under the EU AI Act, models placed on the EU market must respect machine-readable opt-out signals. Public availability doesn’t mean the content is free to reuse for any purpose.

What is an LLM scraper?

The term is used two ways: a scraper that collects data from LLM products like ChatGPT, or a scraper that uses an LLM to extract structured fields from pages. This guide covers a third, more common job: collecting web text to feed into an LLM for RAG, fine-tuning, or pretraining.

How do I handle duplicate pages in a scraped dataset?

Hash each document to catch exact duplicates, then use near-duplicate detection like MinHash for pages sharing large boilerplate blocks. Store a content hash and fetch date with every record so you can re-crawl and diff later. Deduplicate before indexing or training.

How much text do I need to train an LLM?

It depends heavily on your goal. Pretraining ranges from millions to billions of words. Fine-tuning typically needs only low thousands to tens of thousands of examples. RAG needs the least. Often, just a few hundred to a few thousand documents, since the model retrieves at query time instead of learning the content.

Is raw scraped data ready for training?

No. Raw scraped data usually contains navigation menus, ads, duplicate pages, and other noise that survives extraction. Clean, deduplicate, and screen it for PII before use, See the dataset-cleaning section above for exact-hash and near-duplicate methods, quality filters, and the metadata to store alongside the text.

Why use proxies for web scraping?

Websites may block repeated requests from the same IP address. Proxies, or a proxy API like ScrapingBee, help reduce blocks and make large-scale web scraping more reliable.

Should I use multithreading when scraping websites?

Yes, but carefully. Multithreading can speed up data extraction by processing multiple pages at once, but too many workers can overload the target website or trigger rate limits.