Prepare website content for a RAG knowledge base.

Keep the source with the text.

Retrieval-augmented generation, or RAG, retrieves relevant passages before asking a model to answer. This guide prepares those passages. You will produce a local JSONL file, not a finished chatbot or vector index.

You need Node.js 24, a UTF-8 Markdown file and the public URL it came from. If you already have those, no API call or key is needed for the chunking step. To collect a page first, use the Node.js guide or the Python guide.

For a small site, the documentation collection guide creates a manifest pairing each Markdown file with its source_url. Preserve those pairs when preparing your dataset.

Convert a local file.

Download prepare_rag.mjs. In the same folder as your Markdown, run this command with the actual source URL:

node prepare_rag.mjs page.md https://example.com chunks.jsonl

The output contains one JSON object per line. The script refuses to overwrite an existing file. You can also download the small sample document to try the transformation without making a paid API request:

node prepare_rag.mjs rag-sample.md https://www.crawlvolt.com/examples/rag-sample.md sample-chunks.jsonl

Track the document and its version.

This baseline uses 1,200 Unicode characters per chunk with 150 characters of overlap. It normalizes line endings and removes outer whitespace. It keeps a document ID derived from the URL and a separate hash derived from the content.

import { createHash } from 'node:crypto';
import { readFile, writeFile } from 'node:fs/promises';
import { pathToFileURL } from 'node:url';

const sha256 = (value) => createHash('sha256').update(value).digest('hex');

export function prepareChunks(markdown, source, size = 1200, overlap = 150) {
  const url = new URL(source);
  if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password) {
    throw new Error('Provide the public HTTP(S) source URL.');
  }
  url.hash = '';
  if (!Number.isInteger(size) || !Number.isInteger(overlap) || size < 1 || overlap < 0 || overlap >= size) {
    throw new Error('Chunk size must be positive and overlap must be smaller than size.');
  }
  const normalized = markdown.replace(/\r\n?/g, '\n').trim();
  if (!normalized) throw new Error('The Markdown file is empty.');
  const characters = Array.from(normalized);
  const documentId = sha256(url.href);
  const contentHash = sha256(normalized);
  const title = normalized.match(/^#\s+(.+)$/m)?.[1] ?? url.pathname;
  const chunks = [];
  for (let start = 0; start < characters.length; start += size - overlap) {
    const text = characters.slice(start, start + size).join('');
    chunks.push({
      id: sha256(`${documentId}:${contentHash}:${size}:${overlap}:${start}`),
      document_id: documentId,
      content_sha256: contentHash,
      source_url: url.href,
      title,
      chunk_index: chunks.length,
      start_character: start,
      end_character: Math.min(start + size, characters.length),
      text,
    });
    if (start + size >= characters.length) break;
  }
  return chunks;
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
  try {
    const [file, source, output = 'chunks.jsonl'] = process.argv.slice(2);
    if (!file || !source) throw new Error('Usage: node prepare_rag.mjs page.md https://source.example/page chunks.jsonl');
    const chunks = prepareChunks(await readFile(file, 'utf8'), source);
    await writeFile(output, chunks.map((chunk) => JSON.stringify(chunk)).join('\n') + '\n', { flag: 'wx' });
    console.log(`Saved ${chunks.length} chunks to ${output}. Review the text before indexing.`);
  } catch (error) {
    console.error(error.message);
    process.exitCode = 1;
  }
}

The same URL, normalized content and chunk settings produce the same chunk IDs on reruns. Changed content produces a new version. The URL's fragment is removed; query parameters remain part of its identity. Use a consistent canonical URL when you collect pages.

Check usefulness before generating embeddings.

• Read the first and last chunks. Confirm the page title and the content you expected are present.
• Pick three questions the page should answer. Locate the passage that supports each answer.
• Check chunk boundaries around tables, code blocks and long explanations. Character-based splitting can cut these in half.
• Remove navigation, repeated footers and error pages before indexing. Empty content is rejected; misleading content still needs review.

Characters are not model tokens. Measure your embedding provider's token limits and adjust splitting for your documents. A Markdown-aware or tokenizer-aware splitter can be a better next step for code-heavy documentation; this small script makes its limits explicit.

Add retrieval as the next tested step.

Send each chunk's text to your chosen embedding model, then store the vector with source_url, document_id, content_sha256 and id. Those metadata fields let your application cite sources and replace an older document version.

When content changes, stage the new version, check retrieval, then remove obsolete chunks for the same document ID. Appending new versions forever leaves stale passages in search. Test retrieval with your three questions before connecting answer generation.

For a recurring source, continue with detecting website content changes in n8n. That guide produces a change signal you can connect to your own storage and indexing steps.