How the crawler renders JavaScript
Headless Chromium, what we wait for, hydration handling, and why some SPAs still need SSR.
Modern websites are mostly JavaScript. Our crawler renders every page with a headless Chromium instance so React, Vue, Next.js, Nuxt, Svelte, Angular and Astro all work without configuration. This page documents how rendering works and the edge cases that still need attention.
The rendering pipeline
- We fetch the page with a real Chromium browser (same rendering engine Googlebot uses).
- We wait for the load event.
- We wait an additional 2 seconds for late-binding JS to settle.
- We capture the rendered HTML (innerHTML of <html>).
- We extract links, headings, meta tags and structured data from the rendered HTML.
Both the rendered HTML and the raw response are available in the report – useful for comparing what Googlebot sees vs what users see.
Wait strategy
Default wait is load + 2s. Sites with slow third-party scripts can configure longer waits in Settings → Crawl:
| load | Default. Waits for the standard load event. |
| networkidle0 | Waits until 0 network requests are in flight for 500ms. Best for heavily-deferred SPAs. |
| networkidle2 | Waits until ≤2 network requests are in flight for 500ms. Tolerates analytics beacons. |
| domcontentloaded | Fastest. Skips most JS execution. Useful only for SSR-only pages. |
Per-audit override is also available in the New audit → Advanced panel.
Resources we block
To keep crawls fast, we block these resource types by default: video, audio, font, websocket, eventsource. We do load: html, css, js, xhr, fetch, image (low-res). Toggle in Settings → Crawl → Resource types.
JavaScript timeout
Per-page rendering timeout is 20 seconds. If a page hasn't fired load + 2s settle by then, we capture whatever's rendered so far and mark the page with a "rendering timeout" warning. Common causes:
- A third-party script (chat widget, video player) that never resolves.
- A WebSocket connection blocking the load event.
- A huge JS bundle on a slow server.
- Infinite redirects or 5xx loops.
Why some SPAs still need SSR
We render the page, but we capture it once. Content that loads only after a user interaction (clicking a tab, opening a modal) won't be in our captured HTML. Same problem Googlebot has. SSR or SSG (Next.js, Nuxt, Astro, etc.) avoids this entirely.
User agent
Default UA: `SEOAuditBot/1.0 (+https://www.semoptimiser.com)`. Override in Settings → Crawl → User agent. Some sites serve different markup to Googlebot vs other bots – to test as Googlebot, set UA to `Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)`.
Comparing rendered vs raw HTML
Every audit report has a "Rendering diff" tab per URL that highlights elements present in the rendered HTML but absent from the raw response. Heavily-different pages depend hard on client-side rendering; that's your signal to consider SSR for SEO-critical content.
One platform. Five fewer subscriptions.
Audit, index and rank –
without a fifth subscription.
Join 1,200+ agencies and in-house teams using SEMOptimiser to replace Semrush, Ahrefs, GA4 add-ons and rank trackers – with one workflow that actually ships fixes.