Guides / Can AI Crawlers Read React Websites?

Can AI Crawlers Read React Websites?

How initial HTML, server rendering, static generation, hydration, and client-side rendering affect what automated systems can retrieve from React sites.

AI crawlers can read a React website when the requested response contains usable content or when the specific crawler can execute the JavaScript needed to create it. React itself is not the problem. The important distinction is whether the page is server-rendered or statically generated, or whether its primary content appears only after client-side JavaScript runs.

The three common rendering patterns

Static generation

The application generates HTML at build time. The server returns the completed page content immediately, and React may hydrate it later for interaction.

This usually gives crawlers, link-preview tools, accessibility tools, and users a strong initial document.

Server-side rendering

The server renders React components to HTML for each request or through a cached server response. React documents server APIs that generate initial HTML before client hydration.

The crawler receives meaningful text and structure even when it never executes the JavaScript bundle.

Client-side rendering

The server returns a minimal shell, often including a root element and JavaScript bundles. The browser executes the application, requests data, and then inserts the meaningful page content.

A browser-capable crawler may render this successfully. A simpler fetcher may only see the shell.

Why initial HTML matters

Google documents a JavaScript rendering stage, but it also recommends server-side rendering, static rendering, or hydration over dynamic rendering workarounds. Google additionally notes that not every bot can run JavaScript.

AI-search crawlers and user-triggered fetchers do not all publish identical rendering capabilities. A site should therefore avoid assuming that behaviour documented for Googlebot applies to every other automated client.

For high-value public pages, important information should ideally be available in the first HTML response:

organisation and product name;
page title and primary heading;
product category and audience;
key explanatory text;
navigation and internal links;
canonical and robots metadata;
visible evidence and source links.

Interactive enhancements can still hydrate on the client.

A quick diagnostic

Fetch the page without a browser renderer:

curl -L --compressed https://example.com/ > page.html

Then inspect page.html.

Questions to ask:

Is the page’s main heading present?
Is the core product or article text present?
Are important links present as ordinary anchor elements?
Do the title, canonical URL, and robots directives exist?
Does the response contain the real content, or mainly script tags and an empty root element?

Do not send repeated or high-volume test requests to systems you do not own.

Compare browser text with response text

A useful manual test is to compare:

the text visible after the page finishes loading in a browser; and
the meaningful text in the raw HTTP response.

A large difference is not automatically a failure. It is a signal that the page depends on browser execution and should be tested against the actual clients that matter.

OpenForBots labels this as a heuristic when it cannot verify a provider’s rendering behaviour.

React implementation approaches

Use framework-supported static or server rendering

Modern React frameworks can produce initial HTML through static generation, server-side rendering, streaming, or server components. Choose the framework-native approach rather than maintaining a bot-only rendering path.

Keep critical content outside client-only effects

Content loaded only in useEffect appears after the first client render. That may be appropriate for personalised or secondary data, but it is a fragile place for the only description of the page’s subject.

Hydrate interactive components instead of the whole meaning

The document can include complete explanatory HTML while React adds calculators, filters, dialogs, and form behaviour after hydration.

Return meaningful HTTP status codes

A client-rendered error message on top of an HTTP 200 response can mislead crawlers. Return correct status codes for missing, redirected, unavailable, or restricted resources.

Make links crawlable

Use real <a href="..."> links for navigation destinations. Click handlers without discoverable URLs may not provide a reliable relationship between pages.

Keep metadata server-visible

The title, canonical URL, robots directives, social metadata, and relevant structured data should be correct in the response for each route. Do not rely on a late client-side update when the server can provide the right values.

Approaches to avoid

Bot-specific pages with materially different content

Serving one subject to crawlers and another to users can become cloaking. Google describes dynamic rendering as a workaround rather than a recommended long-term architecture.

Assuming Googlebot behaviour is universal

Google may render JavaScript, but that does not prove that every AI crawler, user-triggered fetcher, social preview, or lightweight client will do the same.

Adding prerendering only for the homepage

Important product, documentation, guide, pricing, and comparison pages can have the same client-only dependency.

Treating a low text count as proof of unreadability

A short raw response is a useful heuristic. It is not proof that a specific provider cannot render or understand the page.

What OpenForBots checks

The Release 1 audit is intentionally conservative:

fetches the public root response without a headless browser;
records status, final URL, content type, and response limits;
extracts meaningful text and document structure from initial HTML;
checks whether basic business information is identifiable;
compares visible metadata and structural signals;
labels heavy JavaScript dependency as a heuristic;
states that browser-rendered interpretation requires a manual or provider-specific test.

The normal scan path does not execute page JavaScript.

Verification after a rendering change

Build and deploy the route using the intended production adapter.
Fetch the public URL without browser execution.
Confirm the key content appears in the response.
Confirm canonical and robots metadata are correct.
Test the page with JavaScript disabled for a human-readable baseline where practical.
Use the search engine’s supported inspection tools for that engine.
Monitor server logs and Search Console without assuming every user-agent header is authentic.
Re-run the deterministic initial-HTML audit.

What server rendering does not guarantee

Server-rendered HTML does not guarantee:

crawling or indexing;
citation by an AI system;
higher rankings;
accurate interpretation of vague copy;
correct handling of blocked resources;
a good user experience;
strong evidence for product claims.

It removes one avoidable retrieval dependency. The content still needs to be useful, clear, connected, and trustworthy.

Next step

Preview the initial-HTML checks to see how OpenForBots separates an observed response from a heuristic about browser-rendered content.