Learn/How AI agents read websites
How AI agents read websites
Last updated 2026-10-15
AI agents read websites by fetching the raw HTML over HTTP — without executing JavaScript — and converting it into plain text, a list of links, and a map of form fields. They rely on the same signals a screen reader uses: semantic landmarks, heading hierarchy, accessible names, and metadata like JSON-LD. If your content only appears after JavaScript runs, most agents see an empty page.
The fetch pipeline
An agent requests your page, receives HTML, strips scripts and styles, and keeps the text. Then it extracts links with their anchor text, images with their alt text, and form fields with their labels. Anything that required JavaScript to render does not exist yet. The agent decides what to do next based entirely on this flattened view.
What agents struggle with
Client-side rendered apps that ship an empty body. Icon-only buttons with no accessible name. Unlabeled form fields. Pages that hide their main content behind a login wall without saying so in text. Infinite scroll with no real links. All of these are invisible or ambiguous in the flattened view.
What helps most
Server-render your primary content. Use real semantic HTML: header, nav, main, footer, one h1, logical heading levels. Give every interactive element an accessible name. Publish llms.txt and JSON-LD so the agent does not have to infer structure. Expose real links instead of click handlers.
FAQ
Do AI agents execute JavaScript?
Some browser-based agents do, but many fetch-and-read agents do not. Designing for the no-JavaScript case guarantees both can use your site.
Is this the same as accessibility?
Almost. The checks that make a site readable by screen readers — labels, names, landmarks, heading structure — are the same ones that make it readable by agents.