Building research sites from data files: Astro and fast search
Build a searchable research website from data files with Astro and search that runs in the browser.
Many researchers find themselves needing to share research outputs and data in various ways. Including results, intermediate steps, datasets, interactive views.
Since I hand-wrote my first website in 1998, I have built sites in most of the ways available, and this guide describes the one I have settled on. The traditional route has been a content management system (CMS): WordPress, Drupal or an institutional web platform with a complicated server running a database behind it. CMSs were trying to solve the problem of editing websites without knowing how to make HTML files - especially when more than one person had to edit the website. That's why we went from websites that were basically a folder full of files read by a browser to websites that were essentially database queries.
Another advantage of a CMS was that it separated the content layer from a presentation layer. A hand-written website contained all the formatting and content in one file. That was fine when websites were just content with links and some minimal navigation but modern websites are more like applications. Thus we paste the content into a CMS which saves it into a database and a click on a link will run a database query, it will retrieve all the text components required to build the page we want and the page is generated on the fly from the latest information in our database.
That sounds like an idea state but for all their advantages, CMSs come at a cost. They need a server with server software, a database and all that involves - security, maintenance and also a lot of expense at keeping them running and dealing with large numbers of visitors. That's why there's been a movement for a while among certain groups to use static site generators like Hugo. These will let the authors write text files and then run a script that will build a website out of those text files and put them on a server. They work well with version control tools like git which allow for collaboration and back up. Plus the added benefit is that running a static website like that is dirt cheap but setting one up and maintaining it requires quite a bit of technical knowledge and skill.
But the recurring theme I'm finding being true in the age of AI agents is: everything old is new again! AI agents are great at working for lots of files in folders. And they're great at using version control like git and running and modifying scripts. For them, using a static site generator is easy and fast.
That's why I switched all my sites from CMSs (Drupal and Wordpress) to static sites - which I can run for free. After a bit of research with my agents, I picked Astro as my static site generator. This allows me to keep all the data in files I can easily review and keep in a shared repository. It's easy to modify and my agents are great at using it.
Paradoxically, this overcomes a unexpected problem that crops up for a CMS with a slow database: fast search. My sites now have a search that runs in the visitor's browser, with no server behind it, and shows results as you type lightning fast.
This is for public data: outputs and data you are free to publish. A static site built from data files is the cheapest and most durable way I know, and it is how most of my own sites are made—dominiklukes.net, Semantic Machines, metaphorhacker.net, Czechly.com, Strč prst skrz krk, the People and Computers history site, LDD Navigator and many others. The companion guide, Publishing research results with Cloudflare, covers how to put the finished site online.
This guide is part of a series:
- Publishing research results with Cloudflare
- Building research sites from data files: Astro and fast search (this guide)
- Git and GitHub for sharing data with agents (coming soon)
- Spreadsheets and tabular data with agents (coming soon)
- From a pile of mixed documents to an agent-friendly knowledge repository (coming soon)
What Astro is
Astro is a static site generator. You write the templates for your pages once—what a record page looks like, what a list of records looks like, what the home page shows—and Astro takes your data files and produces a plain HTML page for each item. The result is a folder of HTML files that any host can serve. Nothing runs on a server when someone visits; the pages were made in advance, at build time.
Data first
The most important decision comes before any page is designed: keep the data in files, separate from the presentation. In my sites, content lives as JSON or Markdown files committed to a Git repository. The pages render that content; the content never lives inside the page code.
This has several practical consequences.
- Reuse. The same data files can feed the website and anything else you build from them. Changing the design of the site does not touch the data.
- Collaborators. When the data sits in a GitHub repository, collaborators can work on it directly: correct a record, add a batch of new ones, fix a label. Every change is recorded with who made it and when.
- Agents. AI agents such as Claude Code or Codex work well with files in a repository. A collaborator can ask an agent to check or extend the data, and the change goes through the same record.
- Citation. A file in a repository has a history, so you can point to the version of the data you used.
It may seem that thousands of records need a database. For a site that presents and searches thousands of records, a database makes the site slower, not faster. A pre-built index shipped with the site is the right approach at that scale. Yes, it still needs a server but the server just gives people access to static html files (just like the server I used when I hand-wrote my first site in 1998).
Fast search without a server
Search on my sites needs no server, and the results appear as you type. This is how it works.
- At build time, when Astro writes every page as HTML, it also writes one extra file:
search-index.json. This is a compact list of every item on the site, each with its title, a short description, keywords, the section it belongs to and its link. Short field names keep the file small. - The first time someone opens the search box, the browser downloads that one file.
- As they type, the browser filters the list itself. No request goes to a server for each keystroke, which is why the results are instant.
None of my sites uses a search library for this. Each builds its own index from its own data.
The same index file is also readable by AI agents which makes the site is readable by agents as well as by people and easy to find.
Reading tools that also need no server
This approach strangely works even for seemingly very complicated sites. I have published a number of guides as online highly accessible e-books using an online reader framework. The online reader framework that hosts my guides like Developing Competence, Deliberate Practice and Czech Navigator gives full-text phrase search inside the book, highlights, notes, reading progress and a listen mode that reads the text aloud. All of it is stored in the reader's own browser, so there are no accounts and nothing is sent to a server.
These tools all work from the same source. The search text, the text the listen mode speaks and the anchors that attach a highlight to a passage all come from the same rendered content.
Images, audio, video and other large files
Text and data files go in the Git repository. Large media files—images, audio, video, 3D models, PDFs—are stored separately, and the pages reach them over the web.
The largest example I have worked on is an archive I helped rescue and rebuild. It has about 10,500 records in Markdown and YAML files (sources, media items, annotations, collections, units and papers), with about 2.4 million words. It is a review site, not a public one, so it sits behind the shared password gate described in the Cloudflare guide. It also has about 6,300 media files, images and 3D models, which come to 14.5 GB. This is how they are handled.
- Not in Git. The repository keeps the text, the code and an inventory of the media files with a checksum for each (a fingerprint that shows whether a file has changed).
- Originals and display versions. The originals are kept unchanged, with their checksums. For display, web-sized versions are generated: smaller images and compressed 3D models.
- Storage. The files are stored in R2, Cloudflare's file storage. Pages fetch them through one small Pages Function path on the site. The site stays a static build, and the heavy files live separately.
- Backups. The policy is two independent off-site copies. GitHub is not a backup for media files.
On two of my sites with video, a small Worker serves the video in pieces, so viewers can jump to any point without downloading the whole file.
How far it scales
For search, what matters is how much text the collection holds. These are my sites, measured from the built HTML in November 2026. The word counts include repeated page furniture such as menus and footers, so the real figures are lower.
| Site | Pages | Words (approx.) | Search index |
|---|---|---|---|
| Semantic Machines | 73 | 100,000 | 122 KB |
| metaphorhacker.net | 498 | 300,000 | 56 KB |
| techczech.net | 1,031 | 400,000 | 383 KB |
| dominiklukes.net | 1,128 | 850,000 | 414 KB |
| Conceptual metaphor site | 1,999 | 2 million | None: filterable bibliography and library explorers instead, plus llms.txt and llms-full.txt files for AI agents |
The size of the index depends on what goes into it (titles, descriptions and keywords, or full text), not on the total number of words. metaphorhacker.net has three times the words of Semantic Machines and a smaller index.
These are my estimates of how the approach scales by total words.
- Up to about a million words, or a few thousand documents. One index of titles, descriptions and keywords, a few hundred KB. Search is instant. All my public sites with a search index are in this range.
- About 1–10 million words, or tens of thousands of documents. Indexes of metadata alone reach several MB, and full text in one index is too heavy. Split the index (by type, section or period) and load each part only when it is needed. Another option is a static-search tool that splits its index into small pieces and fetches only the ones a search needs, such as Pagefind; I have not used it. At this size, Cloudflare Pages' limits come into view: 20,000 files per site and 25 MiB per file. One page per document, plus images and scripts, can reach the file count.
- Tens of millions of words and beyond, or question-style search. A database full-text index behind a small Worker, and meaning-based search with embeddings and a vector index. This is where costs and upkeep start (see "When you outgrow it" below).
The archive as a large example
The archive, with about 2.4 million words in 10,500 records, sits in the middle range. Its keyword search runs in the browser over pre-built indexes: a main index of about 10 MB, a separate index of sources of about 2.9 MB and an index for exploring the collection of about 4 MB. That is the size where a single browser index stops feeling light, especially on phones.
We built and trialled the next step. A Pages Function queried a D1 database using SQLite's full-text search (FTS5), together with stored vectors for search by meaning; a small hosted model turned each query into an embedding. It worked. We deferred deploying it and kept the combined search as a local demo. The server step is real work and a running cost, and keyword search in the browser stayed the default.
Large numerical data
Research data is often numbers in tables. I have not built a site with a large numerical dataset, so apart from the first point this is what I would do.
- Small and medium tables. Publish the data files (CSV or JSON) alongside the pages, and compute the summaries and charts at build time. Research Snacks does this, with its data in a public GitHub repository.
- Large tables, with millions of rows. Do not send them to the browser as one file. There are three options. Aggregate at build time, so the pages show summaries. Store the table as columnar files (Parquet, a compact format for large tables) in R2 and query them in the browser with an engine such as DuckDB-WASM, which reads only the parts it needs; I have not used it. Or put the table in a D1 database behind a Worker, within D1's size limits.
- Licensed and personal data. This guide is about public sites. Data under a licence that does not allow publication, and personal data, do not belong on a public site; for private uses, check your institution's data policies first.
A feed of new outputs
Every one of my Astro sites publishes an RSS feed - most people don't realise this but an RSS feed is just a text file like HTML with mark up code for all news (except it uses XML instead of HTML). Astro writes it at build time as a file, rss.xml, from the same data that builds the pages. Some of my sites use Astro's small RSS helper. On dominiklukes.net and Semantic Machines the feed is written by a few lines of our own code, with no plugin and no service.
A feed lets readers, and AI agents, follow new outputs as they appear. The links in my feeds are absolute (full addresses), so feed readers and agents can follow them.
A feed of new results or outputs takes a few lines of code, not a platform.
Moving an old WordPress or Drupal site
metaphorhacker.net was my WordPress blog. It was rebuilt as a static Astro site from a WordPress export, about 500 pages, and now runs on Cloudflare Pages. My other old WordPress blogs, techczech.net (about 1,000 pages) and Hermeneutic Heretic (about 260), are now Astro sites of the same kind. So are two sites that ran on Drupal: dominiklukes.net itself and my old conceptual metaphor site (about 2,000 pages).
An agent did the migration. It took the export, converted the posts to Markdown files, rebuilt the templates and kept the old addresses working. metaphorhacker.net keeps WordPress's year/month/post-name addresses, so old links still work.
An old WordPress or Drupal site can become a fast, free, static archive with search and a feed. It then no longer needs updates and security patches.
When you outgrow it
A browser search over an index matches the words you type: 'cat' finds cat. Researchers often want to ask a question in ordinary language and get the related work back, or a summary of it. That needs the question to be turned into an embedding at the moment it is asked and compared with embeddings of the records. Something has to run at search time, which means a running service and some cost.
Embeddings on their own are blunt. They find records that are roughly related, and a second step that re-ranks those results against the question improves them.
The archive trial used D1's full-text search and stored vectors. For a deployed version on Cloudflare, I would also try Workers AI, which runs embedding models on Cloudflare's network, and Vectorize, which stores embeddings and finds those closest to the question. Any of these costs something at volume and needs keeping running in a way a folder of HTML files does not. The Cloudflare guide explains what Workers, D1 and the other services are.
Getting it built
For most people, using a static site without an AI agent is too cumbersome. But with an agent, it is as easy as talking to a web editor. Except the changes you want are almost instantaneous. An AI agent like Claude Code or Codex can scaffold an Astro site from a folder of data files: it reads the data, proposes the templates, generates the pages and the search index, and builds the site. You describe what each record page should show and what the search should cover, and check the result.
Once you have a built site - or really a folder full of interlinked HTML files. Deploy it to Cloudflare Pages as described in Publishing research results with Cloudflare.
The principle behind this, one source of data from which the pages, the search index and the feed are all generated, is written up for agents in my text-as-graph skill. It is in the philosophy-for-agents folder of my public skills collection, dominiks-agent-skills.
Give this guide to your agent
Copy this and give it to your agent (Claude Code, Codex or another). The agent can read the whole guide and the rest of the series on this site, each page also as a Markdown file written for agents.
Read https://semanticmachines.fyi/guides/astro-and-fast-search (and https://semanticmachines.fyi/llms.txt for the rest of the series), then help me turn my data files into a research site with fast search the way it describes. Start by asking me what I want to share.
What your agent takes from this guide
This is the practice the guide describes, written for the agent. It is here so that you can see what your agent will work from.
# Practice: research sites built from data files, with search in the browser
Purpose: present public research data as a static Astro site with fast
search and no server behind it.
## Data
- Keep the content as JSON or Markdown files in the Git repository.
Templates render the data; never copy data into page code.
- Generate the pages, the search index and the feed from the same data.
See the text-as-graph skill in
https://github.com/techczech/dominiks-agent-skills
(philosophy-for-agents/text-as-graph).
- Keep large media (images, audio, video, 3D models, PDFs) out of Git.
Keep the originals unchanged, generate web-sized versions for display,
store the files in file storage such as Cloudflare R2, and commit an
inventory with a checksum for each file.
## Pages
- Write the templates once: a page for each record, list pages for
browsing, a home page. Build to a folder of static HTML.
## Search
- At build time, write search-index.json: one entry per item with title,
short description, keywords, section and link. Use short field names.
- Download the index the first time the search box is opened and filter it
in the browser as the user types. No search library is needed.
- Report the size of the index after each build.
- Up to about a million words, one index of titles, descriptions and
keywords is enough. Beyond that, split the index by type, section or
period and load each part only when it is needed. Watch Cloudflare Pages'
limits of 20,000 files per site and 25 MiB per file.
- Search by meaning (embeddings, a database full-text index behind a
Worker) needs something running at search time. Tell the user that this
brings a running service and some cost before building it.
## Numerical data
- Publish small and medium tables as CSV or JSON files alongside the pages
and compute summaries and charts at build time.
- Do not send a table with millions of rows to the browser as one file.
## Feed
- Write rss.xml at build time from the same data, with absolute links.
## Moving an old WordPress or Drupal site
- Convert the posts from the export to Markdown files, rebuild the
templates and keep the old addresses working.