How to Write a robots.txt File for a Small Website
Someone on a forum tells you your site needs a robots.txt file, so you paste in a block of rules you found, upload it, and move on.

Someone on a forum tells you your site needs a robots.txt file, so you paste in a block of rules you found, upload it, and move on. Three weeks later a third of your pages fall out of Google and the Pages report in Search Console fills up with "Blocked by robots.txt". That is how most beginners meet this file: not by writing one, but by breaking something with one they never read.
A robots.txt file is four or five lines of plain text sitting at the root of your domain. It tells crawlers which paths they are allowed to fetch. It is not a security feature, it does not remove a page from search results, and on a 40 page brochure site it barely earns its keep. Knowing precisely what it can and cannot do saves you from the two mistakes that cost people real traffic.
Where the file lives and what it can say
The file has to sit at the root of the domain and nowhere else. If your site is at yoursite.tld, the only address a crawler checks is yoursite.tld/robots.txt. A copy in /blog/ or /assets/ is ignored. The name is lowercase, and the server has to return it as plain text with a 200 status.
Four directives cover almost everything a small site needs:
User-agentnames the crawler the following rules apply to. An asterisk means every crawler.Disallowgives a path prefix the crawler should not fetch.Allowcarves an exception out of a broader Disallow.Sitemappoints at your XML sitemap, written as a full address.
There is also Crawl-delay, which Google ignores outright. Bing and Yandex read it. Unless your shared host is falling over under crawler load, leave it out.
A workable file for a small WordPress site looks like this, and that is the whole thing:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Sitemap: https://yoursite.tld/sitemap.xml
Six lines. Googlebot reads the first 500 kibibytes of the file and stops, which no small site will ever reach. If the file returns a 404, crawlers treat that as permission to fetch everything, which for most new sites is the right outcome anyway.
The one job it does genuinely well
Robots.txt earns its place when your site generates URLs nobody should ever see. Internal search result pages are the classic case. A visitor searches for "pricing" on your site and that produces /?s=pricing. Google finds a link to it, crawls it, then crawls /?s=pricing+2024 and forty other variants, and now a crawler that was willing to fetch maybe 200 URLs a day on a young site is spending that budget on pages that are a list of three excerpts.
I looked at a small florist site last spring that had a colour filter on its product listing. Each combination made its own URL. Search Console reported roughly 11,400 crawled URLs against 88 pages the owner had actually written. Two Disallow lines for the filter parameters, and within about five weeks the crawl stats report showed Googlebot back on the real pages. Nothing else changed.
The other honest uses are small: a staging subfolder, a directory of client PDFs you would rather not have indexed, a print stylesheet route that duplicates every article. If you cannot name the junk URLs, you do not need the file yet.
Why blocking a page will not hide it
Here is the advice I would like to see disappear from forums: "if you do not want a page in Google, disallow it in robots.txt." It does not work, and it quietly makes the problem harder to fix.
Disallow stops a crawler fetching the page. It says nothing about whether the URL can appear in search results. If another site links to that URL, or your own sitemap lists it, Google can index the address without ever reading the content. You get a result with the bare URL and no description, and Search Console labels it "Indexed, though blocked by robots.txt". The page you tried to hide is now in search results looking broken.
There is a worse version of this. If you put a noindex meta tag on a page and also disallow that page, Google never fetches the page, so it never sees the noindex tag, so the page can stay in the index indefinitely. The two tools work against each other. To keep a page out of search results, leave it crawlable and let the noindex tag do its job. Once the page has dropped out, you can disallow it if you want to save the crawl.
And if the content is actually private, neither of these is the answer. Robots.txt is a public file that anyone can read, so a Disallow line is a signpost to the directory you were trying to keep quiet. Put a password on it.
Writing yours in ten minutes
- Find your junk URLs first. Open the Pages report in Search Console, export the list, and sort by URL pattern. Look for query strings, pagination past page three, and anything from a plugin.
- Write the file in a plain text editor. Notepad, TextEdit in plain text mode, or VS Code. Not Word, which adds formatting characters that break the first line.
- Start with
User-agent: *, then add one Disallow per line. Each path is a prefix match, so/wp-admin/covers everything inside that folder. - Add the Sitemap line last, with the full address including https. If you have not made one yet, our guide to creating an XML sitemap for a small website takes about as long as this does.
- Upload it to the web root, the same folder as your index file, then load yoursite.tld/robots.txt in a browser and read what comes back. If you see your hosting panel's directory listing instead, it is in the wrong folder.
The mistakes that cost the most
A staging site is usually built with Disallow: / to keep it out of search. When the site goes live, that line travels with it. One forward slash, and the entire site is off limits. I have seen this twice, both times on sites a developer had handed over, and both times the owner spent a month hunting for a content problem that did not exist. If your traffic dropped off a cliff after a launch, read the file before you read anything else.
The second expensive one is blocking your own assets. Older advice told people to add Disallow: /wp-content/ or Disallow: /assets/ to stop image theft. Google renders pages now, so blocking the folder with your CSS and JavaScript means the page it renders is unstyled text with no layout. Mobile usability checks fail and the page looks thin to a quality rating system that is reading the rendered version.
Two smaller traps worth knowing. Paths are matched as prefixes, not as folders, so Disallow: /blog also blocks /blog-archive and /blogging-tips. Add the trailing slash when you mean the folder. And paths are case sensitive, so a rule for /Admin leaves /admin wide open.
Finally, robots.txt is not a duplicate content tool. If two URLs show the same article, a canonical tag tells Google which one to keep. Blocking one of them means Google cannot read the canonical tag either, and you end up with a split that nobody can resolve. The same logic applies when pages are missing from the index for other reasons, which is covered in fixing crawled currently not indexed pages.
Checking it actually works
Search Console has a robots.txt report under Settings that shows the file Google last fetched, when it fetched it, and any parse errors. Google retired the old interactive tester, so for testing a single URL use URL Inspection and run a live test. It tells you whether fetching is allowed and shows the rendered page.
Keep a copy of the file in your notes with a date, because you will edit it rarely and forget what you changed. After you remove a wrong Disallow, expect one to three weeks before the affected pages come back, sometimes longer on a site with few incoming links. Nothing you do in this file is instant, which is a decent argument for keeping it short enough to understand at a glance. More on the rest of the technical basics sits in our SEO section.
FAQ
Does every website need a robots.txt file?
No. A missing file returns 404 and crawlers read everything, which is usually fine for a small site. Add one when you have specific paths to exclude or want to declare your sitemap.
Will robots.txt stop AI crawlers and scrapers?
Only the ones that choose to obey it. The file is a request, not a lock. Several AI companies publish user-agent names you can disallow, and plenty of scrapers ignore the file completely.
Can I write rules for one search engine only?
Yes. Name it in a User-agent line, such as Bingbot, and put its rules under that. A crawler reads only the most specific group that matches it, so rules under the asterisk do not also apply.
Will a Disallow remove a page already in Google?
No, and it may freeze the page in place. Use a noindex tag on a crawlable page, or the Removals tool in Search Console for something urgent.
More in SEO
SEO
How to Fix Crawled Currently Not Indexed Pages in Google
You open the Pages report in Google Search Console and find 37 URLs sitting under "Crawled, currently not indexed." Google...
SEO
How to Set Up 301 Redirects When You Change a URL
You tidy up your site. A page called /services-page-2 becomes /web-design , an old blog post gets a cleaner address, and a...
SEO
How to Write SEO Friendly URLs for a Small Website
You publish a new page, glance at the address bar and see something like /index.php?p=4823 or...