We found this the way you find most of these: something in a report looked wrong, and the explanation was worse than the report.
We were reading a backlink profile — the list of other websites linking to yours — and among the inbound links were our own deployment domains. The .pages.dev addresses that our host generates automatically for every site it builds were listed as external sites linking to our production websites.
They were not external. They were us. And for a backlink tool to see them at all, a crawler had to have found them, crawled them, and decided they were a website worth recording.
Every page existed twice
If your site is on Cloudflare Pages, Netlify, Vercel or something similar, it almost certainly has two addresses.
There is the one you bought and put on your business cards. And there is the one the platform generated — something like your-project.pages.dev — which serves the complete, identical site.
Not a cut-down preview. Not a password-protected staging copy. The whole website, every page, live, public, and crawlable.
So every blog post, every service page and every suburb landing page on your site exists at two addresses. Google has to decide which one is the real one, and it does not always decide the way you would like.
The canonical tag helped, and was not enough
Here is the part that had kept this from being a disaster, and the part that is worth understanding properly.
Our pages already emitted a canonical tag pointing at the real domain. Even when served from the .pages.dev address, each page declared “the authoritative version of me lives at cosmoswebtech.com.au”. That is correct implementation, and it is why the damage was bounded rather than severe.
But a canonical tag is a hint, not a directive. Google’s own documentation is explicit about this. It is a strong signal that the search engine will usually respect, and it is not an instruction it is obliged to obey.
More importantly, a canonical does nothing about crawling. The duplicate still gets fetched. Your crawl budget, which is a real and finite thing, still gets spent on a second copy of a site Google already has. And a crawler that is willing to record your own deployment host as a backlink is a crawler that has been treating it as a site in its own right.
The right mental model: a canonical tells Google which copy to rank. It does not tell Google to stop looking.
What we changed
One rule, applied at the edge before any page is served.
If the incoming request is for a hostname ending in .pages.dev, the response gets an X-Robots-Tag: noindex, nofollow header attached. The page still renders — the deployment URL stays useful for previewing a build before it goes live — but it now carries an explicit instruction not to index it and not to follow its links.
Unlike a canonical, X-Robots-Tag is a directive. It is the same instruction as a noindex meta tag, delivered in the HTTP headers rather than the HTML, which means it applies to everything the domain serves rather than only to pages you remembered to tag.
Thirteen lines, applied to three sites the same afternoon.
Check yours in two minutes
Find your deployment URL. In Cloudflare Pages, Netlify or Vercel, it is on the project’s overview page. It usually looks like project-name.pages.dev or project-name.netlify.app.
Open it. If your complete website loads, it is public.
Search for it. Put site:your-project.pages.dev into Google. Any results mean it has been indexed, not merely crawled.
Check the header. A free HTTP header checker will show you whether x-robots-tag is present on that hostname. On ours it now reads noindex, nofollow. If yours shows nothing, the duplicate is fully eligible for indexing.
Check your backlink report if you have one. Your own hostnames appearing as inbound links is the symptom that made us look. It is an unusually clear one.
If it is already indexed
Adding the header stops the problem getting worse. It does not clear what is already there, and the two need different actions.
Leave the noindex in place and wait. Google has to re-crawl a page to see the new instruction. For a small site that is usually days to a few weeks. The pages then drop out on their own.
Do not block it in robots.txt. This is the mistake that makes the situation permanent, and it is counter-intuitive enough to be worth spelling out. A Disallow rule stops the crawler fetching the page — which means it never sees the noindex instruction, which means the page stays in the index indefinitely, now with no way to tell Google to remove it. Blocking and de-indexing are opposite instructions. You need the crawler to visit in order to be told to leave.
Use Search Console’s removal tool if you need it gone quickly. That requires verifying the deployment hostname as a separate property, since Search Console treats it as a different site. Removals through that tool last about six months, which is more than enough time for the noindex to do the permanent work.
For most small business sites, the header plus patience is sufficient. The removal tool is worth the setup only if you can see the duplicate ranking for something you care about.
Why this is worth ten minutes of someone’s time
This is not the most damaging technical problem a small business website can have. The canonical tag was doing real work, and we found no evidence that a .pages.dev page had outranked its production twin.
It is worth fixing because of what it represents. A second complete copy of your site, publicly crawlable, that nobody decided to publish and nobody was watching. It arrived with the hosting platform, it has been there since the day the site launched, and it will not appear in any content audit, any page speed report or any ranking tool — because none of those are looking at a hostname you do not know you have.
The fix is small. Finding out whether you need it is smaller.
If you would like someone to run the technical side of this properly — the hostnames, the status codes, the canonicals and the things that only show up when you look at your site the way a crawler does — that is part of how we approach SEO in Parramatta and across the Hills District.
Ashish Ganda is the founder of Cosmos Web Tech, which builds and maintains websites for small businesses across Western Sydney and the Hills District.
