- No custom domain: Always hidden from search engines. This default behavior cannot be changed.
- Preview environment with a custom domain: Hidden by default. You can make it visible.
- Production environment with a custom domain (new project): When you add a custom domain, the site becomes visible to search engines by default. You can change this at any time.
- Production environment with a custom domain (existing project): Retains its current setting when a custom domain is added. You can change this at any time.
If multiple environments have custom domains, ensure that at most one is visible to avoid duplicate content and SEO dilution.
- In the Console
- Using the CLI
Prerequisite: The environment must have a custom domain.
- Select the project you want to update.
- From the Environment menu, select the environment.
- Click Settings.
- Select or clear the Hide from search engines checkbox.
Upsun can’t guarantee that indexers follow the instructions.
To restrict access entirely, set up HTTP access control.
How it works
When Hide from search engines is enabled, Upsun adds the following header to responses, which instructs search engines to not index and to not traverse your site:Alternative method: Use a robots.txt file
You can also send instructions to search engine indexers using arobots.txt file.
Your app can serve this as a static file from its disk or as a dynamic response from its passthru.
Control either with the location section of your app configuration.
If your robots.txt file instructs search engines to ignore a page, that instruction takes effect regardless of whether Upsun is configured to send the X-Robots-Tag header.