Update your robots.txt file

If you use a site hosting service, such as Wix or Blogger, you might not need to (or be able to) edit your robots.txt file directly. Instead, your provider might expose a search settings page or some other mechanism to tell search engines whether or not to crawl your page.

If you want to hide or unhide one of your pages from search engines, search for instructions about modifying your page visibility in search engines on your hosting service, for example, search for "wix hide page from search engines".

To update the rules in your existing robots.txt file, download a copy of your robots.txt file from your site and make the necessary edits.

Download your robots.txt file

You can download your robots.txt file various ways, for example:

• Navigate to your robots.txt file, for example https://example.com/robots.txt and copy its contents into a new text file on computer. Make sure you follow the guidelines related to the file format when creating the new local file.
• Download an actual copy of your robots.txt file with a tool like cURL. For example:

  curl https://example.com/robots.txt -o robots.txt

• Use the robots.txt Tester in Search Console to download a copy of your robots.txt file.
  1. Click Submit in the bottom-right corner of the robots.txt editor. This action opens up a Submit dialog.
  2. Download your robots.txt code from the robots.txt Tester page by clicking Download in the Submit dialog.

Edit your robots.txt file

Open the robots.txt file you downloaded from your site in a text editor and make the necessary edits to the rules. Make sure you use the correct syntax and that you save the file with UTF-8 encoding.

Upload your robots.txt file

Upload your new robots.txt file to the root of your domain as a text file named robots.txt. The way you upload a file to your site is highly platform and server dependent. Check out our tips for finding help with uploading a robots.txt file to your site.

If you do not have permission to upload files to the root of your domain, contact your domain manager to make changes.

For example, if your site home page resides under subdomain.example.com/site/example/, you likely cannot update the robots.txt file at subdomain.example.com/robots.txt. In this case, contact the owner of example.com/ to make any necessary changes to the robots.txt file.

Refresh Google's robots.txt cache

During the automatic crawling process, Google's crawlers notice changes you made to your robots.txt file and update the cached version every 24 hours. If you need to update the cache faster, use the Submit function of the robots.txt Tester.

1. Click View uploaded version to see that your live robots.txt is the version that you want Google to crawl.
2. Click Submit to notify Google that changes have been made to your robots.txt file and request that Google crawl it.
3. Check that your newest version was successfully crawled by Google by refreshing the page in your browser to update the tool's editor and see your live robots.txt code. After you refresh the page, you can also click the dropdown to view the timestamp of when Google first saw the latest version of your robots.txt file.