[[["易于理解","easyToUnderstand","thumb-up"],["解决了我的问题","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["没有我需要的信息","missingTheInformationINeed","thumb-down"],["太复杂/步骤太多","tooComplicatedTooManySteps","thumb-down"],["内容需要更新","outOfDate","thumb-down"],["翻译问题","translationIssue","thumb-down"],["示例/代码问题","samplesCodeIssue","thumb-down"],["其他","otherDown","thumb-down"]],[],[],[],null,["# Robots Refresher: robots.txt — a flexible way to control how machines explore your website\n\nFriday, March 7, 2025\n\n\nA long-standing tool for website owners, robots.txt has been in active use for over 30 years and\nis broadly supported by crawler operators (such as tools for site owners, services, and search\nengines). In this edition of the [robots refresher series](/search/blog/2025/02/intro-robots-refresher),\nwe'll take a closer look at robots.txt as a flexible way to tell robots what you want them to do\n(or not do) on your website.\n\nGetting started with robots.txt\n-------------------------------\n\n\nThe way these files work is simple: you make a text file called \"robots.txt\" and then upload it to\nyour website---and if you're using a content management system (CMS), it's likely even easier. You\ncan leave your robots.txt file empty (or not have one at all) if your whole\nsite may be crawled, or you can add rules to manage crawling. For example, to tell all bots (also\nknown as crawlers, robots, spiders) to stay out of your \"add to cart\" page, you could write this\nin your robots.txt file: \n\n```\nuser-agent: *\ndisallow: /cart\n```\n\nMore specific things you can do with robots.txt\n-----------------------------------------------\n\n\nrobots.txt is the Swiss Army knife of expressing what you want different robots to do or\nnot do on your website: it can be just a few lines, or it can be complex with more elaborate\nrules targeting very specific URL patterns. You can use a robots.txt file for solving technical\nissues (such as [unnecessary paginated pages](/search/docs/specialty/ecommerce/pagination-and-incremental-page-loading#avoid-indexing-variations)),\nor for editorial or personal reasons (such as just not wanting certain things crawled). For\nexample, you could:\n\n|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|\n| **Inform multiple bots (but not all) about the same rule** This group tells both `examplebot` and `otherbot` to stay away from the `/search` path. | ``` user-agent: examplebot user-agent: otherbot disallow: /search ``` |\n| **Tell one bot to avoid paths that contain a specific piece of text** For example, you could tell `documentsbot` not to crawl any file that contains \".pdf\" in its name. | ``` user-agent: documentsbot disallow: *.pdf ``` |\n| **Tell a bot it may crawl your blog, but not the drafts** | ``` user-agent: documentsbot allow: /blog/ disallow: /blog/drafts/ ``` |\n| **Block a crawler from part of your website, while allowing other crawlers to access your site** This robots.txt file disallows the mentioned `aicorp-trainer-bot` from accessing anything other than the home page, while allowing other crawlers (such as search engines) to access the site. | ``` user-agent: * allow: / user-agent: aicorp-trainer-bot disallow: / allow: /$ ``` |\n| **Leave a comment for your future self** You can start a line with `#` to remind yourself about why you put a certain rule there. | ``` # I don't want bots in my highschool photos user-agent: * disallow: /photos/highschool/ ``` |\n\n\nFor even more, you can check out our [list of useful robots.txt rules](/search/docs/crawling-indexing/robots/create-robots-txt#useful-robots.txt-rules).\n\nMaking changes to your robots.txt file (practically)\n----------------------------------------------------\n\n\nThe [Robots Exclusion Protocol (REP)](https://datatracker.ietf.org/doc/rfc9309/) works\nby putting together rules (\"allow\" or \"disallow\")\nand specifying which robots these rules apply to. You don't need to learn programming or fiddle\nwith tools; you can just put these rules in a text file and upload it to your website.\n\n\nFor most websites, it's even simpler than that! If you're using a CMS, it usually has\nsomething already built in to help you change your robots.txt file. For example, some CMSes let\nyou customize your robots.txt file using checkboxes or with a simple form, and many have plugins\nthat help you set up and write rules for your robots.txt file. To check what's possible within\nyour CMS, you can do a search for the name of your CMS +\"edit robots.txt file\".\n\n\nOnce you've got things set up, you can also test to make sure your file is set up how you\nintended. There are many testing tools built by the web community to help with this, such as\n[TametheBot's robots.txt testing tool](https://tamethebots.com/tools/robotstxt-checker)\nand this [robots.txt parser](https://www.realrobotstxt.com/) that\nare using the [open-source robots.txt parser library](https://github.com/google/robotstxt).\n\n\nIf you have any questions about robots.txt, you can find us on [LinkedIn](https://www.linkedin.com/showcase/googlesearchcentral/)\nor chat with like-minded experts in [our community forums](https://support.google.com/webmasters/community).\n\nPosted by [Martin Splitt](/search/blog/authors/martin-splitt)\nand [John Mueller](/search/blog/authors/john-mueller),\nSearch relations team\n\n*** ** * ** ***\n\nCheck out the rest of the Robots Refresher series:\n--------------------------------------------------"]]
