some text
some text
This text can be shown in a snippet.
However, this is not in snippet.
- Stuff not in snippet
- More stuff not in snippet
```
Google typically renders pages in order to index them, however rendering is not guaranteed.
Because of this, extraction of `data-nosnippet` may happen both
before and after rendering. To avoid uncertainty from rendering, do not add or remove the
`data-nosnippet` attribute of existing nodes through JavaScript.
When adding DOM elements through JavaScript, include the
`data-nosnippet` attribute as necessary when initially adding the
element to the page's DOM. If custom elements are used, wrap or render them with
`div`, `span`, or
`section` elements if you need to use
`data-nosnippet`.
## Using structured data
Robots `meta` tags govern the amount of content that Google extracts automatically from web
pages for display as search results. But many publishers also use schema.org structured data
to make specific information available for
[search presentation](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data). Robots `meta` tag
limitations don't affect the use of that structured data, with the exception of
`article.description` and the
`description` values for structured data specified for other
creative works. To specify the maximum length of a preview based on these
`description` values, use the
`max-snippet` rule. For example,
`recipe` structured data on a page is eligible for inclusion in the
recipe carousel, even if the text preview would otherwise be limited. You can limit the length
of a text preview with `max-snippet`, but that robots `meta` tag
doesn't apply when the information is provided using structured data for rich results.
To manage the use of structured data for your web pages, modify the structured data types and
values themselves, adding or removing information in order to provide only the data you want
to make available. Also note that structured data remains usable for search results when
declared within a `data-nosnippet` element.
## Practical implementation of `X-Robots-Tag`
You can add the `X-Robots-Tag` to a site's HTTP responses through
the configuration files of your site's web server software. For example, on Apache-based web
servers you can use .htaccess and httpd.conf files. The benefit of using an
`X-Robots-Tag` with HTTP responses is that you can specify crawling
rules that are applied globally across a site. The support of regular expressions allows
a high level of flexibility.
For example, to add a `noindex, nofollow`
`X-Robots-Tag` to the HTTP response for all `.PDF` files across an
entire site, add the following snippet to the site's root `.htaccess` file or `httpd.conf` file on
Apache, or the site's `.conf` file on NGINX.
### Apache
```
Header set X-Robots-Tag "noindex, nofollow"
```
### NGINX
```
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
```
You can use the `X-Robots-Tag` for non-HTML files like image files
where the usage of robots `meta` tags in HTML is not possible. Here's an example of adding a
`noindex` `X-Robots-Tag` rule for
images files (`.png`, `.jpeg`, `.jpg`, `.gif`) across an entire site:
### Apache
```
Header set X-Robots-Tag "noindex"
```
### NGINX
```
location ~* \.(png|jpe?g|gif)$ {
add_header X-Robots-Tag "noindex";
}
```
You can also set the `X-Robots-Tag` headers for individual static files:
### Apache
```
# the htaccess file must be placed in the directory of the matched file.
Header set X-Robots-Tag "noindex, nofollow"
```
### NGINX
```
location = /secrets/unicorn.pdf {
add_header X-Robots-Tag "noindex, nofollow";
}
```
## Combining robots.txt rules with indexing and serving rules
robots `meta` tags and `X-Robots-Tag` HTTP headers are discovered when
a URL is crawled. If a page is disallowed from crawling through the robots.txt file, then any
information about indexing or serving rules will not be found and will therefore be
ignored. If indexing or serving rules must be followed, the URLs containing those
rules cannot be disallowed from crawling.