Stay organized with collections
Save and categorize content based on your preferences.
Wednesday, December 19, 2007
In 2003, Google introduced a "supplemental index" as a way of showing more documents to users.
Most webmasters will probably snicker about that statement, since supplemental docs were famous
for refreshing less often and showing up in search results less often. But the supplemental
index served an important purpose: it stored unusual documents that we would search in more
depth for harder or more esoteric queries. For a long time, the alternative was to simply not
show those documents at all, but this was always unsatisfying—ideally, we would search all of
the documents all of the time, to give users the experience they expect.
This led to a major effort to rethink the entire supplemental index. We improved the crawl
frequency and decoupled it from which index a document was stored in, and once these
"supplementalization effects" were gone, the "supplemental result" tag itself—which only
served to suggest that otherwise good documents were somehow suspect—
was eliminated a few months ago.
Now we're coming to the next major milestone in the elimination of the artificial difference
between indices: rather than searching some part of our index in more depth for obscure queries,
we're now searching the whole index for every query.
From a user perspective, this means that you'll be seeing more relevant documents and a much
deeper slice of the web, especially for non-English queries. For webmasters, this means that
good-quality pages that were less visible in our index are more likely to come up for queries.
Hidden behind this are some truly amazing technical feats; serving this much larger of an index
doesn't happen easily, and it took several fundamental innovations to make it possible. At this
point it's safe to say that the Google search engine works like nothing else in the world. If
you want to know how it actually works, you'll have to come join Google Engineering; as usual,
it's all triple-hush-hush secrets.*
* Originally, I was going to give the stock Google answer, "If I told you, I'd have to kill you."
However, I've been informed by management that killing people violates our "Don't be evil" policy,
so I'm forced to replace that with sounding mysterious and suggesting that good engineers come and
join us. Which I'm dead serious about; if you've got the technical chops and want to work on some
of the most complex and advanced large-scale software infrastructure in the world,
we want you here.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eGoogle has eliminated the supplemental index, now searching the entire index for every query, leading to more relevant results and deeper web coverage.\u003c/p\u003e\n"],["\u003cp\u003ePreviously, the supplemental index stored unusual documents for more specialized searches, but it was refreshed less frequently and yielded limited visibility.\u003c/p\u003e\n"],["\u003cp\u003eThis change delivers a significantly improved user experience, particularly for non-English searches and less visible high-quality pages.\u003c/p\u003e\n"],["\u003cp\u003eThis upgrade required major technical innovation to handle the increased index size, demonstrating Google's commitment to advanced search technology.\u003c/p\u003e\n"],["\u003cp\u003eGoogle encourages talented engineers to join their team and contribute to their cutting-edge infrastructure.\u003c/p\u003e\n"]]],["In 2003, Google used a \"supplemental index\" for unusual documents, which were less frequently updated and shown. Google then worked to improve this, increasing crawl frequency and removing the \"supplemental result\" tag. They now search the entire index for every query, offering users more relevant documents. This change means webmasters will have more visibility for quality pages. This was possible because of fundamental innovations.\n"],null,["# The Ultimate Fate of Supplemental Results\n\nWednesday, December 19, 2007\n\n\nIn 2003, Google introduced a \"supplemental index\" as a way of showing more documents to users.\nMost webmasters will probably snicker about that statement, since supplemental docs were famous\nfor refreshing less often and showing up in search results less often. But the supplemental\nindex served an important purpose: it stored unusual documents that we would search in more\ndepth for harder or more esoteric queries. For a long time, the alternative was to simply not\nshow those documents at all, but this was always unsatisfying---ideally, we would search all of\nthe documents all of the time, to give users the experience they expect.\n\n\nThis led to a major effort to rethink the entire supplemental index. We improved the crawl\nfrequency and decoupled it from which index a document was stored in, and once these\n\"supplementalization effects\" were gone, the \"supplemental result\" tag itself---which only\nserved to suggest that otherwise good documents were somehow suspect---\n[was eliminated a few months ago](/search/blog/2007/07/supplemental-goes-mainstream).\nNow we're coming to the next major milestone in the elimination of the artificial difference\nbetween indices: rather than searching some part of our index in more depth for obscure queries,\nwe're now searching the whole index for every query.\n\n\nFrom a user perspective, this means that you'll be seeing more relevant documents and a much\ndeeper slice of the web, especially for non-English queries. For webmasters, this means that\ngood-quality pages that were less visible in our index are more likely to come up for queries.\nHidden behind this are some truly amazing technical feats; serving this much larger of an index\ndoesn't happen easily, and it took several fundamental innovations to make it possible. At this\npoint it's safe to say that the Google search engine works like nothing else in the world. If\nyou want to know how it actually works, you'll have to come join Google Engineering; as usual,\nit's all triple-hush-hush secrets.\\*\n\n*** ** * ** ***\n\n\n\\* Originally, I was going to give the stock Google answer, \"If I told you, I'd have to kill you.\"\nHowever, I've been informed by management that killing people violates our \"Don't be evil\" policy,\nso I'm forced to replace that with sounding mysterious and suggesting that good engineers come and\njoin us. Which I'm dead serious about; if you've got the technical chops and want to work on some\nof the most complex and advanced large-scale software infrastructure in the world,\n[we want you here](https://careers.google.com/jobs/).\n\nPosted by Yonatan Zunger, Search Quantity Team"]]
