The Connector SDK and Cloud Search REST API allow the creation of Cloud Search Indexing Queues used to perform the following tasks:
Maintain the per-document state (status, hash values, and so on) which can be used to keep your index in sync with your repository.
Maintain a list of items to be indexed as discovered during the traversal process.
Prioritize items in queues based on item status.
Maintain additional state information for efficient integration such as checkpoints, change token, and so on.
Status & priority
ERROR- Item encountered asynchronous error during the indexing process and needs to be re-indexed.
MODIFIED- Item that was previously indexed and has since been modified in the repository since the last indexing.
NEW_ITEM— Item that is not indexed.
ACCEPTED— Document that was previously indexed and has not changed in the repository since the last indexing.
When two items in a queue have the same status, higher priority is given to the items that have been in the queue for the longest period of time.
Queue operations (Connector SDK)
The Content Connector SDK provides operations for pushing items to, and pulling items from, a queue.
To package and push an item to a queue, use the
You do not need to do anything specific to pull items from a queue for
processing. Instead, the SDK automatically pulls items from the queue, in priority
order, using the
Queue operations (REST API)
The REST API provides the following two methods for pushing items to and pulling items from a queue:
method adds IDs to the queue. This method can be called with a specific
value which determines the result of push operation. For a list of
type values, refer
item.type field in the Items.push
Pushing a new ID results in adding a new entry with an
The optional payload is always stored, treated as an opaque value, and returned
When an item is polled, it is reserved meaning it cannot be returned by
another call to
polled entries. For further information about reserved and unreserved entries,
refer to the Items.poll section..
Items.push with hashes
The Cloud Search REST API supports specifying metadata and content hash values
requests. Instead of specifying
the metadata and/or content hash values
can be specified with a push request. The Cloud Search Indexing Queue compares
the provided hash values with the stored values available with the item in the
data source. If mismatched, that entry is marked as
MODIFIED. If a corresponding
item doesn't exist in the index, then the status is
The Items.poll method retrieves the highest priority entries from the queue. The requested and returned status values indicate the status(es) of the priority queue(s) requested or the status of the returned IDs.
By default, entries from any section of the queue may be returned, based on
priority. Each returned entry is reserved, and is not returned by other
until one of the following cases is met: