Monitor and trigger file based data source processing

Merchant API lets you monitor the processing status of your file-based data sources and manually trigger an immediate fetch of them.

Get the latest file upload status

To check the status of the most recent processing attempt for a file-based data source, use the fileUploads.get method. You must use the alias latest for the fileuploadId to retrieve the status of the most recent upload or fetch.

This method returns details about:

• the processing state (such as, SUCCEEDED, FAILED, IN_PROGRESS)
• the number of items processed
• any issues encountered
• the upload timestamp

GET https://merchantapi.googleapis.com/datasources/v1beta/accounts/{ACCOUNT_ID}/dataSources/{DATASOURCE_ID}/fileUploads/latest

A successful request returns the FileUpload resource containing the status details.

This is an example of a response for a successful upload with warnings.

{
  "name": "accounts/123456789/dataSources/111222333/fileUploads/latest",
  "dataSourceId": "111222333",
  "processingState": "SUCCEEDED",
  "issues": [
    {
      "title": "Missing recommended attribute",
      "description": "Products are missing a recommended attribute 'description'.",
      "code": "validation/missing_recommended_attribute",
      "count": "5",
      "severity": "WARNING",
      "documentationUri": "https://support.google.com/merchants/answer/6324468"
    }
  ],
  "itemsTotal": "100",
  "itemsCreated": "90",
  "itemsUpdated": "10",
  "uploadTime": "2023-10-26T10:30:00Z"
}

This is an example of a response for an upload in progress.

{
  "name": "accounts/123456789/dataSources/111222333/fileUploads/latest",
  "dataSourceId": "111222333",
  "processingState": "IN_PROGRESS",
  "uploadTime": "2023-10-26T11:00:00Z"
}

This is an example of a response for a failed upload.

{
  "name": "accounts/123456789/dataSources/111222333/fileUploads/latest",
  "dataSourceId": "111222333",
  "processingState": "FAILED",
  "issues": [
    {
      "title": "Invalid file format",
      "description": "The uploaded file is not a valid XML or CSV file.",
      "code": "validation/invalid_file_format",
      "count": "1",
      "severity": "ERROR",
      "documentationUri": "https://support.google.com/merchants/answer/188494"
    }
  ],
  "uploadTime": "2023-10-26T11:15:00Z"
}

The following code samples show how to get the status of the most recent file upload using the fileUploads.get method with the latest alias.

Java

import com.google.api.gax.core.FixedCredentialsProvider;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.shopping.merchant.datasources.v1beta.FileUpload;
import com.google.shopping.merchant.datasources.v1beta.FileUploadName;
import com.google.shopping.merchant.datasources.v1beta.FileUploadsServiceClient;
import com.google.shopping.merchant.datasources.v1beta.FileUploadsServiceSettings;
import com.google.shopping.merchant.datasources.v1beta.GetFileUploadRequest;
import shopping.merchant.samples.utils.Authenticator;
import shopping.merchant.samples.utils.Config;

/** This class demonstrates how to get the latest data source file upload. */
public class GetFileUploadSample {

  public static void getFileUpload(Config config, String datasourceId) throws Exception {

    // Obtains OAuth token based on the user's configuration.
    GoogleCredentials credential = new Authenticator().authenticate();

    // Creates service settings using the credentials retrieved above.
    FileUploadsServiceSettings fileUploadsServiceSettings =
        FileUploadsServiceSettings.newBuilder()
            .setCredentialsProvider(FixedCredentialsProvider.create(credential))
            .build();

    // Creates FileUpload name with datasource id to identify datasource.
    String name =
        FileUploadName.newBuilder()
            .setAccount(config.getAccountId().toString())
            .setDatasource(datasourceId)
            .setFileupload("latest")
            .build()
            .toString();

    // Calls the API and catches and prints any network failures/errors.
    try (FileUploadsServiceClient fileUploadsServiceClient =
        FileUploadsServiceClient.create(fileUploadsServiceSettings)) {

      // The name has the format: accounts/{account}/datasources/{datasource}/fileUploads/latest
      GetFileUploadRequest request = GetFileUploadRequest.newBuilder().setName(name).build();

      System.out.println("Sending get FileUpload request:");
      FileUpload response = fileUploadsServiceClient.getFileUpload(request);

      System.out.println("Retrieved FileUpload below");
      System.out.println(response);
      //   return response;
    } catch (Exception e) {
      System.out.println(e);
    }
  }

  public static void main(String[] args) throws Exception {
    Config config = Config.load();
    // An ID assigned to a datasource by Google.
    String datasourceId = "123456789";

    getFileUpload(config, datasourceId);
  }
}
GetFileUploadSample.java

cURL

curl \
"https://merchantapi.googleapis.com/datasources/v1beta/accounts/{ACCOUNT_ID}/dataSources/{DATASOURCE_ID}/fileUploads/latest" \
-H "Authorization: Bearer <API_TOKEN>" \
-H "Content-Type: application/json"

Fetch a data source immediately

This method lets you request an immediate fetch and processing of a file-based data source using the dataSources.fetch method. This is useful if you've updated your source file and want Google to retrieve it sooner than the next scheduled fetch.

Note: This method should be used sparingly. The dataSources.fetch method has low quota limits, and reprocessing an entire data source can take a significant amount of time (potentially hours) depending on its size. For regular updates, especially frequent ones, consider using the Products sub-API to update your product data directly. It also lets you update individual products.

POST https://merchantapi.googleapis.com/datasources/v1beta/accounts/{ACCOUNT_ID}/dataSources/{DATASOURCE_ID}:fetch

A successful request to initiate a fetch returns an empty response as the actual processing happens asynchronously. You can monitor the outcome using the fileUploads.get method described earlier.

The following code samples show how to use the dataSources.fetch method to request an immediate fetch of a given data source.

Java

import com.google.api.gax.core.FixedCredentialsProvider;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.shopping.merchant.datasources.v1beta.DataSourceName;
import com.google.shopping.merchant.datasources.v1beta.DataSourcesServiceClient;
import com.google.shopping.merchant.datasources.v1beta.DataSourcesServiceSettings;
import com.google.shopping.merchant.datasources.v1beta.FetchDataSourceRequest;
import shopping.merchant.samples.utils.Authenticator;
import shopping.merchant.samples.utils.Config;

/**
 * This class demonstrates how to fetch a specific FileInput DataSource for a given Merchant Center
 * account.
 */
public class FetchFileDataSourceSample {

  public static void fetchDataSource(Config config, String dataSourceId) throws Exception {

    // Obtains OAuth token based on the user's configuration.
    GoogleCredentials credential = new Authenticator().authenticate();

    // Creates service settings using the credentials retrieved above.
    DataSourcesServiceSettings dataSourcesServiceSettings =
        DataSourcesServiceSettings.newBuilder()
            .setCredentialsProvider(FixedCredentialsProvider.create(credential))
            .build();

    // Creates datasource name to identify datasource.
    String name =
        DataSourceName.newBuilder()
            .setAccount(config.getAccountId().toString())
            .setDatasource(dataSourceId)
            .build()
            .toString();

    // Calls the API and catches and prints any network failures/errors.
    try (DataSourcesServiceClient dataSourcesServiceClient =
        DataSourcesServiceClient.create(dataSourcesServiceSettings)) {

      // The name has the format: accounts/{account}/datasources/{datasource}
      FetchDataSourceRequest request = FetchDataSourceRequest.newBuilder().setName(name).build();

      System.out.println("Sending FETCH DataSource request:");
      // Fetch works ONLY for FileInput DataSource type.
      dataSourcesServiceClient.fetchDataSource(request); // No response returned on success

      System.out.println("Successfully fetched DataSource.");

    } catch (Exception e) {
      System.out.println(e);
      System.exit(1);
    }
  }

  public static void main(String[] args) throws Exception {
    Config config = Config.load();
    String datasourceId = "<DATASOURCE_ID>"; // Replace with your FileInput DataSource ID.

    fetchDataSource(config, datasourceId);
  }
}
FetchFileDataSourceSample.java

cURL

curl -X POST \
"https://merchantapi.googleapis.com/datasources/v1beta/accounts/{ACCOUNT_ID}/dataSources/{DATASOURCE_ID}:fetch" \
-H "Authorization: Bearer <API_TOKEN>" \
-H "Content-Type: application/json" \
-d '{}'