Configure NTP on a VM


Many software systems that depend on careful sequencing of events rely on a stable, consistent system clock. System logs written by most services include a timestamp, which helps debug issues that occur between various components of your system. To help keep system clocks in sync, Compute Engine instances are preconfigured to use network time protocol (NTP).

In addition to keeping server time in sync, NTP is helpful in the rare case of a leap second. A leap second is a one-second adjustment made to UTC time to account for changes in the Earth's rotation. Leap seconds don't happen at routine intervals, because the Earth's rotation speed varies irregularly in response to climatic and geological events. Previous leap seconds have noticeably impacted a variety of services and applications on the web. NTP servers help ensure that all servers report the same time during the event of a leap second.

This document describes how to configure NTP servers on your virtual machines (VMs) to behave properly in the case of a leap second.

Google NTP servers and leap smearing

Leap seconds for a Unix operating system are commonly implemented by repeating the last second of the day. This can cause problems with software that expects timestamps to only ever increase. To get around this problem, the time servers at Google Cloud "smear" the extra second over twenty-four hours—twelve before and twelve after the leap second event—so that computers don't see the extra second all at once as a repeated timestamp. This reduces risk in systems that depend on a consistent timestamp. It's recommended that all Compute Engine virtual machine (VM) instances are configured to use the internal Google NTP services.

Configure NTP for your instances

Google Cloud can't predict how external NTP services, such as pool.ntp.org, will handle the leap second. If at all possible, it's recommended that you don't use external NTP sources with Compute Engine VMs. Even worse, using both Google's NTP service and an external service can result in unpredictable changes in the system time. Using only a single external NTP source is preferable to using a mix, but external NTP services, such as pool.ntp.org, will likely use stepping to handle the leap second. As a result, your VMs might see a repeated timestamp.

The safest approach is to configure your Compute Engine VMs to use a single NTP server—the internal NTP server provided by Google. Don't mix external NTP servers and Google NTP servers, as this could result in unexpected behavior.

To ensure your VMs are correctly configured, follow these instructions.

Linux (chrony)

By default, the majority of new Linux releases use chrony to manage their NTP settings and time synchronization. To make sure that chrony uses only the internal NTP service, check the chrony configuration and remove external NTP servers.

  1. Use ssh to connect to your instance.

    Console

    To use the console to connect to the VM using SSH, follow these steps:

    1. Go to the VM instances page in the Google Cloud console.

      Go to VM instances

    2. Click the SSH button for the VM that you want to configure.

      SSH button.

    gcloud

    To use the Google Cloud CLI to connect to the VM using SSH, run the following command:

    gcloud compute instances ssh VM_NAME
    

    Replace VM_NAME with the name of the VM that you're connecting to.

  2. On your instance, run chronyc sources to check the current state of your NTP configuration:

    $ chronyc sources
    

    The output looks similar to the following:

     210 Number of sources = 2
     MS Name/IP address         Stratum Poll Reach LastRx Last sample
     ===============================================================================
     ^* metadata.google.internal      2   6   377     4    -14us[  -28us] +/-  257us
     ^- 38.229.53.9                   2   6    37     4   -283us[ -297us] +/-   28ms
    

    If you see a single record pointing at metadata.google or metadata.google.internal, you don't need to make any changes. If you see multiple sources, mixed between metadata.google and a public source such as pool.ntp.org, update your sources to remove any external NTP servers.

    In the example output, there are two records, one pointing at metadata.google.internal and another pointing to an external address. Because there are multiple sources, you would update your NTP servers to remove the 38.229.53.9 address, as described in the next step.

  3. Configure your NTP servers to remove external NTP servers.

    To remove the additional NTP server from the list, edit the /etc/chrony/chrony.conf file using your favorite text editor. Find all the lines that start with server external_source_ip_or_name and remove them.

    After editing your /etc/chrony/chrony.conf file, restart the chrony service. The command to restart might vary depending on the Linux distribution, as shown in the following examples:

    sudo service chrony restart
    
    sudo systemctl restart chrony
    
  4. Verify your configuration by running the chronyc sources command again:

    $ chronyc sources
    

    The output should look similar to the following:

     210 Number of sources = 1
     MS Name/IP address         Stratum Poll Reach LastRx Last sample
     ===============================================================================
     ^* metadata.google.internal      2   7   377    98  -1343ns[-1588ns] +/-  396us
    

Linux (ntpd)

Most older Linux distributions use ntpd to manage their NTP settings and time synchronization. To make sure that ntpd uses only the internal NTP service, check the ntpd configuration and remove external NTP servers.

  1. Use ssh to connect to your instance.

    Console

    To use the console to connect to the VM using SSH, follow these steps:

    1. Go to the VM instances page in the Google Cloud console.

      Go to VM instances

    2. Click the SSH button for the VM that you want to configure.

      SSH button.

    gcloud

    To use the Google Cloud CLI to connect to the VM using SSH, run the following command:

    gcloud compute instances ssh VM_NAME
    

    Replace VM_NAME with the name of the VM that you're connecting to.

  2. On your instance, run ntpq -p to check the current state of your NTP configuration:

    $ ntpq -p
    

    The output looks similar to following:

    remote           refid           st t when poll reach   delay   offset  jitter
    
    ==============================================================================
    *metadata.google 255.28.23.83     2 u   27   64    1    0.634   -2.537   2.285
    *217.162.232.173 130.149.17.8     2 u  191 1024  176   79.245    3.589  27.454
    

    If you see a single record pointing at metadata.google or metadata.google.internal, you don't need to make any changes. If you see multiple sources, mixed between metadata.google and a public source such as pool.ntp.org, you need to update your sources to remove any external NTP servers.

    In the example output, there are two records, one pointing at metadata.google and another pointing to an external address. Because there are multiple sources, you would need to update your NTP servers to remove the *217.162.232.173 address, as described in the next step.

  3. Configure your NTP servers to remove external sources.

    To configure your NTP servers, edit the /etc/ntp.conf file using your favorite text editor. Find the servers section of the configuration, and remove all non-Google NTP sources, for example:

    vim /etc/ntp.conf
    
    # You do need to talk to an NTP server or two (or three).
    #server ntp.your-provider.example
    ...
    server metadata.google.internal iburst
    

    After editing your /etc/ntp.conf file, restart the NTP service. The command to restart might vary based on the Linux distribution:

    sudo service ntp reload
    
  4. Verify your configuration by running the ntpq -p command again:

    ntpq -p
    
    remote           refid           st t when poll reach   delay   offset  jitter
    ==============================================================================
    *metadata.google 255.28.23.83     2 u   27   64    1    0.634   -2.537   2.285
    

Windows

  1. Go to the VM instances page in the Google Cloud console.

    Go to VM instances

  2. Click the RDP button next to the Windows instance you want to connect to.

    SSH button.

  3. After you are logged in, right-click on the PowerShell icon and select Run as administrator.

    The PowerShell icon.

  4. When the command-prompt loads, run the following command to see the current NTP configuration:

    w32tm /query /configuration
    
    [Configuration]
    ...
    Type: NTP (Local)
    NtpServer: metadata.google.internal,
    ...
    

    If you see a single record pointing at metadata.google or metadata.google.internal, you don't need to make any changes. If you see multiple sources, mixed between metadata.google and a public source, you need to remove the external server. Follow the Windows' guide for configuring your NTP server.

  5. To ensure the most widespread software compatibility on Windows VMs, Google recommends that you use the gVNIC driver to ensure sub-millisecond NTP accuracy with metadata.google.

    If you must use VirtIO with your Windows VM, to get sub-millisecond accuracy with NTP servers, Google recommends that you don't use the Windows Time Service (stop and unregister w32tm).

    1. Stop the Windows Time Service:

      net stop w32time
      
    2. Remove the Windows Time Service from the registry:

      w32tm /unregister
      
    3. After the Windows Time Service has been stopped and removed from the registry, install the Meinberg NTP client.

      Follow the configuration instructions as provided in the Meinberg documentation.

    4. Configure the NTP server for the Meinberg NTP client as metadata.google.internal.

      After you finish configuring NTP, wait between 5 and 15 minutes for the system clock in the VM to stabilize with the NTP server.

      For information about why using w32tm is not recommended, see the Known Issues documentation.

Using leap smearing with systems external to Google Cloud

The leap smearing feature of Google's NTP servers is a convenient way to manage the risk involved with replaying a second on time sensitive systems. Other NTP services may provide an acceptable work around for most software systems. However, it's important that you don't mix Google leap smearing NTP services with public NTP stepping services.

To synchronize devices outside Google Cloud to smeared time, you can use Google Public NTP for those devices. The Google Public NTP uses the same leap second smearing that's provided to Compute Engine VMs.

What's next