Google Compute Engine

Tips, Troubleshooting, & Known Issues

This page describes tip, troubleshooting, and known issues that you might find helpful if you run into problems using Google Compute Engine.

Contents

General Tips

Saving flag values

To save gcutil flag values for quick reuse, use the --cache_flag_values=True and optional --cached_flags_file=<somefile> flags in your calls.

Viewing full JSON responses

gcutil performs most of its actions by making REST API calls. The pretty-printed results show only the most important information returned by any specific method. To see the full JSON response, use the --print_json flag which will display additional information that may not appear in the simpler, default output.

Viewing full API request and responses

If you want to see information about the REST request as well as the JSON response, use the --dump_request_response flag.

Selecting Resource Names

When selecting names for your resources, keep in mind that these friendly-names may be visible on support and operational dashboards within Google Compute Engine. For this reason, we recommend selecting resource names that do not expose any sensitive information.

Communicating from the Internet to your instances

Every project has a default network that has the following firewalls:

  • default-allow-internal - Allows network connections of any protocol and port between instances on the network.
  • default-ssh - Allows TCP connections from any source to any instance on the network, over port 22.

If you need to communicate outside of these firewall rules, you can add new network with custom firewalls. You can also set up a network proxy. Also, keep in mind that some connections are blocked by default and you need to contact the Google Compute Engine team to request access.

Accessing Google Compute Engine as a different ssh user

By default, gcutil uses the $USER variable to add users to the /etc/passwd file for ssh'ing. You can specify a different user by including the --ssh_user=<user> flag when ssh'ing into your instance.

Setting non-OAuth2 Access Key and secret credentials for gsutil/boto with Google Cloud Storage

The Google Compute Engine standard images have a boto configuration that enables automatic usage of service accounts. If you want to disable this and revert to Interoperable Storage Access Keys, add this to your .boto file:

[GSUtil]
default_api_version = 1
default_project_id = <project_number>

For more information, see Enabling API v1.0 access for Google Cloud Storage.

Troubleshooting

My persistent disk doesn't boot in the v1 API. What can I do?

Here are some tips to help troubleshoot your persistent boot disk if it doesn't load in the v1 API.

  • Examine your virtual machine instance's serial port output.

    An instance's BIOS, bootloader, and kernel will print their debug messages into the instance's serial port output, providing valuable information about any errors or issues that the instance experienced. To get your serial port information, run:

    gcutil --project=<project-id> getserialportoutput <instance-name>

    You can also access this information in the Google Developers Console.

  • Validate that your disk has a valid filesystem.

    If your filesystem is corrupted or otherwise invalid, you won't be able to launch your instance. Validate your disk's filesystem:

    1. Start an instance using the latest Google-provided image:
      gcutil --project=<project-id> addinstance <instance-name> --image=debian-7
    2. Attach your disk as a non-boot disk but don't mount it:
      gcutil --project=<project-id> attachdisk <instance-name> --disk=<disk>
    3. ssh into your instance:
      gcutil --project=<project-id> ssh <instance-name>
    4. Run the filesystem check.
      user@myinst:~$ sudo fsck <device-file>
      fsck from util-linux 2.20.1
      e2fsck 1.42.5 (29-Jul-2012)
      /: clean, 19829/655360 files, 208111/2621184 blocks
    5. Mount your disk:
      user@myinst:~$ sudo mkdir /mydisk
      user@myinst:~$ sudo mount <device-file> /mydisk
    6. Check that the disk has kernel files:
      user@myinst~:$ ls /mydisk/boot/vmlinuz-*
      /mydisk/boot/vmlinuz-3.2.0-4-amd64
  • Validate that your disk has a valid master boot record (MBR).

    Run the following command on an instance that is using the persistent boot disk in question:

    sudo dd if=<device-file> bs=512 count=1 | xxd

    If your MBR is valid, it should have the following last two bytes: 0x55 0xAA.

What does it mean for my instance to be in TERMINATED state?

If you shut down your instance using sudo shutdown or sudo poweroff, it is the equivalent of terminating it. There is no way to "freeze" an instance and restart it at a later time. You must recreate your instance if you choose to shut it down. When an instance is shut down from inside, it goes into the TERMINATED state but will still appear in the API (such as when you list instances). To remove it from the list, you must delete the instance explicitly. However, uptime for a TERMINATED instance is not billed.

Why was my instance terminated with status "Planned termination by system"?

A "Planned termination by system" status means that your instance lived in a zone that was scheduled for maintenance and has been terminated since that maintenance window went into affect.

Why is network traffic to my instance being dropped?

Google Compute Engine only allows network traffic that is explicitly permitted by your project's firewall rules to reach your instance. By default, all projects automatically comes with a default network that only allows ssh traffic. If you add any new networks, make sure you set up the appropriate firewall rules to allow network traffic because new networks deny all traffic by default, including ssh. For more information, see Communicating from the Internet to your Instances in the General Tips section of this page or the review the Networking page.

SSH Errors

Under certain conditions, it is possible a Google Compute Engine instance will no longer accept SSH connections. There are many reasons this could happen, from a full disk to an accidental misconfiguration of sshd. If this happens, accessing the instance can be quite challenging. This section describes a number of tips and approaches to troubleshoot and resolve common ssh issues.

Check your firewall rules

Google Compute Engine provisions each project with a default set of firewall rules which permit ssh traffic. If the default firewall rule that permits ssh connections is somehow removed, you’ll be unable to access your instance. Check your list of firewalls with gcutil and ensure the default-ssh rule is present. If it is missing, add it back:

user@local:~$ gcutil --project=myproject listfirewalls
+------------------------+---------------------------------------+---------+------------+-------------+-------------+
|          name          |              description              | network | source-ips | source-tags | target-tags |
+------------------------+---------------------------------------+---------+------------+-------------+-------------+
| default-allow-internal | Internal traffic from default allowed | default | 10.0.0.0/8 |             |             |
| http2                  |                                       | default | 0.0.0.0/0  |             |             |
+------------------------+---------------------------------------+---------+------------+-------------+-------------+

user@local:~$ gcutil --project=myproject addfirewall --allowed=tcp:22 default-ssh
Test the network

You can use the netcat tool to connect to your instance on port 22, and see if the network connection is working. If you connect and see an ssh banner (e.g. SSH-2.0-OpenSSH_6.0p1 Debian-4), your network connection is working, and you can rule out firewall problems:

user@local:~$ gcutil --project=myproject getinstance myinstance --format=table | grep external-ip
|     external-ip        | 173.255.115.70

user@local:~$ nc 173.255.115.70 22 # Check for SSH banner
SSH-2.0-OpenSSH_6.0p1 Debian-4
Try a fresh user

The issue that prevents you from logging in may be limited to your account (e.g. if the permissions on your ~/.ssh/authorized_keys file were set incorrectly). The first thing to try is creating a new account on the machine. Because gcutil sets up keys and accounts based on your username, the easiest way to do this is to create a new instance (using a f1-micro machine type is fine), log in, add a new user, and switch to this user’s account. Then, you can use gcutil to try to ssh to your existing instance. If this works, you will be able to use this new account to fix the permissions on your primary user’s account.

user@local:~$ gcutil --project=myproject addinstance --service_account_scopes=compute-rw temp-machine
...

user@local:~$ gcutil --project=myproject ssh temp-machine
...

user@temp-instance:~$ sudo useradd -m tempuser
user@temp-instance:~$ sudo su - tempuser
user@temp-instance:~$ gcutil --project=myproject ssh your-instance
Mount your disk on a temporary instance

If the above set of steps doesn’t work for you, and the instance you’re interested in is booted from a persistent disk, you can delete the instance (but save its disk), and attach this disk to another machine. On the temporary machine, you can mount it and determine what prevented your ssh connection from working, and finally recreate the original instance with the same boot disk.

user@local:~$ gcutil --project=myproject deleteinstance myoldinstance --nodelete_pd # Delete instance but keep the disk
...

user@local:~$ gcutil --project=myproject addinstance debugger --disk=boot-mydisk,boot
....

user@local:~$ gcutil --project=myproject ssh debugger
...

user@debugger:~$ sudo su -
user@debugger:~$ mkdir /mnt/myinstance
user@debugger:~$ mount /dev/disk/by-id/scsi-0Google_PersistentDisk_boot-myinstance /mnt/myinstance
user@debugger:~$ cd /mnt/myinstance/var/log
user@debugger:~$ ls # Identify the issue preventing ssh from working
user@debugger:~$ exit
user@local:~$ gcutil --project=myproject addinstance myoldinstance --disk=mydisk # Re-add your instance and persistent disk
Inspect an instance without shutting it down

You may have an instance you can’t ssh to that continues to correctly serve production traffic. In this case, you may wish to inspect its disk without interrupting its ability to serve users. The steps here are similar to the previous section, but you’ll make use of persistent disk snapshots. First, take a snapshot of the instance’s boot disk, then create a new disk from that snapshot, create a temporary instance, and finally attach and mount the new persistent disk to your temporary instance.

user@local:~$ gcutil --project=myproject addsnapshot --source_disk=<source-disk> myinstance-snapshot
...

user@local:~$ gcutil --project=myproject adddisk --source_snapshot=myinstance-snapshot myinstance-debugging
...

user@local:~$ gcutil --project=myproject addinstance debugger
...

user@local:~$ gcutil --project=myproject attachdisk --disk=myinstance-debugging debugger
...

user@local:~$ gcutil --project=myproject ssh debugger
...

user@debugger:~$ sudo su -
user@debugger:~$ mkdir /mnt/myinstance
user@debugger:~$ mount /dev/disk/by-id/scsi-0Google_PersistentDisk_myinstance-debugging /mnt/myinstance
user@debugger:~$ cd /mnt/myinstance/var/log
user@debugger:~$ ls # Identify the issue preventing ssh from working

Known Issues

CentOS image v20131120 introduced a breaking change where iptables are turned on by default.

The v20131120 release of CentOS 6 image, centos-6-v20131120, has a breaking change where iptables are turned on by default. This prevents external traffic from reaching CentOS instances that are running centos-6-v20131120, even if there is a relevant Firewall Rule resource permitting the connection.

As a workaround, users will need to disable iptables or update iptables to permit the desired connection (in addition to permitting the traffic using firewall rules). To disable iptables, run:

# Save your iptable settings
user@centos-instance:~$ sudo service iptables save

# Stop the iptables service
user@centos-instance:~$ sudo service iptables stop

# Disable iptables on start up
user@centos-instance:~$ sudo chkconfig iptables off

To update iptables, review the iptables documentation.

Google-provided images have known bug with the ext4/scsi driver in the stable Debian and CentOS kernels

A known ext4 bug may cause memory leak and eventual crash of a virtual machine instance under heavy persistent disk load. Both centos-6-v20131120 and debian-7-wheezy-v20131120 images are affected. For details, please refer to this Linux Kernel Mailing list thread.

As a workaround, you can use the Debian backport image which contains fixes to this bug.

Updating project instance metadata does not provide optimistic locking

Date Reported: December 2013

It is currently not possible to perform opportunistic locking on project-level metadata which may cause some metadata, such as sshKeys, to be overwritten because of race conditions. This is a known bug, because the Projects resource does not currently provide a fingerprint to use for updating requests. If you noticed that your sshKeys are missing from your project metadata, the current work around is to reinsert your credentials, either through the Google Developers Console or using gcutil.

Instance names longer than 32 characters can cause problems with various UNIX tools.

Date Reported: June 2012

Although instance names can be up to 63 characters, names that are longer than 32 characters may cause some tools to be unreliable, including tools that may run during boot. As a workaround, choose instance names that are shorter than 32 characters.

Deleting an image being used by instances will cause those instances to terminate once the instance hits a failure.

Date Reported: June 2012

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.