This page describes tip, troubleshooting, and known issues that you might find helpful if you run into problems using Google Compute Engine.
- Saving flag values
- Viewing full JSON responses
- Viewing full API request and responses
- Selecting Resource Names
- Communicating from the Internet to your instances
- Accessing Google Compute Engine as a different ssh user
- Setting non-OAuth2 Access Key and secret credentials for gsutil/boto with Google Cloud Storage
Saving flag values
To save gcutil flag values for quick reuse, use the
--cache_flag_values=True and optional
--cached_flags_file=<somefile> flags in your calls.
Viewing full JSON responses
gcutil performs most of its actions by making REST API calls. The pretty-printed
results show only the most important information returned by any specific method.
To see the full JSON response, use the
--print_json flag which
will display additional information that may not appear in the simpler, default
Viewing full API request and responses
If you want to see information about the REST request as well as the JSON
response, use the
Selecting Resource Names
When selecting names for your resources, keep in mind that these friendly-names may be visible on support and operational dashboards within Google Compute Engine. For this reason, we recommend selecting resource names that do not expose any sensitive information.
Communicating from the Internet to your instances
Every project has a default network that has the following firewalls:
- default-allow-internal - Allows network connections of any protocol and port between instances on the network.
- default-ssh - Allows TCP connections from any source to any instance on the network, over port 22.
If you need to communicate outside of these firewall rules, you can add new network with custom firewalls. You can also set up a network proxy. Also, keep in mind that some connections are blocked by default and you need to contact the Google Compute Engine team to request access.
Accessing Google Compute Engine as a different ssh user
By default, gcutil uses the
$USER variable to add users to the
/etc/passwd file for ssh'ing. You can specify a different user by
--ssh_user=<user> flag when ssh'ing
into your instance.
Setting non-OAuth2 Access Key and secret credentials for gsutil/boto with Google Cloud Storage
The Google Compute Engine standard images have a boto configuration that enables automatic usage of service accounts. If you want to disable this and revert to Interoperable Storage Access Keys, add this to your .boto file:
[GSUtil] default_api_version = 1 default_project_id = <project_number>
For more information, see Enabling API v1.0 access for Google Cloud Storage.
- My virtual machine instance hangs during boot time in the v1 API.
- My persistent disk doesn't boot in the v1 API.
- What does it mean for my instance to be in
- Why was my instance terminated with status "Planned termination by system"?
- Why is network traffic to my instance being dropped?
- SSH errors
My virtual machine instance hangs during boot time in the v1 API.
If you tried to boot your virtual machine instance in the new v1 API but ran into an issue where the instance hangs, it may be possible your instance is using a Google-supplied kernel that is not longer available in v1. Previously, in the v1beta16 API, Compute Engine injected a Google-supplied kernel at boot time. As of the v1 API, Compute Engine now uses stock kernels from the boot image or boot persistent disk. This allows for more flexibility but does require a one-time manual step to update persistent disks and images that were created before the release of v1.
To verify that this is the issue, review the contents of the virtual
machine instance's serial port console, either using
gcutil getserialportoutput or
in the Google Developers Console. In gcutil, run:
gcutil --project=<project-id> getserialportoutput <instance-name>
If your instance hangs with the following as the final message:
<snip> Booting from Hard Disk... Booting from 0000:7c00 <end of output>
You may need to update your disk or image. For detailed steps on how to do so, see the v1 transition guide.
My persistent disk doesn't boot in the v1 API. What can I do?
Here are some tips to help troubleshoot your persistent boot disk if it doesn't load in the v1 API.
- Examine your virtual machine instance's serial port output.
An instance's BIOS, bootloader, and kernel will print their debug messages into the instance's serial port output, providing valuable information about any errors or issues that the instance experienced. To get your serial port information, run:
gcutil --project=<project-id> getserialportoutput <instance-name>
You can also access this information in the Google Developers Console.
- Validate that your disk has a valid filesystem.
If your filesystem is corrupted or otherwise invalid, you won't be able to launch your instance. Validate your disk's filesystem:
- Start an instance using the latest Google-provided image:
gcutil --project=<project-id> addinstance <instance-name> --image=debian-7
- Attach your disk as a non-boot disk but don't mount it:
gcutil --project=<project-id> attachdisk <instance-name> --disk=<disk>
- ssh into your instance:
gcutil --project=<project-id> ssh <instance-name>
- Run the filesystem check.
user@myinst:~$ sudo fsck <device-file> fsck from util-linux 2.20.1 e2fsck 1.42.5 (29-Jul-2012) /: clean, 19829/655360 files, 208111/2621184 blocks
- Mount your disk:
user@myinst:~$ sudo mkdir /mydisk user@myinst:~$ sudo mount <device-file> /mydisk
- Check that the disk has kernel files:
user@myinst~:$ ls /mydisk/boot/vmlinuz-* /mydisk/boot/vmlinuz-3.2.0-4-amd64
- Start an instance using the latest Google-provided image:
- Validate that your disk has a valid master boot record (MBR).
Run the following command on an instance that is using the persistent boot disk in question:
sudo dd if=
bs=512 count=1 | xxd
If your MBR is valid, it should have the following last two bytes:
What does it mean for my instance to be in
If you shut down your instance using
sudo shutdown or
it is the equivalent of terminating it. There is no way to "freeze" an instance
and restart it at a later time. You must recreate your instance if you choose
to shut it down. When an instance is shut down from inside, it goes into the
TERMINATED state but will still appear in the API (such as when
you list instances). To remove it from the list, you must delete the instance
explicitly. However, uptime for a
TERMINATED instance is not billed.
Why was my instance terminated with status "Planned termination by system"?
A "Planned termination by system" status means that your instance lived in a zone that was scheduled for maintenance and has been terminated since that maintenance window went into affect.
Why is network traffic to my instance being dropped?
Google Compute Engine only allows network traffic that is explicitly permitted by your project's firewall rules to reach your instance. By default, all projects automatically comes with a default network that only allows ssh traffic. If you add any new networks, make sure you set up the appropriate firewall rules to allow network traffic because new networks deny all traffic by default, including ssh. For more information, see Communicating from the Internet to your Instances in the General Tips section of this page or the review the Networking page.
Under certain conditions, it is possible a Google Compute Engine instance will no longer accept SSH connections. There are many reasons this could happen, from a full disk to an accidental misconfiguration of sshd. If this happens, accessing the instance can be quite challenging. This section describes a number of tips and approaches to troubleshoot and resolve common ssh issues.
Check your firewall rules
Google Compute Engine provisions each project with a default set of firewall
rules which permit ssh traffic. If the default firewall rule that permits ssh
connections is somehow removed, you’ll be unable to access your instance. Check
your list of firewalls with gcutil and ensure the
default-ssh rule is
present. If it is missing, add it back:
user@local:~$ gcutil --project=myproject listfirewalls +------------------------+---------------------------------------+---------+------------+-------------+-------------+ | name | description | network | source-ips | source-tags | target-tags | +------------------------+---------------------------------------+---------+------------+-------------+-------------+ | default-allow-internal | Internal traffic from default allowed | default | 10.0.0.0/8 | | | | http2 | | default | 0.0.0.0/0 | | | +------------------------+---------------------------------------+---------+------------+-------------+-------------+ user@local:~$ gcutil --project=myproject addfirewall --allowed=tcp:22 default-ssh
Test the network
You can use the netcat tool to connect to your instance on port 22, and see
if the network connection is working. If you connect and see an ssh banner
SSH-2.0-OpenSSH_6.0p1 Debian-4), your network connection is working, and
you can rule out firewall problems:
user@local:~$ gcutil --project=myproject getinstance myinstance --format=table | grep external-ip | external-ip | 126.96.36.199 user@local:~$ nc 188.8.131.52 22 # Check for SSH banner SSH-2.0-OpenSSH_6.0p1 Debian-4
Try a fresh user
The issue that prevents you from logging in may be limited to your account
(e.g. if the permissions on your
~/.ssh/authorized_keys file were set incorrectly).
The first thing to try is creating a new account on the machine. Because gcutil
sets up keys and accounts based on your username, the easiest way to do this
is to create a new instance (using a f1-micro machine type is fine), log in,
add a new user, and switch to this user’s account. Then, you can use gcutil
to try to ssh to your existing instance. If this works, you will be able to use
this new account to fix the permissions on your primary user’s account.
user@local:~$ gcutil --project=myproject addinstance --service_account_scopes=compute-rw temp-machine ... user@local:~$ gcutil --project=myproject ssh temp-machine ... user@temp-instance:~$ sudo useradd -m tempuser user@temp-instance:~$ sudo su - tempuser user@temp-instance:~$ gcutil --project=myproject ssh your-instance
Mount your disk on a temporary instance
If the above set of steps doesn’t work for you, and the instance you’re interested in is booted from a persistent disk, you can delete the instance (but save its disk), and attach this disk to another machine. On the temporary machine, you can mount it and determine what prevented your ssh connection from working, and finally recreate the original instance with the same boot disk.
user@local:~$ gcutil --project=myproject deleteinstance myoldinstance --nodelete_pd # Delete instance but keep the disk ... user@local:~$ gcutil --project=myproject addinstance debugger --disk=boot-mydisk,boot .... user@local:~$ gcutil --project=myproject ssh debugger ... user@debugger:~$ sudo su - user@debugger:~$ mkdir /mnt/myinstance user@debugger:~$ mount /dev/disk/by-id/scsi-0Google_PersistentDisk_boot-myinstance /mnt/myinstance user@debugger:~$ cd /mnt/myinstance/var/log user@debugger:~$ ls # Identify the issue preventing ssh from working user@debugger:~$ exit user@local:~$ gcutil --project=myproject addinstance myoldinstance --disk=mydisk # Re-add your instance and persistent disk
Inspect an instance without shutting it down
You may have an instance you can’t ssh to that continues to correctly serve production traffic. In this case, you may wish to inspect its disk without interrupting its ability to serve users. The steps here are similar to the previous section, but you’ll make use of persistent disk snapshots. First, take a snapshot of the instance’s boot disk, then create a new disk from that snapshot, create a temporary instance, and finally attach and mount the new persistent disk to your temporary instance.
user@local:~$ gcutil --project=myproject addsnapshot --source_disk=<source-disk> myinstance-snapshot ... user@local:~$ gcutil --project=myproject adddisk --source_snapshot=myinstance-snapshot myinstance-debugging ... user@local:~$ gcutil --project=myproject addinstance debugger ... user@local:~$ gcutil --project=myproject attachdisk --disk=myinstance-debugging debugger ... user@local:~$ gcutil --project=myproject ssh debugger ... user@debugger:~$ sudo su - user@debugger:~$ mkdir /mnt/myinstance user@debugger:~$ mount /dev/disk/by-id/scsi-0Google_PersistentDisk_myinstance-debugging /mnt/myinstance user@debugger:~$ cd /mnt/myinstance/var/log user@debugger:~$ ls # Identify the issue preventing ssh from working
v20131120 introduced a breaking
change where iptables are turned on by default.
v20131120 release of CentOS 6 image,
has a breaking change where iptables
are turned on by default. This prevents external traffic from reaching CentOS
instances that are running
centos-6-v20131120, even if there is
a relevant Firewall Rule resource
permitting the connection.
As a workaround, users will need to disable iptables or update iptables to permit the desired connection (in addition to permitting the traffic using firewall rules). To disable iptables, run:
# Save your iptable settings user@centos-instance:~$ sudo service iptables save # Stop the iptables service user@centos-instance:~$ sudo service iptables stop # Disable iptables on start up user@centos-instance:~$ sudo chkconfig iptables off
To update iptables, review the iptables documentation.
Google-provided images have known bug with the ext4/scsi driver in the stable Debian and CentOS kernels
A known ext4 bug may cause memory leak and eventual crash of a virtual machine instance under heavy persistent disk load. Both centos-6-v20131120 and debian-7-wheezy-v20131120 images are affected. For details, please refer to this Linux Kernel Mailing list thread.
As a workaround, you can use the Debian backport image which contains fixes to this bug.
Updating project instance metadata does not provide optimistic locking
Date Reported: December 2013
It is currently not possible to perform opportunistic locking on project-level
metadata which may cause some metadata, such as
to be overwritten because of race conditions. This is a known bug, because the Projects
resource does not currently provide a
fingerprint to use for
updating requests. If you noticed that your
sshKeys are missing
from your project metadata, the current work around is to reinsert your
credentials, either through the Google Developers Console
or using gcutil.
Instance names longer than 32 characters can cause problems with various UNIX tools.
Date Reported: June 2012
Although instance names can be up to 63 characters, names that are longer than 32 characters may cause some tools to be unreliable, including tools that may run during boot. As a workaround, choose instance names that are shorter than 32 characters.
Deleting an image being used by instances will cause those instances to terminate once the instance hits a failure.
Date Reported: June 2012